OpenJDK的替代方法:Kafka流的8个高山



我正在使用OpenJDK:8-Alpine用于部署Kafka Streams应用程序。我正在使用窗口,它崩溃了,以下错误:

Exception in thread "app-4a382bdc55ae-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni94709417646402513.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/librocksdbjni94709417646402513.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
    at org.rocksdb.Options.<clinit>(Options.java:22)
    at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:116)
    at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:43)
    at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:91)
    at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:100)
    at org.apache.kafka.streams.state.internals.RocksDBSessionStore.put(RocksDBSessionStore.java:122)
    at org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:78)
    at org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:33)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.putAndMaybeForward(CachingSessionStore.java:177)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.access$000(CachingSessionStore.java:38)
    at org.apache.kafka.streams.state.internals.CachingSessionStore$1.apply(CachingSessionStore.java:88)
    at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:142)
    at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:100)
    at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:127)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.flush(CachingSessionStore.java:193)
    at org.apache.kafka.streams.state.internals.MeteredSessionStore.flush(MeteredSessionStore.java:169)
    at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:244)
    at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(AbstractTask.java:195)
    at org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:332)
    at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:312)
    at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:307)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:297)
    at org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:67)
    at org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:357)
    at org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:347)
    at org.apache.kafka.streams.processor.internals.TaskManager.commitAll(TaskManager.java:403)
    at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:994)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:811)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)

搜索上述问题,我遇到了https://issues.apache.org/jira/browse/kafka-4988。但这没有帮助。

因此,阿尔卑斯山使用musl-libc,但它不受RocksDB的支持。为RockSDB增加支持Musl-LIBC的问题:Facebook/Rocksdb#3143。

问题:是否有可以使用我可以使我的kafka流应用程序运行的OpenJDK Docker映像,并且不会出现Rocksdb问题?

edit-1:我尝试了 RUN apk add --no-cache bash libc6-compat,但是它也失败了,以下错误:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000000011e336, pid=1, tid=0x00007fc6a3cc8ae8
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.9.0
# Distribution: Custom build (Tue Oct 23 11:27:22 UTC 2018)
# Problematic frame:
# C  0x000000000011e336
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

对我有用的解决方案是将docker映像从 openjdk:8-alpine更改为 adoptopenjdk/openjdk8:alpine-slim

adoptopenjdk/openjdk8:alpine-slim glibc兼容。

我从http://blog.gilliard.lol/2018/11/11/05/alpine-jdk11-images.html。

了解了此图像。

希望它对某人有帮助。

您已链接的票,https://issues.apache.org/jira/jira/browse/kafka-4988,对问题提供了很多见解。

如前所述,它看起来与Musl Libc不兼容,并且需要Glibc。

安装libc6-compact可能不会做:它在Musl libc上提供了一个兼容层,该层模仿了glibc库结构并实现了一些丢失的功能,但这与安装glibc perse并不相同。GLIBC是一个复杂的实现,因此兼容库和实际GLIBC之间可能没有一对点1的相关性。有关微妙的Musl/Glibc差异,请参见此处。

阅读票证评论,错误的库可能是 librocksdbjni.so,依赖于libstdc 6。

因此,我将尝试以下(将openjdk:8-alpine作为您的基本图像):

  • 确保安装了libstdc++软件包;
  • 在此Dockerfile中手动安装glibc(由anapsix/docker-alpine-java删除的Glibc Dockerfile的版本)。

而不是更改默认的docker base图像,您可以为高山发行版构建glibc。甚至比这更好,您可以从Sasha Gerrand的Github页面中获取预先构建的APK。这是我们在Dockerfile中添加的内容,以使所有这些都与他的预制APK一起使用:

# # GLIBC - Kafka Dependency (RocksDB)
# Used by Kafka for default State Stores.
# glibc's apk was built for Alpine Linux and added to our repository
# from this source: https://github.com/sgerrand/alpine-pkg-glibc/
ARG GLIBC_APK=glibc-2.30-r0.apk
COPY ${KAFKA_DIR}/${GLIBC_APK} opt/
RUN apk add --no-cache --allow-untrusted opt/${GLIBC_APK}
# C++ Std Lib - Kafka Dependency (RocksDB)
RUN apk add --no-cache libstdc++

有一个已知的问题,即来自https://issues.apache.org/jira/browse/browse/kafka-4988的Kafka流和Alpine Linux不兼容。对于使用Java 11的人,adoptopenjdk/openjdk11:alpine-slim对我来说很好。另一个解决方案是仍然使用OpenJDK:11-JDK-Alpine图像作为基础,但随后安装Snappy-Java lib

FROM openjdk:11-jdk-alpine
RUN apk update && apk add --no-cache gcompat
...

最新更新