Apache Kafka 和 Zookeeper 连接问题

Question

我正在尝试使用Docker群创建一个多节点Apache Kafka集群。为此，我使用以下 docker-compose.yml

version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:latest
deploy:
mode: global
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
environment:
HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" # Normal instances
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
KAFKA_ADVERTISED_PORT: 9094
KAFKA_PROTOCOL_NAME: INSIDE
KAFKA_PORT: 9092
KAFKA_CREATE_TOPICS: myTopic:1:1
volumes:
- /var/run/docker.sock:/var/run/docker.sock

但是，创建kafka_kafka服务时，我遇到以下错误：

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://ugur:9094': No security protocol defined for listener PLAINTEXT
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:312)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1300)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1360)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1338)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1028)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1008)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.Kafka$.main(Kafka.scala:59)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.Kafka.main(Kafka.scala)
kafka_kafka.0.wx4chyussuuc@ugur    | Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener PLAINTEXT
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$$anonfun$securityProtocol$1$1.apply(EndPoint.scala:48)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.AbstractMap.getOrElse(Map.scala:59)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:47)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:56)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$$anonfun$6.apply(CoreUtils.scala:309)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$$anonfun$6.apply(CoreUtils.scala:309)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
kafka_kafka.0.wx4chyussuuc@ugur    |    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
kafka_kafka.0.wx4chyussuuc@ugur    |    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:309)

我还打开了必要的端口，这些端口是这样的：

到操作从

-- ------ ----

9094 允许在任何地方进入 9092 允许在任何地方进入 8081 允许进入任何地方


8080 允许进入任何地方
2377 允许任何地方进入
7946 允许任何地方进入
4789 允许任何地方进入 9094 (v6( 允许任何地方进入 (v6(

9092 (v6( 允许任何地方进入 (v6(
8081 (v6( 允许随处进入 (v6(8080 (v6( 允许随处进入 (v6(2377 (v6( 允许随处进入 (v6(7946 (v6( 允许随处进入 (v6(4789 (v6( 允许随处进入 (v6(

Answer 1

您是否尝试使用 GitHub 上提供的 docker-compose-swarm.yml？

https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-swarm.yml

也许缺少的属性是关键：

KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094

Answer 2

我添加了以下选项：

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
ALLOW_PLAINTEXT_LISTENER: "yes"

显然，纯文本侦听器并不安全，但它应该让您入门。