如何将Kafka与docker和python结合使用

Question

我正在尝试使用Docker建立到Kafka的连接。当在本地运行代码时，一切都正常工作，但当尝试使用Docker Compose运行它时，我得到了以下错误：

Traceback (most recent call last):
File "alerts_monitor_service.py", line 23, in <module>
collect_data_from_db_and_alert()
File "alerts_monitor_service.py", line 11, in collect_data_from_db_and_alert
kafka_dic = kafka_set_up()
File "/monitor_service/src/logic.py", line 49, in kafka_set_up
producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
File "/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py", line 381, in __init__
client = KafkaClient(metrics=self._metrics, metric_group_prefix='producer',
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 244, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 900, in check_version
raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

这就是我在docker-compose.yaml文件中设置kafka的方式：

version: "3.6"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- 2181:2181
depends_on:
- mongo
kafka:
image: wurstmeister/kafka
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- mongo
- zookeeper
mongo:
container_name: mongo_db
image: mongo:latest
ports:
- "27017:27017"
volumes:
- "./data/mongo:/data/db"

这就是我得到错误的地方：

producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
producer = KafkaProducer()
producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))

Answer 1

以下是Dockerfile示例：https://github.com/up9inc/async-ms-demo/tree/main/kafka

您可以将其构建为docker build . -t kafka，然后作为docker run -it -p 9092:9092 kafka在本地运行。然后它会在当地为你工作。

对于docker compose，您需要使用KAFKA_ADVERTISED_LISTENERSenv-var集运行相同的docker映像，以便其他主机正确找到它。请参阅此处的示例：https://github.com/up9inc/async-ms-demo/blob/main/docker-compose.yml#L39