将 Presto 连接到 Kafka 失败 - 目录'kafka'不存在

Question

我尝试做一些类似于这里概述的说明。就我的情况而言，我想在docker中使用docker-compose来启动presto和kafka。

所以我的docker-compose。Yaml看起来像这样:

version: '2'  
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 22181:2181
networks:
- shared
kafka1:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,PLAINTEXT_HOST://localhost:29092,LISTENER_DOCKER_INTERNAL://kafka1:19092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,LISTENER_DOCKER_INTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- shared
kafdrop:
image: obsidiandynamics/kafdrop:latest
restart: "no"
ports:
- "8089:9000"
environment:
KAFKA_BROKERCONNECT: "kafka1:19092"
depends_on:
- "kafka1"
networks:
- shared
presto:
image: ahanaio/prestodb-sandbox:latest
ports:
- 8087:8080
volumes:
- ./presto/kafka.properties:/etc/catalog/kafka.properties
networks:
- shared
networks:
shared:
name: kappa-playground
driver: bridge

挂载的文件kafka。属性包含以下内容:

connector.name=kafka
kafka.nodes=kafka1:19092
kafka.table-names=example_topic

我确保kafka用下面的小脚本创建主题:

# requires kafka-python
from kafka import KafkaClient
from kafka.admin import KafkaAdminClient, NewTopic
client = KafkaClient(bootstrap_servers='localhost:29092')
admin_client = KafkaAdminClient(
bootstrap_servers="localhost:29092", 
client_id='setup'
)
future = client.cluster.request_update()
client.poll(future=future)
metadata = client.cluster
topics = metadata.topics()
if(len(topics) > 0 ):
print("topics: " + " ".join(topics))
else:
print("no topics exist yet")
if("example_topic" not in topics):
topic_list = []
topic_list.append(NewTopic(name="example_topic", num_partitions=1, replication_factor=1))
admin_client.create_topics(new_topics=topic_list, validate_only=False)

我可以验证主题"example_topic"与kafdrop一起存在

现在我试着验证presto可以像这样从kafka准备主题:

presto --server=localhost:8087 --catalog kafka --schema default
presto:default> SHOW TABLES;

显示以下错误:

Query 20220622_080948_00005_t2k7a failed: line 1:1: Catalog 'kafka' does not exist

这里出了什么问题?

Answer 1

找到问题。卡夫卡。属性文件挂载到错误的路径。

应该是:

presto:
image: ahanaio/prestodb-sandbox:latest
ports:
- 8087:8080
volumes:
- ./presto/kafka.properties:/opt/presto-server/etc/catalog/kafka.properties
networks:
- shared