Kafka JDBC sink连接器没有访问正确的连接器配置

Question

我有一个postgres运行在pi在一个docker容器。Debezium连接器在我的本地机器上运行(与zookeeper和kafka相同)。

kafka主题已经启动并运行，我可以看到我在postgres中对kafka主题所做的更改。到目前为止一切顺利。

现在我在本地启动了另一个docker容器，它不是来自与我的其他容器相同的docker组成文件。这应该是我的副本数据库。

我把confluence -kafka-connect-jdbc-10.5.0拷贝到docker容器中。

sudo docker cp confluentinc-kafka-connect-jdbc-10.5.0 CONTAINER_ID:/kafka/connect/ 

修改用户和用户组，并重新定义容器。

docker exec -it --user root <container-id> /bin/bash
chown -R <username>:<groupname> <folder/file>

现在我创建了jdbc-sink连接器。

curl --location --request POST 'http://localhost:8083/connectors/' 
--header 'Accept: application/json' 
--header 'Content-Type: application/json' 
--data-raw '{
"name": "jdbc-sink",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "piserver.public.customers",
"connection.url": "jdbc:postgresql:192.168.128.2:5432/postgres",
"connection.user": "postgres",
"connection.password": "postgres",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",   
"auto.create": "true",
"insert.mode": "upsert",
"pk.fields": "id",
"pk.mode": "record_value"                                               
}
}
'

我得到201创建。

我得到的错误是运行几秒钟后:

curl --location --request GET 'localhost:8083/connectors/jdbc-sink/status' 
--data-raw ''

<<p>错误跟踪/em>

{
"id": 0,
"state": "FAILED",
"worker_id": "192.168.112.4:8083",
"trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.ntat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:611)ntat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:333)ntat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:234)ntat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)ntat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)ntat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)ntat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)ntat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)ntat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)ntat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)ntat java.base/java.lang.Thread.run(Thread.java:829)nCaused by: org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.ntat io.confluent.connect.jdbc.util.CachedConnectionProvider.getConnection(CachedConnectionProvider.java:59)ntat io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:64)ntat io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:84)ntat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581)nt... 10 morenCaused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.ntat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:319)ntat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)ntat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)ntat org.postgresql.Driver.makeConnection(Driver.java:400)ntat org.postgresql.Driver.connect(Driver.java:259)ntat java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)ntat java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)ntat io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.getConnection(GenericDatabaseDialect.java:250)ntat io.confluent.connect.jdbc.dialect.PostgreSqlDatabaseDialect.getConnection(PostgreSqlDatabaseDialect.java:103)ntat io.confluent.connect.jdbc.util.CachedConnectionProvider.newConnection(CachedConnectionProvider.java:80)ntat io.confluent.connect.jdbc.util.CachedConnectionProvider.getConnection(CachedConnectionProvider.java:52)nt... 13 morenCaused by: java.net.ConnectException: Connection refused (Connection refused)ntat java.base/java.net.PlainSocketImpl.socketConnect(Native Method)ntat java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)ntat java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)ntat java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)ntat java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)ntat java.base/java.net.Socket.connect(Socket.java:609)ntat org.postgresql.core.PGStream.createSocket(PGStream.java:241)ntat org.postgresql.core.PGStream.<init>(PGStream.java:98)ntat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:109)ntat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)nt... 23 moren"
}

短:Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

我在配置中尝试的主机:

"connection.url": "jdbc:postgresql:192.168.128.2:5432/postgres" // got this IP from docker inspect POSTGRES_CONTAINER
"connection.url": "jdbc:postgresql:host.docker.internal:5432/postgres" 
"connection.url": "jdbc:postgresql:localhost:5432/postgres" 

这些都不起作用

总是得到与cannot access localhost:5432相同的错误。

还尝试连接docker postgres容器(副本)到我的docker-compose网络。

有什么想法吗?

小简历。

POSTGRES(on PI)->DEBEZIUM Connector(local)——>JDBC-SINK从KAFKA ->POSTGRES(将是副本，在本地运行)

Answer 1

不要在容器之间使用IP地址，也不要在容器内使用localhost试图访问其他容器- https://docs.docker.com/network/bridge/

理想情况下，你应该使用Docker Compose启动所有服务，否则您需要自己创建网络桥接

docker network create database-bridge
docker run --network=database-bridge --name=postgres ...  
docker run --network=database-bridge  ... # repeat for zookeeper, kafka, and debezium

或者查看组成创建的网络，并将新容器附加到其中，因为您说

在本地启动另一个docker容器，该容器不是来自同一个docker组合文件

docker network ls  # look for a name that matches the folder where you ran docker-compose
docker run --network=<name> ... jdbc-connector

然后使用jdbc:postgresql://postgres:5432/postgres通过主机名连接到该容器。

如果JDBC连接器与connect-distributed.sh而不是Docker一起运行，只有这样你才能使用localhost:5432，但是你需要一个从Postgres容器到主机的端口映射。