如何从cp-enterprise-kafka:6.2.1迁移到cp-kafka:6.2.1 docker镜像?这对我来说

Question

当我在docker-compose中更改图像时。cp-enterprise-kafkacp-kafka

开头失败:

broker-1       | [2021-10-19 09:22:32,042] WARN [RequestSendThread controllerId=1] Controller 1 epoch 38 fails to send request (type: UpdateMetadataRequest=, controllerId=1, controllerEpoch=38, brokerEpoch=7362, partitionStates=[], liveBrokers=UpdateMetadataBroker(id=1, v0Host='', v0Port=0, endpoints=[UpdateMetadataEndpoint(port=19092, host='broker-1', listener='INSIDE', securityProtocol=0), UpdateMetadataEndpoint(port=9092, host='192.168.100.14', listener='OUTSIDE', securityProtocol=0)], rack=null)) to broker broker-1:19092 (id: 1 rack: null). Reconnecting to broker. (kafka.controller.RequestSendThread)
broker-1       | java.io.IOException: Client was shutdown before response was read
broker-1       |        at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:109)
broker-1       |        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:253)
broker-1       |        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
broker-1       | [2021-10-19 09:22:32,045] ERROR [RequestSendThread controllerId=1] Controller 1 fails to send a request to broker broker-1:19092 (id: 1 rack: null) (kafka.controller.RequestSendThread)
broker-1       | java.lang.InterruptedException
broker-1       |        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367)
broker-1       |        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
broker-1       |        at kafka.utils.ShutdownableThread.pause(ShutdownableThread.scala:82)
broker-1       |        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:234)
broker-1       |        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

我在docker-compose中的配置。yml:

version: '3'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:6.2.1
hostname: zookeeper-1
container_name: zookeeper-1
ports:
- "2181:2181"
- "2888:2888"
- "3888:3888"
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
- ./kafka-data/zookeeper-1:/var/lib/zookeeper/data
- ./kafka-data/zookeeper-logs-1:/var/lib/zookeeper/log
networks:
- mynet
broker-1:
image: confluentinc/cp-enterprise-kafka:6.2.1
hostname: broker-1
container_name: broker-1
depends_on:
- zookeeper-1
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: '192.168.100.14:2181'
KAFKA_LISTENERS: OUTSIDE://0.0.0.0:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: OUTSIDE://192.168.100.14:9092
KAFKA_INTER_BROKER_LISTENER_NAME: OUTSIDE
KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: 'LogAppendTime'
KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_CONNECTIONS_MAX_IDLE_MS: 31536000000 # 1 year
CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: 192.168.100.14:9092
CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: 192.168.100.14:2181
CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
CONFLUENT_METRICS_ENABLE: 'true'
CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
volumes:
- ./kafka-data/kafka-1:/var/lib/kafka/data
networks:
- mynet
networks:
mynet:
driver: bridge

。我只将broker-1的镜像从cp-enterprise-kafka更改为cp-kafka。

我注意到controllerEpoch比brokerEpoch小得多-也许这是一个原因?

任何想法?谢谢!

Answer 1

已解决(但不知道具体是如何解决的):

1. stop zookeeper and broker
2. 复制所有zookeeper数据(备份)
3. 启动zookeeper和broker并得到这个(错误的集群):

broker-1       | [2021-10-19 09:25:25,648] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
broker-1       | kafka.common.InconsistentClusterIdException: The Cluster ID MlRI76IVQ8y4nTZgigyUGw doesn't match stored clusterId Some(HXaFk1peQcaR23TAqh0Z2g) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
broker-1       |        at kafka.server.KafkaServer.startup(KafkaServer.scala:218)
broker-1       |        at kafka.Kafka$.main(Kafka.scala:109)
broker-1       |        at kafka.Kafka.main(Kafka.scala)

4. 已停止zookeeper和broker
5. 复制(先前备份)zookeeper数据返回
6. 开始……一切开始顺利!

也许甚至zookeeper数据备份根本不需要，只是更改代理的集群就可以了。/var/lib/kafka/data/meta.properties