使用Zookeeper-shall.sh rmr经纪/主题删除主题与在kafka10上的kafka-topics.sh上



从我在线发现的内容和其他堆栈溢出帖子中,似乎主要有两种删除Kafka上的主题的方法。第一个是:A)delete.topic.enable = true并运行./kafka-topics.sh ---delete --topic <topicName>第二种方法:./zookeeper-shell.sh localhost:2181 rmr brokers/topics

我确实注意到第一种方法标记了要删除的每个主题,并且在几分钟的时间内将其删除,因为第二种方法会立即删除它们。我还注意到,在重新启动服务器时,这需要数小时,这是正常的吗?我在一个经纪人上有1000多个主题(用于测试目的)。

第一个方法将在Zookeper admin/delete_topics/<topic>中创建一个节点,如果您像您一样启用了主题删除,则在KAFKA BROKER(topicDeletionManager)中的一个给定线程,该线程监视delete_topics CHILDS,将处理此操作,这是从Zookeper中删除的,但也从所有Kafka副本中登录,确保您不会处于无效状态。这里描述了整个过程:https://github.com/apache/kafka/kafka/blob/0.11.0/core/src/main/scala/kafka/kafka/controller/topicdeletionmanager.scala

/**
 * This manages the state machine for topic deletion.
 * 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic>
 * 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
 * 3. The controller's ControllerEventThread handles topic deletion. A topic will be ineligible
 *    for deletion in the following scenarios -
  *   3.1 broker hosting one of the replicas for that topic goes down
  *   3.2 partition reassignment for partitions of that topic is in progress
 * 4. Topic deletion is resumed when -
 *    4.1 broker hosting one of the replicas for that topic is started
 *    4.2 partition reassignment for partitions of that topic completes
 * 5. Every replica for a topic being deleted is in either of the 3 states -
 *    5.1 TopicDeletionStarted Replica enters TopicDeletionStarted phase when onPartitionDeletion is invoked.
 *        This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state
 *        change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the
 *        StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica
 *        is received from every replica)
 *    5.2 TopicDeletionSuccessful moves replicas from
 *        TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse
 *    5.3 TopicDeletionFailed moves replicas from
 *        TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse.
 *        In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the
 *        respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a
 *        broker fails before the request is sent and after the replica is in TopicDeletionStarted state,
 *        it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion
 *        will not be retried when the broker comes back up.
 * 6. A topic is marked successfully deleted only if all replicas are in TopicDeletionSuccessful
 *    state. Topic deletion teardown mode deletes all topic state from the controllerContext
 *    as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand,
 *    if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then
 *    it marks the topic for deletion retry.

直接从Zookeeper中删除只是意味着从编目中删除。当然,当请求元数据时,主题不再在这里(嗯,也许可以从缓存中),但是没有删除日志文件(至少现在不是现在,我认为经纪人会检测到日志无效并在某个时候删除它们),但是您可能对经纪人有一些不连贯的意义(如果您在重新平衡的中间,您可能会损害很多事情)。这可能意味着有些经纪人会认为它已删除,而另一些则认为它仍然存在……远非理想。

删除FOM Zookeeper(并从经纪人那里的日志)似乎确实可能是可能的,但是要当心,它可能会发生冲突,无效状态,随机错误,并且在将来的版本中可能根本不起作用。

最新更新