从我在线发现的内容和其他堆栈溢出帖子中,似乎主要有两种删除Kafka上的主题的方法。第一个是:A)delete.topic.enable = true
并运行./kafka-topics.sh ---delete --topic <topicName>
第二种方法:./zookeeper-shell.sh localhost:2181 rmr brokers/topics
我确实注意到第一种方法标记了要删除的每个主题,并且在几分钟的时间内将其删除,因为第二种方法会立即删除它们。我还注意到,在重新启动服务器时,这需要数小时,这是正常的吗?我在一个经纪人上有1000多个主题(用于测试目的)。
第一个方法将在Zookeper admin/delete_topics/<topic>
中创建一个节点,如果您像您一样启用了主题删除,则在KAFKA BROKER(topicDeletionManager)中的一个给定线程,该线程监视delete_topics
CHILDS,将处理此操作,这是从Zookeper中删除的,但也从所有Kafka副本中登录,确保您不会处于无效状态。这里描述了整个过程:https://github.com/apache/kafka/kafka/blob/0.11.0/core/src/main/scala/kafka/kafka/controller/topicdeletionmanager.scala
/**
* This manages the state machine for topic deletion.
* 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic>
* 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
* 3. The controller's ControllerEventThread handles topic deletion. A topic will be ineligible
* for deletion in the following scenarios -
* 3.1 broker hosting one of the replicas for that topic goes down
* 3.2 partition reassignment for partitions of that topic is in progress
* 4. Topic deletion is resumed when -
* 4.1 broker hosting one of the replicas for that topic is started
* 4.2 partition reassignment for partitions of that topic completes
* 5. Every replica for a topic being deleted is in either of the 3 states -
* 5.1 TopicDeletionStarted Replica enters TopicDeletionStarted phase when onPartitionDeletion is invoked.
* This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state
* change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the
* StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica
* is received from every replica)
* 5.2 TopicDeletionSuccessful moves replicas from
* TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse
* 5.3 TopicDeletionFailed moves replicas from
* TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse.
* In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the
* respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a
* broker fails before the request is sent and after the replica is in TopicDeletionStarted state,
* it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion
* will not be retried when the broker comes back up.
* 6. A topic is marked successfully deleted only if all replicas are in TopicDeletionSuccessful
* state. Topic deletion teardown mode deletes all topic state from the controllerContext
* as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand,
* if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then
* it marks the topic for deletion retry.
直接从Zookeeper中删除只是意味着从编目中删除。当然,当请求元数据时,主题不再在这里(嗯,也许可以从缓存中),但是没有删除日志文件(至少现在不是现在,我认为经纪人会检测到日志无效并在某个时候删除它们),但是您可能对经纪人有一些不连贯的意义(如果您在重新平衡的中间,您可能会损害很多事情)。这可能意味着有些经纪人会认为它已删除,而另一些则认为它仍然存在……远非理想。
删除FOM Zookeeper(并从经纪人那里的日志)似乎确实可能是可能的,但是要当心,它可能会发生冲突,无效状态,随机错误,并且在将来的版本中可能根本不起作用。