Apache Kafka 日志保留配置问题或如何配置 Kafka 保留策略?

Question

我正在使用kafka_2.12-1.1.0。以下是我的保留信息。我正在使用 Win OS。

log.retention.ms=120000
log.segment.bytes=2800
log.retention.check.interval.ms=30000

我看到日志被滚动并生成 *0000.log 到 *0069.log然后是 *00103.log 等，然后我希望它会删除旧文件，但是当保留逻辑启动时，我得到以下错误进程无法访问该文件，因为它正被另一个进程使用，并且 kafka 服务器崩溃并且无法从那里恢复。当它尝试删除 00000000000000000000000000000000000000000000000000000000000000000000000.log 0000000000000000000000000000您能否让我知道检查它们是否是活动段的命令，或者为什么它试图删除它并崩溃。

PFB 日志文件(如果要查看(。

[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 39
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Rolled new log segment at offset 39 in 39 ms. 
[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 69 
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Rolled new log segment at offset 69 in 52 ms.
[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 91
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Rolled new log segment at offset 91 in 18 ms. 
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Found deletable segments with base offsets [0,69] due to retention time 120000ms breach (kafka.log.Log)
[ProducerStateManager partition=myLocalTopic-0] Writing producer snapshot at offset 103 (kafka.log.ProducerStateManager)
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Rolled new log segment at offset 103 in 25 ms. (kafka.log.Log)
[Log partition=myLocalTopic-0, dir=D:tmpkafka-logs] Scheduling log segment [baseOffset 0, size 2746] for deletion. (kafka.log.Log)
ERROR Error while deleting segments for myLocalTopic-0 in dir D:tmpkafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: D:tmpkafka-logsmyLocalTopic-00000000000000000000.log -> D:tmpkafka-logsmyLocalTopic-00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
.............   at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:98)
.......................................................
ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for myLocalTopic-0 in dir D:tmpkafka-logs
Caused by: java.nio.file.FileSystemException: D:tmpkafka-logsmyLocalTopic-00000000000000000000.log -> D:tmpkafka-logsmyLocalTopic-00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

单击此处查看文件列表图像

Answer 1

Apache Kafka 为我们提供了以下保留策略

1. 基于时间的保留
2. 基于大小的保留

对于基于时间的保留，请尝试以下命令，

设置保留时间的命令

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1680000

删除主题级别保留时间的命令

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.ms

对于基于大小的保留，请尝试以下命令， 设置保留大小：

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.bytes=104857600

要删除，

./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic  --delete-config retention.bytes