卡夫卡抵消滞后远远落后于原因



我们一直在尝试修复不想使用数据的消费者。它每4秒轮询一次,通常只收到大约100条记录。最大拉力约为18000。然后它将原始字节写入文件。所有快速操作。

但消费者团体电话显示以下

[xyz@AZ254PP01 kafka]$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --new-consumer --group postproc-red2
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
postproc-red2, targeting-seg, 0, 21921226, 21921388, 162, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 0, 72891022, 78349037, 5458015, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 0, 16665679, 18122272, 1456593, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 0, 50847858, 56468186, 5620328, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 1, 20733636, 24858954, 4125318, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 1, 75084847, 89556283, 14471436, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 1, 19648100, 19648100, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 1, 66549015, 66549015, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-seg, 2, 24903106, 28041684, 3138578, consumer-2_/192.168.201.121
postproc-red2, targeting-visit, 2, 99480656, 99480656, 0, consumer-2_/192.168.201.121
postproc-red2, targeting-vis, 2, 19982077, 24848188, 4866111, consumer-2_/192.168.201.121
postproc-red2, targeting-vc, 2, 69611742, 76253273, 6641531, consumer-2_/192.168.201.121

当它记录时,它似乎只想从targeting-seg-0消耗,但一次只消耗60-90条记录。它有时会同时从所有0个分区中提取几千个。这些数据毫无意义,当涉及到其他问题时,我们也不知所措。它只是一个消费者。一个进程,它会消耗这些主题的所有分区。

最近两天的生产者似乎只写入了0个分区。我们也不知道原因。

我试着看看在所有超过60秒没有被拉出的分区上seekToEnd是否有效。

14:58:33.896 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3263 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-1
14:58:38.005 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vc-2
14:58:38.005 [Kafka-0] INFO  c.t.e.p.s.red2.ConsumerService - Processing records 3284 of Topics [targeting-seg-0, targeting-visit-0, targeting-vis-0, targeting-vc-0]
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-seg-2
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-visit-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-1
14:58:42.117 [Kafka-0] INFO  com.twyn.service.kafka.KafkaService - Initiated Seek to END for targeting-vis-2

代码已订阅(主题)。投票(500)。请参见结束。然后我希望在下一次民意调查中,seekToEnd能解决一个有问题的越位委员会。

任何关于卡夫卡设置的问题或如何解决的想法。它几乎在默认设置下运行。没有一个分区具有复制副本->复制因子1。我们打算切换到复制因子3,因为生产者是三台服务器。

Producer根据要"生成"的消息的键的哈希值选择目标分区。您应该检查最近两天的邮件是否始终具有相同的密钥。

在消费者方面,你确定postdoc-red2只用于这个消费者吗?您确定在同一组中具有唯一的消费者id吗?悬停的最佳实践是使使用者编号等于源主题的分区编号。

希望这能帮助

最新更新