__consumer_offsets主题内容的含义

Question

我正在使用kafka转储日志工具查看__consumer_offset主题的内容。有人能解释一下该内容的确切含义吗(offset、createtime、isvalid、keysize、valuesize、compresscodec、producterId、producterEpoch、isTransactional、headerKeys、key、offset、payload(？

[root@localhost kafka_2.11-2.0.0]# bin/kafka-dump-log.sh --files data/kafka/__consumer_offsets-32/00000000000000036360.log --value-decoder-class "kafka.serializer.StringDecoder" --offsets-decoder
Dumping data/kafka/__consumer_offsets-32/00000000000000036360.log
Starting offset: 36360
offset: 36360 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 0 isTransactional: false headerKeys: [] key: offset::b0147:test:48 payload: 35
offset: 36361 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 1 isTransactional: false headerKeys: [] key: offset::b0147:test:64 payload: 34
offset: 36362 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 2 isTransactional: false headerKeys: [] key: offset::b0147:test:14 payload: 34
...
offset: 36460 position: 5497 CreateTime: 1623678503139 isvalid: true keysize: 9 valuesize: 24 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: {"metadata":"b0147"} payload: {"protocolType":"consumer","protocol":null,"generationId":2,"assignment":"{}"}
offset: 36461 position: 5598 CreateTime: 1623678554656 isvalid: true keysize: 19 valuesize: -1 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: []
...

Answer 1

您看到的字段不是__consumer_offsets特有的，它们只是日志中每个消息/批次的元数据，您将获得所有主题的这些字段，请参阅https://kafka.apache.org/documentation/#recordbatch

如果你想探索__consumer_offsets的内容，你需要知道它包含两种类型的记录：

• 偏移元数据：这是关于组提交的偏移的信息
• GroupMetadata：这是有关使用者组成员的信息

要显示偏移元数据，可以使用：

./bin/kafka-console-consumer.sh 
--formatter "kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter" 
--bootstrap-server localhost:9092 --topic __consumer_offsets

要显示GroupMetadata，可以使用：

./bin/kafka-console-consumer.sh 
--formatter "kafka.coordinator.group.GroupMetadataManager$GroupMetadataMessageFormatter" 
--bootstrap-server localhost:9092 --topic __consumer_offsets