由于我们在几个服务的流中引入了GlobalKTables,服务的启动时间变得难以忍受。我们有一个侦听器来观察状态存储恢复的状态,这就是日志中的内容:
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Subscribed to partition(s): wrwks-bef-equipmenttyp-aggregat-privat-1-4
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Seeking to offset 4 for partition wrwks-bef-equipmenttyp-aggregat-privat-1-4
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreStart for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-4
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - started state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, remaining partitions: 1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreEnd for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-4
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - finished state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Subscribed to partition(s): wrwks-bef-equipmenttyp-aggregat-privat-1-1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Seeking to offset 1 for partition wrwks-bef-equipmenttyp-aggregat-privat-1-1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreStart for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - started state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, remaining partitions: 1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreEnd for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - finished state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Subscribed to partition(s): wrwks-bef-equipmenttyp-aggregat-privat-1-6
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Seeking to offset 1 for partition wrwks-bef-equipmenttyp-aggregat-privat-1-6
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreStart for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-6
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - started state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, remaining partitions: 1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreEnd for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-6
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - finished state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Subscribed to partition(s): wrwks-bef-equipmenttyp-aggregat-privat-1-7
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Seeking to offset 1 for partition wrwks-bef-equipmenttyp-aggregat-privat-1-7
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreStart for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-7
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - started state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, remaining partitions: 1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreEnd for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-7
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - finished state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Subscribed to partition(s): wrwks-bef-equipmenttyp-aggregat-privat-1-3
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO o.a.k.clients.consumer.KafkaConsumer - [Consumer clientId=wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-global-consumer, groupId=null] Seeking to offset 4 for partition wrwks-bef-equipmenttyp-aggregat-privat-1-3
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] DEBUG a.w.b.p.h.StreamAndStateStoreStateListener - onRestoreStart for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, partition wrwks-bef-equipmenttyp-aggregat-privat-1-3
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - started state restore for store wrwks-bef-equipmenttyp-aggregat-privat-1-STATE-STORE-0000000005, remaining partitions: 1
[wrwks-bef-projekt-export-projektAggregate-5e88016d-ae03-49fe-80b0-46c2238f528d-GlobalStreamThread] INFO a.w.b.p.h.StreamAndStateStoreStateListener - publishing availability information: liveness false, readiness false
该主题由20条记录组成,如果完成,则引导时间为几分钟。在其他情况下,它根本没有完成,这导致k8s看门狗在等待5分钟后在我们的集群中重复重新启动。
一个非常恼人的解决方法是删除状态存储目录并重新启动服务,然后服务通常会在合理的时间内启动。
此行为适用于所有使用GlobalKTables的服务。所有服务都通过Spring Cloud Stream 2021.0.5使用Kafka Streams。
与此同时,服务(到Spring Cloud 2022.0.1与Kafka Streams 3.3.2)以及代理(Kafka 3.0.1)都已经升级,因此问题消失了。