如何在Flink中减少或禁用检查点日志



我正在使用Flink 1.11.1,并在Kubernetes上以独立模式运行它,并使用hdfs进行存储和HA。这些天,我尝试启用Flink检查点功能。但我注意到jobmanager和taskmanager都记录了太多与检查点相关的日志,这很烦人。示例如下:

作业经理

2020-10-08 19:54:23,237 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 1 (type=CHECKPOINT) @ 1602186863226 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:42,818 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 1 for job fbf26a33a6d5d235085d10e7a10c1cab (702534 bytes in 19488 ms).
2020-10-08 19:54:42,825 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 2 (type=CHECKPOINT) @ 1602186882820 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:43,384 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 2 for job fbf26a33a6d5d235085d10e7a10c1cab (729357 bytes in 494 ms).
2020-10-08 19:54:43,392 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 3 (type=CHECKPOINT) @ 1602186883388 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:44,295 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 3 for job fbf26a33a6d5d235085d10e7a10c1cab (736969 bytes in 836 ms).
2020-10-08 19:54:44,302 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 4 (type=CHECKPOINT) @ 1602186884298 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:44,794 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 4 for job fbf26a33a6d5d235085d10e7a10c1cab (748787 bytes in 431 ms).
2020-10-08 19:54:44,800 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 5 (type=CHECKPOINT) @ 1602186884796 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:45,198 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 5 for job fbf26a33a6d5d235085d10e7a10c1cab (755308 bytes in 327 ms).
2020-10-08 19:54:45,703 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 6 (type=CHECKPOINT) @ 1602186885698 for job fbf26a33a6d5d235085d10e7a10c1cab.
2020-10-08 19:54:45,897 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Completed checkpoint 6 for job fbf26a33a6d5d235085d10e7a10c1cab (757353 bytes in 163 ms).
2020-10-08 19:54:45,903 [INFO] org.apache.flink.runtime.checkpoint.CheckpointCoordinator    - Triggering checkpoint 7 (type=CHECKPOINT) @ 1602186885899 for job fbf26a33a6d5d235085d10e7a10c1cab.

任务管理器

2020-10-08 20:04:02,090 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 571 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187440992} from checkpoint 571
2020-10-08 20:04:03,086 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 572 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187441992} from checkpoint 572
2020-10-08 20:04:03,086 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 572 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187441992} from checkpoint 572
2020-10-08 20:04:04,099 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 573 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187442992} from checkpoint 573
2020-10-08 20:04:04,099 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 573 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187442992} from checkpoint 573
2020-10-08 20:04:05,130 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 574 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187443999} from checkpoint 574
2020-10-08 20:04:05,130 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 574 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187443999} from checkpoint 574
2020-10-08 20:04:06,096 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 575 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187444995} from checkpoint 575
2020-10-08 20:04:06,096 [INFO] org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction - FlinkKafkaProducer 1/2 - checkpoint 575 complete, committing transaction TransactionHolder{handle=KafkaTransactionState [transactionalId=null, producerId=-1, epoch=-1], transactionStartTime=1602187444992} from checkpoint 575

有什么方法可以禁用或减少来自检查点的日志吗?任何帮助都将不胜感激!

禁用恼人日志的最简单方法是为目标组件指定所需的日志级别。在您的情况下,如果您想禁用来自org.apache.flink.runtime.checkpoint的日志,或者更广泛地禁用来自所有flink组件org.apache.flink的日志,那么您可以将其日志级别提高到WARN。为此,编辑flink/conf/log4j.properties文件并添加以下内容(或取消注释现有行(:
logger.flink.name = org.apache.flink
logger.flink.level = WARN

在应用程序停止/启动后,将获取更改。

相关内容

  • 没有找到相关文章

最新更新