我们在Kube pod上运行了Kafka连接。我们在工作日志中看到以下错误。我们试着重新启动吊舱。
[2020-04-02 14:40:13,237] WARN Aborting multi-part upload with id '0vbfuRZCRIkc431LJN.hyaPuo3ZQzuAsfTrMSdBE_Q9.sZP8g-' (io.confluent.connect.s3.storage.S3OutputStream)
[2020-04-02 14:40:13,274] ERROR Multipart upload failed to complete for bucket 'my_bucket' key '/myfile': (io.confluent.connect.s3.TopicPartitionWriter)
org.apache.kafka.connect.errors.DataException: Multipart upload failed to complete.
at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:159)
at io.confluent.connect.s3.format.avro.AvroRecordWriterProvider$1.commit(AvroRecordWriterProvider.java:96)
at io.confluent.connect.s3.TopicPartitionWriter.commitFile(TopicPartitionWriter.java:503)
at io.confluent.connect.s3.TopicPartitionWriter.commitFiles(TopicPartitionWriter.java:483)
at io.confluent.connect.s3.TopicPartitionWriter.commitOnTimeIfNoData(TopicPartitionWriter.java:294)
at io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:184)
at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:538)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Part upload failed:
at io.confluent.connect.s3.storage.S3OutputStream.uploadPart(S3OutputStream.java:136)
at io.confluent.connect.s3.storage.S3OutputStream.commit(S3OutputStream.java:153)
... 17 more
[2020-04-02 14:40:13,275] INFO Committing files after waiting for rotateIntervalMs time but less than flush.size records available. (io.confluent.connect.s3.TopicPartitionWriter)
在我的案例中，这是两件事：
- KMS加密已在bucket上启用，因此附加到连接器用户的策略需要访问有问题的KMS密钥；连接器需要以下KMS权限：
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
- 如果
"s3:x-amz-server-side-encryption": "false"，则在禁用s3:PutObject的帐户上启用SCP-删除该SCP解决了问题
此文档非常有用：https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/
DataException: Multipart upload failed to complete. java.io.IOException: Part upload failed
看起来与S3的连接中断了。重新启动连接器，并检查网络连接。
根本原因是缺少适当的KMS密钥策略。