使用EFK堆栈从kibana上的S3获取AWS日志

Question

我有一个在EKS上运行的EFK堆栈。我如何从S3 bucket中收集日志并通过Kibana查看它们？

Answer 1

您可以使用此插件为s3添加新的源代码https://github.com/tomohisaota/fluent-plugin-forward-aws

首先，您需要添加源配置

<source>
type forward_aws
aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX
aws_sqs_endpoint      sqs.ap-northeast-1.amazonaws.com
aws_sqs_queue_url     https://sqs.ap-northeast-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXX
</source>

然后你需要用match解析它们，并将它们转发给弹性搜索