我查看了github的错误，发现了一些相似但不同的东西，我真的很困惑如何用安全启用的Elasticsearch流实现FluentD。
我一直得到的错误是：
2022-04-08 15:39:03+0000[warn]：#0无法与Elasticsearch通信，正在重置连接并重试。连接被拒绝-0.0.0.0:9200的连接(2((错误号：：ECONNREFUSED(
我也尝试过将fluentd中的主机"弹性搜索"，想到了dns。同样的问题。
如果用户选择帮助调试这个问题，我将附上一堆信息，供他们用来构建自己的容器，但其中大部分都与在弹性搜索下创建的已发布的docker图像有关。
对于文件创建，我有以下内容：
docker-compose.yaml
fluentd/
Dockerfile
log/
conf/
fluent.conf
对于Dockerfile，我使用的基本样本是fluent用户，但由于我需要能够访问证书，我将其设置为root，因为证书所有权是为root级别的用户。
FluentDockfile:
FROM fluent/fluentd:v1.12-debian-1
# Use root account to use apt
USER root
# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev"
&& apt-get update
&& apt-get install -y --no-install-recommends $buildDeps
&& sudo gem install fluent-plugin-secure-forward fluent-plugin-elasticsearch
&& sudo gem sources --clear-all
&& SUDO_FORCE_REMOVE=yes
apt-get purge -y --auto-remove
-o APT::AutoRemove::RecommendsImportant=false
$buildDeps
&& rm -rf /var/lib/apt/lists/*
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
COPY fluent.conf /fluentd/etc/fluent.conf
COPY entrypoint.sh /bin/.
# USER fluent
USER 0
ENTRYPOINT ["tini", "--", "/bin/entrypoint.sh"]
CMD ["fluentd"]
对于我创建的配置文件，它可能是我的一个问题孩子所在的位置，将如下所示。我有一部分认为其中一个问题与创建的证书有关，但No on确实向我解释了很多
# fluentd/conf/fluent.conf
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
@type copy
<store>
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
flush_mode interval
user elastic
password foobar
ssl_verify true
ca_file /usr/share/fluentd/certs/ca/ca.crt
client_crt /usr/share/fluentd/certs/fluentd/fluentd.crt
client_key /usr/share/fluentd/certs/fluentd/fluentd.key
</store>
<store>
@type stdout
</store>
</match>
现在，docker编写了将所有内容连接在一起的文件。这几乎是一个精简版的弹性暴露。他们最初做了一个带有1个kibana实例的3搜索集群，当我将其转换为带有kibana和一个fluentd实例的单节点集群时。
version: "3"
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f certs/certs.zip ]; then
echo "Creating certs";
echo -ne
"instances:n"
" - name: elasticsearchn"
" dns:n"
" - elasticsearchn"
" - localhostn"
" ip:n"
" - 127.0.0.1n"
" - name: fluentdn"
" dns:n"
" - fluentdn"
" - localhostn"
" ip:n"
" - 127.0.0.1n"
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 {} ;;
find . -type f -exec chmod 640 {} ;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://elasticsearch:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://elasticsearch:9200/_security/user/kibana_system/_password -d "{"password":"${KIBANA_PASSWORD}"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/elasticsearch/elasticsearch.crt ]"]
interval: 1s
timeout: 5s
retries: 120
elasticsearch:
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata01:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=elasticsearch
- cluster.name=${CLUSTER_NAME}
- discovery.type=single-node
# IF i want to switch from a single node cluster to a cluster of 2+, remove the above discovery.type, and implement the below.
# - cluster.initial_master_nodes=elasticsearch,es02,es03
# - discovery.seed_hosts=es02,es03
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
elasticsearch:
condition: service_healthy
build: fluentd/.
image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
volumes:
- certs:/usr/share/kibana/config/certs
- kibanadata:/usr/share/kibana/data
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://elasticsearch:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
mem_limit: ${MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
fluentd:
depends_on:
elasticsearch:
condition: service_healthy
build: fluentd/.
volumes:
- certs:/usr/share/fluentd/certs
- ./fluentd/log:/fluentd/log
- "./fluentd/conf:/fluentd/etc"
ports:
- "24224:24224"
- "24224:24224/udp"
volumes:
certs:
driver: local
esdata01:
driver: local
kibanadata:
driver: local
我只是不知道如何让它在功能上与宿主进行弹性即时对话以供摄入。我想这可能与证书有关，但我真的不确定。即使试图从中删除ALL Security，它仍然无法连接到弹性，认为可能存在我不太清楚的更深层次的潜在问题。
我还更新了conf文件，看看dns是否能够解析，所以我把主机从0.0.0.0改为elasticsearch，改为localhost，所有这些我都认为可能有效，但都没有。仍返回ECONNREFUSED错误。
我开始从Fluentd容器中卷曲，看看我能找到什么，以及我是否能嗅出更多信息。
apt-get update
apt-get install -y curl
curl -cacert /usr/share/fluentd/certs/ca/ca.crt https://elasticsearch:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}},"status":401}
因此，似乎至少有一个连接响应：
｛"错误"｛>root_cause"安全性"，"ApiKey"]}]，"；类型"："security_exception"原因"："REST请求[/]的认证证书丢失"标题"：{"WWW身份验证"：["基本领域="安全性"字符集="UTF-8"，"承载领域="安全性"，"ApiKey"]}}，"；状态"：401}
所以这让我认为我确实需要基本身份验证的用户/密码才能登录，所以当将其添加到Fluentd的Conf文件中时，它似乎确实有效。我在想，这些证书已经足够握手了。
出于某种原因，我的逻辑是，有了证书，我不需要在conf中添加用户/密码。考虑到我看到有使用bin/elasticsearch-certutil的示例，我可以专门传入为FluentD生成的证书，它将遵循流程。但我想我一定错了。