我正在尝试从kubernetes pods收集日志，并使用logstash和filebeats将其传递给elastic-db。我可以在k8集群中部署Elasticsearch和kibana，但无法部署我的logstash和filebeat。
我的elasticisearch.yaml文件是
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 7.14.1 #Make sure you use the version of your choice
http:
service:
spec:
type: LoadBalancer #Adds a External IP
nodeSets:
name: default
count: 1
config:
node.store.allow_mmap: false
Logstash.yaml文件是
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-configmap
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
logstash.conf: |
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "https://External ip:9200"
index =>"testlogs"
user => "elastic"
password => "xxxx"
cacert => "/etc/"
ssl_certificate_verification => false
}
}
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash-deployment
spec:
replicas: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.14.1
ports:
- containerPort: 5044
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/pipeline
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: logstash.conf
path: logstash.conf
kind: Service
apiVersion: v1
metadata:
name: logstash-service
spec:
selector:
app: logstash
ports:
- protocol: TCP
port: 5044
targetPort: 5044
在部署我的logstash pod后崩溃并得到这个错误
[ERROR] 2022-10-21 08:54:16.937 [[main]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<Errno::EISDIR: Is a directory - channel: org.jruby.util.io.ChannelFD@2f88e4ba /etc/>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:642:in block in setup_trust_store'", "org/jruby/RubyIO.java:1158:in open'", "org/jruby/RubyKernel.java:317:in open'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/open-uri.rb:37:in open'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:641:in setup_trust_store'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:629:in ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:396:in pool_builder'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:404:in pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/client.rb:207:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:319:in build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:323:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:62:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.0.3-java/lib/logstash/outputs/elasticsearch.rb:275:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1820:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5b2f9d8f@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2022-10-21 08:54:16.940 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[ERROR] 2022-10-21 08:54:16.957 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
是因为我提到的ca证书路径吗？如果是的话，有人能分享提到ca证书的正确方式吗？我在k8集群中的哪里可以找到它。提前感谢！！
如果您查看Logstash的Elasticsearch输出插件的文档，ca_cert应该是file_system上的有效文件，而不是目录。
您目前正在使用ECK来管理Elasticsearch和Kibana。ECK为您创建了几个kubernetes秘密，包括Elasticsearch的certificate_authority。
您可以使用命令kubectl get secret获得机密列表，并搜索以您的Elasticsearch Cluster名称(即elasticsearch(命名并带有后缀"的机密-远程ca"；，在您的情况下，它应该是elasticsearch-remote-ca。
您可以使用kubectl get secret elasticsearch-remote-ca -o json检查机密的内容。您对json路径data > ca.crt感兴趣。这个字符串是base64编码的，因此要获得有效的证书，您应该对其进行解码
幸运的是，kubectl还支持go模板作为输出格式，这使我们能够编写这个单行命令来提取密钥.data.ca.crt并对其进行解码，而无需外部依赖：
kubectl get secret tiny-es-remote-ca -o go-template='{{index .data "ca.crt" | base64decode}}'
这将输出一个有效的ca证书，类似于：
-----BEGIN CERTIFICATE-----
MIIDOzCCA....
-----END CERTIFICATE-----
我们现在可以将证书复制到名为ca.pem.的文件中
现在，我们可以在pod中装载这个文件，并在配置中使用它。