如何声明多个输出.在kubernetes的单个文件beat DaemonSet日志存储?

Question

我有两个应用程序(Application1, Application2)在Kubernetes集群上运行。我想从Kubernetes集群外部收集我的应用程序的日志，并将它们保存在不同的目录中(例如:/var/log/application1/application1-YYYYMMDD.log和/var/log/application2/application2-YYYYMMDD.log)。

因此，我在Kubernetes集群上部署了一个filebeat DaemonSet来从我的应用程序(Application1, Application2)中获取日志，并在我想保存日志文件的实例上运行logstash服务(在Kubernetes集群之外)。

我创建了2 filebeat.yml(filebeat-application1)。然后将这两个文件作为参数提供给DaemonSet(docker.elastic.co/beats/filebeat:7.10.1)，如下所示。

....
  - name: filebeat-application1
    image: docker.elastic.co/beats/filebeat:7.10.1
    args: [
      "-c", "/etc/filebeat-application1.yml",
      "-c", "/etc/filebeat-application2.yml",
      "-e",
    ]
.....

但是只有/etc/filebeat-application2。Yml受影响。因此，我只从application2获取日志。

你能帮我关于如何喂养两个file_beat配置文件到docker.elastic。有限公司/比/filebeat DaemonSet吗?或者如何配置两个"filebeat.autodiscovery:"2个单独的"output.logstash:"?

下面是我的完整文件beat-kubernetes-whatsapp.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config-application1
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat-application1.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application1"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.pod.name}*.log
    processors:
      - add_locale:
          format: offset
      - add_kubernetes_metadata:
    output.logstash:
      hosts: ["IP:5045"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config-application2
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat-application2.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application2"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.pod.name}*.log
    processors:
      - add_locale:
          format: offset
      - add_kubernetes_metadata:
    output.logstash:
      hosts: ["IP:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat-application1
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          "-c", "/etc/filebeat-application1.yml",
          "-c", "/etc/filebeat-application2.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config-application1
          mountPath: /etc/filebeat-application1.yml
          readOnly: true
          subPath: filebeat-application1.yml
        - name: config-application2
          mountPath: /etc/filebeat-application2.yml
          readOnly: true
          subPath: filebeat-application2.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config-application1
        configMap:
          defaultMode: 0640
          name: filebeat-config-application1
      - name: config-application2
        configMap:
          defaultMode: 0640
          name: filebeat-config-application2
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---

Answer 1

不可能，filebeat只支持一个输出。

从文档

只能定义一个输出。

您需要将日志发送到相同的logstash实例，并根据某些字段过滤输出。

例如，假设您有发送到logstash的事件中的字段kubernetes.pod.name，您可以使用这样的内容。

output {
    if [kubernetes][pod][name] == "application1" {
        your output for the application1 log
    }
    if [kubernetes][pod][name] == "application2" {
        your output for the application2 log
    }
}

Answer 2

我找到了解决问题的方法。也许这不是正确的方法，但它可以满足我的要求。

filebeat-kubernetes-whatsapp.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition:
                equals:
                  kubernetes.namespace: default
            - condition:
                contains:
                  kubernetes.pod.name: "application1"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
            - condition:
                contains:
                  kubernetes.pod.name: "application2"
              config:
                - type: container
                  paths:
                    - /var/log/containers/*${data.kubernetes.container.id}*.log
    processors:
      - add_locale:
            format: offset
      - add_kubernetes_metadata:
            host: ${NODE_NAME} 
            matchers:
              - logs_path:
                  logs_path: "/var/log/containers/"
    output.logstash:
      hosts: ["IP:5044"]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
  labels:
    k8s-app: filebeat
---

/etc/logstash/conf.d/config.conf

input {
  beats {
    port => 5044
  }
}
    
#filter {
#  ...
#}
output {
  if "application1" in [kubernetes][pod][name] {
    file {
      enable_metric => false
      gzip => false 
      codec => line { format => "[%{[@timestamp]}] [%{[kubernetes][node][name]}/%{[kubernetes][pod][name]}/%{[kubernetes][pod][uid]}] [%{message}]"}
      path => "/abc/def/logs/application1%{+YYYY-MM-dd}.log"
    }
  }
  if "application2" in [kubernetes][pod][name] {
    file {
      enable_metric => false
      gzip => false 
      codec => line { format => "[%{[@timestamp]}] [%{[kubernetes][node][name]}/%{[kubernetes][pod][name]}/%{[kubernetes][pod][uid]}] [%{message}]"}
      path => "/abc/def/logs/application2%{+YYYY-MM-dd}.log"
    }
  }
}