Kubernetes logs from kubectl logs命令与GKE集群的/var/log/container

Question

我已经在GCP上创建了一个GKE集群。

Kubernetes logs from kubectl logs命令不同于/var/log/containers

kubectl

{"method":"GET","path":"/healthz","format":"*/*","controller":"Public::PublicPagesController","action":"healthz","status":204,"duration":0.39,"view":0.0,"request_id":"ca29b519-d1e8-49a2-95ae-e5f23b60c36f","params":{},"custom":null,"request_time":"2022-04-27T15:25:43.780+00:00","process_id":6,"@version":"vcam-backend-vvcam-72_shareholder_event-rc16","@timestamp":"2022-04-27T15:25:43.780Z","message":"[204] GET /healthz (Public::PublicPagesController#healthz)"}

和日志在/var/log/containers，添加时间戳到我的容器日志的开始:

2022-04-27T15:25:43.780523421Z stdout F {"method":"GET","path":"/healthz","format":"*/*","controller":"Public::PublicPagesController","action":"healthz","status":204,"duration":0.39,"view":0.0,"request_id":"ca29b519-d1e8-49a2-95ae-e5f23b60c36f","params":{},"custom":null,"request_time":"2022-04-27T15:25:43.780+00:00","process_id":6,"@version":"vcam-backend-vvcam-72_shareholder_event-rc16","@timestamp":"2022-04-27T15:25:43.780Z","message":"[204] GET /healthz (Public::PublicPagesController#healthz)"}

我希望我的应用程序日志将是一致的，我希望它是json格式的日志从kubectl命令，所以我可以解析和分析更多。

我想删除这部分:2022-04-27T15:25:43.780523421Z stdout F

有人遇到这个问题吗?如何使容器日志与kubectl命令日志相同?

GKE版本:

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10-gke.2000", GitCommit:"0823380786b063c3f71d5e7c76826a972e30550d", GitTreeState:"clean", BuildDate:"2022-03-17T09:22:22Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

码头工人daemon.json

{
"pidfile": "/var/run/docker.pid",
"iptables": false,
"ip-masq": false,
"log-level": "warn",
"bip": "169.254.123.1/24",
"mtu": 1460,
"storage-driver": "overlay2",
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}

注意:我注意到日志行开头的时间戳只在我们添加选项docker logs -t时显示，文档在这里但是我仍然不知道如何在GKE集群中解决这个问题。

Answer 1

此问题与容器运行时接口(CRI)有关。你可以在这里阅读有关CRI的内容。

目前，我还不能随心所欲地改变日志格式，我只是适应了这种新的格式。这种格式称为CRI log format, GKE集群的CRI默认总是以这种格式生成日志，聚合日志应用程序也适应这种新的CRI日志格式:

• 这是Grafana/Loki支持CRI日志格式的PR: PR
• 为CRI日志格式创建一个新的解析器:Docs

所以我认为你需要改变处理这个问题的方式:如果我们不能随心所欲地改变日志格式，我们可以使用应用程序支持这种日志格式。

注意:我不确定，但我认为这个问题来自:Kubernetes is removing support for Docker as a container runtime- docs，新的容器运行时产生这个CRI日志格式。