- 您使用的是什么Grafana版本和什么操作系统?
Promtail:最新,Loki:2.2.0, Kubernetes(版本号:v1.18.8)和Helm(版本号:v3.6.2)
什么- 你在努力实现目标吗?
可以抓取我的活动目标并将它们推送给洛基。
- 发生了什么事?
所有目标都标记为"未准备好"。如果我要去/targets页面，我所有的active_targets都被标记为"false"。在洛基我没有日志。在Promtail中/var/logs/文件夹也是空的。我从promtail pod接收的日志如下所示:
level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg="Adding target" key="{app="<app_name>", container="<container_name>", job="<job_name>", namespace="<namesapce_name>", node_name="<nodeName_name>", pod="<pod_name>"}"
level=info ts=2021-08-06T05:30:09.076046169Z caller=filetargetmanager.go:254 msg="Removing target" key="{app="<app_name>", container="<container_name>", job="<job_name>", namespace="<namesapce_name>", node_name="<nodeName_name>", pod="<pod_name>"}"
level=info ts=2021-08-06T05:30:14.095615824Z caller=filetarget.go:150 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/pods/*<some_path>/<container_name>/*.log
Promtail/指标:
HELP promtail_targets_failed_total Number of failed targets.
TYPE promtail_targets_failed_total counter
promtail_targets_failed_total{reason="empty_labels"} 2280
promtail_targets_failed_total{reason="exists"} 470
HELP request_duration_seconds Time (in seconds) spent serving HTTP requests.
- 你想要什么怎样呢?
我的目标被刮掉了，推给了洛基。
- 你能复制/粘贴你有问题的配置吗?
file: |
server:
log_level: {{ .Values.config.logLevel }}
http_listen_port: {{ .Values.config.serverPort }}
health_check_target: false
client:
url: {{ tpl .Values.config.lokiAddress . }}
{{- tpl .Values.config.snippets.extraClientConfigs . | nindent 2 }}
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
{{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
{{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}
scrapeConfigs: |
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
# Pods with a label 'app.kubernetes.io/name'
- job_name: kubernetes-pods-app-kubernetes-io-name
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
target_label: app
- action: drop
regex: ''
source_labels:
- app
- action: replace
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
target_label: component
{{- if .Values.config.snippets.addScrapeJobLabel }}
- action: replace
replacement: kubernetes-pods-app-kubernetes-io-name
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
# Pods with a label 'app'
- job_name: kubernetes-pods-app
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Drop pods with label 'app.kubernetes.io/name'. They are already considered above
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: replace
source_labels:
- __meta_kubernetes_pod_label_app
target_label: app
- action: drop
regex: ''
source_labels:
- app
- action: replace
source_labels:
- __meta_kubernetes_pod_label_component
target_label: component
{{- if .Values.config.snippets.addScrapeJobLabel }}
- action: replace
replacement: kubernetes-pods-app
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
# Pods with direct controllers, such as StatefulSet
- job_name: kubernetes-pods-direct-controllers
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Drop pods with label 'app.kubernetes.io/name' or 'app'. They are already considered above
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- action: drop
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- action: replace
source_labels:
- __meta_kubernetes_pod_controller_name
target_label: app
{{- if .Values.config.snippets.addScrapeJobLabel }}
- action: replace
replacement: kubernetes-pods-direct-controllers
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
# Pods with indirect controllers, such as Deployment
- job_name: kubernetes-pods-indirect-controller
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Drop pods with label 'app.kubernetes.io/name' or 'app'. They are already considered above
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- action: keep
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- action: replace
regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
target_label: app
{{- if .Values.config.snippets.addScrapeJobLabel }}
- action: replace
replacement: kubernetes-pods-indirect-controller
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
# All remaining pods not yet covered
- job_name: kubernetes-other
pipeline_stages:
{{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Drop what has already been covered
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_controller_name
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: app
- action: replace
source_labels:
- __meta_kubernetes_pod_label_component
target_label: component
{{- if .Values.config.snippets.addScrapeJobLabel }}
- action: replace
replacement: kubernetes-other
target_label: scrape_job
{{- end }}
{{- toYaml .Values.config.snippets.common | nindent 4 }}
您是否收到Grafana UI或相关日志中的任何错误?如果是，请告诉我们确切的
您是否遵循任何在线指示?如果是，URL是什么?
我基本上遵循了官方回购的指示。
https://github.com/grafana/helm-charts/tree/main/charts
我创建了以下资源:
洛基:我有一个秘密(与配置)，服务和状态集。
Promtail:我有一个守护进程，秘密，强大的ClusterRole和CluserRoleBinding。
我不确定这是否会对您有所帮助，但是我有添加目标并立即删除它们的相同提示日志。
我的配置有点不同，我在本地运行promtail并抓取一些文件。问题是promtail没有读取这些文件的访问权限。
所以我建议你仔细检查一下你的promtail pod是否有读权限访问你试图抓取的文件，然后重新启动服务。
我也没有在grafana或Loki中看到任何错误，因为日志从未被推送到Loki。
my promtail_error .log
level=info ts=2023-08-09T02:26:00.468678877Z caller=filetargetmanager.go:255 msg="Adding target" key="{env="test", hostname="10.0.12.9", log_type="var-log-messages", service_name="var-log-messages"}"
level=info ts=2023-08-09T02:26:00.468989593Z caller=filetargetmanager.go:255 msg="Adding target" key="{env="test", hostname="10.0.12.9", log_type="var-log-secure", service_name="var-log-secure", test="aaa"}"
level=info ts=2023-08-09T02:26:00.469215283Z caller=filetargetmanager.go:271 msg="Removing target" key="{env="test", hostname="10.0.12.9", log_type="var-log-secure", service_name="var-log-secure", test="aaa"}"
level=info ts=2023-08-09T02:26:00.469320684Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/secure
level=info ts=2023-08-09T02:26:00.46937081Z caller=filetargetmanager.go:271 msg="Removing target" key="{env="test", hostname="10.0.12.9", log_type="var-log-messages", service_name="var-log-messages"}"
level=info ts=2023-08-09T02:26:00.469469895Z caller=filetarget.go:151 msg="filetarget: watcher closed, tailer stopped, positions saved" path=/var/log/messages
我的问题的原因:我的系统中没有/var/log/messages和/var/log/secure
我的解决方案:
- targets:
- localhost
labels:
__path__: /var/log/messages
env: test
hostname: 10.0.12.9
service_name: var-log-messages
log_type: var-log-messages
- targets:
- localhost
labels:
__path__: /var/log/secure
env: test
hostname: 10.0.12.9
service_name: var-log-secure
log_type: var-log-secure
test: aaa
- targets:
- localhost
labels:
__path__: /var/log/syslog
env: test
hostname: 10.0.12.9
service_name: var-log-syslog
log_type: var-log-syslog
test: aaa
promtail成功!