如何添加scrape_configs文件到Prometheus

Question

我已经使用牧场主监控Helm图在cattle-monitoring-system命名空间中部署了Prometheus, Alertmanager, Grafana等。查看这里和这里的文档。

现在我试图部署一个额外的普罗米修斯实例在另一个命名空间称为indigo使用manifest文件。不幸的是，配置文件scrape_configs不会加载。我通过运行命令kubectl exec -it prometheus-indigo-0 -c prometheus -n indigo -- cat /etc/prometheus/config_out/prometheus.env.yaml检查了这一点。在其他Prometheus实例中运行相同的命令将返回所有的scrape_configs配置。

我的deployment.yml文件看起来像这样:

apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: $NAMESPACE
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: monitoring-rancher-monitor-prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: $NAMESPACE
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: indigo
namespace: $NAMESPACE
spec:
serviceAccountName: prometheus
additionalScrapeConfigs:
name: prometheus-scrape-configs-secret
key: prometheus-scrape-configs.yml
resources:
requests:
memory: 400Mi
---
apiVersion: v1
kind: Service
metadata:
name: prometheus-indigo
namespace: $NAMESPACE
spec:
type: NodePort
ports:
- name: web
nodePort: 30900
port: 9090
protocol: TCP
targetPort: web
selector:
prometheus: indigo

prometheus-scrape-configs.yml文件看起来像这样:

global:
evaluation_interval: 30s
scrape_interval: 30s
scrape_configs:
- job_name: blackbox # To get metrics about the exporter itself
metrics_path: /metrics
static_configs:
- targets:
- .....
- job_name: blackbox-http # To get metrics about the exporter’s targets
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- .....
labels:
env: elise
- targets:
- .....
labels:
env: osb
- targets:
- .....
labels:
env: itp
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: .....

最后，在Makefile:

中创建prometheus-scrape-configs-secret

include ../../../common.mk
include ../../environments/$(ENVIRONMENT).mk
NAMESPACE ?= indigo
.PHONY: deploy
deploy: 
init 
deploy-monitoring
.PHONY: init
init:
.PHONY: deploy-monitoring
deploy-monitoring:
kubectl create secret generic prometheus-scrape-configs-secret 
-n $(NAMESPACE) --from-file=prometheus-scrape-configs.yml
envsubst < $(ENVIRONMENT)-deployment.yml | kubectl apply -f -
.PHONY: delete
delete: delete-monitoring
.PHONY: delete-monitoring
delete-monitoring:
kubectl delete secret prometheus-scrape-configs-secret -n $(NAMESPACE)
envsubst < $(ENVIRONMENT)-deployment.yml | kubectl delete -f -

所以有人知道为什么scrape_configs文件无法正确加载吗?

编辑:

运行命令kubectl describe pod prometheus-indigo-0 -n indigo后，我注意到下面的错误。我还注意到，在部署这个Prometheus实例后，它会得到一个错误，立即重新启动，之后它正在运行。

State:       Running
Started:   Tue, 14 Jun 2022 18:01:23 +0200
Last State:  Terminated
Reason:    Error
Message:   ts=2022-06-14T16:01:19.743Z caller=main.go:450 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"

Answer 1

多亏了这个文档，我设法解决了这个问题。我忘记创建一个ServiceMonitor，它通过标签引用blackbox-exporter。最后，您需要使用serviceMonitorSelector将其链接到Kubernetes Prometheus对象中。您可以在这里查看如何配置serviceMonitorSelector

的更多信息。