通知警报:超出上下文截止时间(可能的代理问题)

Question

我正在尝试将EC2 AWS中ubuntu上的Alermanager服务器连接到Slack，但我发现此错误：

Apr 14 18:22:26 prometheus-db-v01-01a.myserver.com alertmanager[5854]: level=error ts=2019-04-14T18:22:26.658601495Z caller=dispatch.go:177 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Post <redacted>: context deadline exceeded"

我的alertManager.yum看起来像：

templates:
- '/etc/alertmanager/template/slack.tmpl'
route:
  receiver: slack_general
  repeat_interval: 5m
  group_by: [alertname]
  routes:
# severity=info alerts will not try to match to any other rule
    - match:
        severity: info
      receiver: slack_general
receivers:
- name: slack_general
  slack_configs:
  - api_url: https://hooks.slack.com/services/ID
    send_resolved: true
    username: 'Prometheus-bot'
    channel: '#errors'
    title: '{{ template "slack.my.title" . }}'
    text: '{{ template "slack.my.text" . }}'

我可以看到触发测试警报的警报：

root@prometheus-db-v01-01a:~# amtool alert --alertmanager.url=http://localhost:9093 -v
Alertname         Starts At                Summary
Cassandra_yellow  2019-04-14 18:11:56 UTC  The cassandra  cluster is in yellow state

我用systemd开始了Prometheus：

root@prometheus-db-v01-01a:~# cat /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager Service
Wants=network-online.target
After=network.target
[Service]
Environment=https_proxy=http://proxy:80/
Environment=http_proxy=http://proxy:80/
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
    --config.file /etc/alertmanager/alertmanager.yml 
    --storage.path /var/lib/alertmanager/data
Restart=always
[Install]
WantedBy=multi-user.target
root@prometheus-db-v01-01a:~#

我尝试使用 2 Environment变量设置代理，但结果相同

服务器使用代理与 Slack 通信，我可以通过以下方式进行测试：

root@prometheus-db-v01-01a:~# curl -X POST --data-urlencode "payload={"channel": "#errors", "username": "webhookbot", "text": "This is posted to #errors and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}" https://hooks.slack.com/services/ID

这行得通，而

root@prometheus-db-v01-01a:~# curl -X POST --noproxy "*" --data-urlencode "payload={"channel": "#errors", "username": "webhookbot", "text": "This is posted to #errors and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}" https://hooks.slack.com/services/ID
curl: (7) Failed to connect to hooks.slack.com port 443: Connection timed out

以上所有命令均来自警报管理器服务器。

我认为这是一个代理问题是对的吗？如何设置？

提前致谢

Answer 1

问题已解决：它是代理，需要在alertmanager.yml文件中设置

templates:
- '/etc/alertmanager/template/slack.tmpl'
global:
  http_config:
    proxy_url: 'http://proxy:80/'
route:
  receiver: slack_general
  repeat_interval: 5m
  group_by: [alertname]
  routes:
    - match:
        severity: minor
      receiver: slack_general
receivers:
- name: slack_general
  slack_configs:
[...]