What did you do?
I implemented oauth2 authentication. I see 2 requests to the token server, one with basic-auth credentials (generated from client_id and client_secret) which gets a http 200 back from my server, but right after this successful request there is another request with credentials inside the payload. This request leads to a http 4xx, because this is not implemented at my oauth server.
So why does the alertmanager make two authentication calls to the oauth2 server?
What did you expect to see?
Alertmanager only calls the oauth2 server once, if the call was successful.
What did you see instead? Under which circumstances?
There is another call after the first successful one, this call leads to an error and because of this, the alertmanager is not firing alerts.
Environment
Alertmanager Version: 0.23
Docker-container (prom/alertmanager:latest) @ CentOS7 Dockerhost
Linux 3.10.0-1160.42.2.el7.x86_64 x86_64
-
Alertmanager version:
alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b1)
build user: root@e21a959be8d2
build date: 20210825-10:48:55
go version: go1.16.7
platform: linux/amd64
-
Prometheus version:
prometheus, version 2.21.0 (branch: HEAD, revision: e83ef207b6c2398919b69cd87d2693cfc2fb4127)
build user: root@a4d9bea8479e
build date: 20200911-11:35:02
go version: go1.15.2
-
Alertmanager configuration file:
global:
resolve_timeout: 5m
http_config:
tls_config:
ca_file: <paht_to_file>
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 2m
repeat_interval: 2h
receiver: 'email'
routes:
- receiver: 'zis'
group_by: ['alertname']
group_wait: 10s
group_interval: 2m
repeat_interval: 2h
receivers:
- name: 'zis'
webhook_configs:
- url: https://example.com/api_esb_prometheus/events
http_config:
oauth2:
client_id: MY_USERNAME
client_secret: MY_SUPER_SECRET
token_url: https://example.com/api_esb_prometheus/auth/token
tls_config:
ca_file: <path_to_file>
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
- Prometheus configuration file:
global:
scrape_interval: 30s
rule_files:
- /etc/prometheus/rules_prometheus.yml
alerting:
alertmanagers:
- static_configs:
- targets:
- <TARGET_SERVER_AND_PORT>
scrape_configs:
- job_name: 'kafka'
scrape_interval: 30s
metrics_path: '/metrics'
static_configs:
- targets: ['<SOME_SERVER_AND_PORT>']
labels:
instance: 'dev'
host: 'myhost'
level=debug ts=2021-10-11T09:13:37.230Z caller=dispatch.go:516 component=dispatcher aggrGroup="{}/{}:{alertname=\"InstanceDown\"}" msg=flushing alerts=[InstanceDown[f96c67a][active]]
level=warn ts=2021-10-11T09:13:37.249Z caller=notify.go:724 component=dispatcher receiver=zis integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://example.com/api_esb_prometheus/events\": oauth2: cannot fetch token: 422 Unprocessable Entity\nResponse: {\"errorMessage\":\"Check your parameters!\",\"errors\":[{\"msg\":\"is required\",\"param\":\"authorization\",\"location\":\"headers\"}]}"
What did you do?
I implemented oauth2 authentication. I see 2 requests to the token server, one with basic-auth credentials (generated from client_id and client_secret) which gets a http 200 back from my server, but right after this successful request there is another request with credentials inside the payload. This request leads to a http 4xx, because this is not implemented at my oauth server.
So why does the alertmanager make two authentication calls to the oauth2 server?
What did you expect to see?
Alertmanager only calls the oauth2 server once, if the call was successful.
What did you see instead? Under which circumstances?
There is another call after the first successful one, this call leads to an error and because of this, the alertmanager is not firing alerts.
Environment
Alertmanager Version: 0.23
Docker-container (prom/alertmanager:latest) @ CentOS7 Dockerhost
Linux 3.10.0-1160.42.2.el7.x86_64 x86_64
Alertmanager version:
alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b1)
build user: root@e21a959be8d2
build date: 20210825-10:48:55
go version: go1.16.7
platform: linux/amd64
Prometheus version:
prometheus, version 2.21.0 (branch: HEAD, revision: e83ef207b6c2398919b69cd87d2693cfc2fb4127)
build user: root@a4d9bea8479e
build date: 20200911-11:35:02
go version: go1.15.2
Alertmanager configuration file: