Maybe related to #10257
What did you do?
- Create a Prometheus configuration with a kubernetes_sd_configs that has a non-empty target discovery.
- Start Prometheus with this configuration.
- Verify that Prometheus is scraping the targets that are discovered by the kubernetes_sd_configs.
- Modify the kubernetes_sd_configs to produce an empty result.
- Reload the Prometheus configuration.
What did you expect to see?
When the kubernetes_sd_configs is modified to produce an empty result, Prometheus should remove all of the targets that were discovered by the kubernetes_sd_configs before it was modified.
When an existing kubernetes_sd_configs is modified so that the targets discovered change without producing an empty result the targets in prometheus update correctly on reload, in both cases where more or less targets are discovered.
What did you see instead? Under which circumstances?
The discovered targets are still present and being scraped until prometheus is restarted
Maybe related: I have observed that if you (re)start Prometheus with a kubernetes_sd_configs that produces an empty result, nothing shows on the URLs /targets or /service-discovery related to this config.
System information
Linux 6.4.15-200.fc38.x86_64 (also seen on different setups local and k8s)
Prometheus version
prometheus --version
prometheus, version 2.47.0 (branch: HEAD, revision: efa34a5840661c29c2e362efa76bc3a70dccb335)
build user: root@409eb5e6b30c
build date: 20230906-10:20:07
go version: go1.21.0
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Also seen on prometheus 2.45.0
Prometheus configuration file
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
external_labels:
k8s_cluster: foo
scrape_configs:
- job_name: bb-discovery
honor_timestamps: true
params:
module:
- http_2xx
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /probe
scheme: http
follow_redirects: true
enable_http2: true
relabel_configs:
- source_labels: [job]
separator: ;
regex: (.*)
target_label: __tmp_prometheus_job_name
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: job
replacement: probe-bb-discovery
action: replace
- source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path]
separator: ;
regex: (.+);(.+);(.+)
target_label: __param_target
replacement: ${1}://${2}${3}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_ingress_name]
separator: ;
regex: (.*)
target_label: ingress
replacement: $1
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
target_label: __tmp_ingress_address
replacement: $1
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __address__
replacement: blackbox:9115
action: replace
- source_labels: [__meta_kubernetes_ingress_annotation_blackbox_discovery]
separator: ;
regex: disabled
replacement: $1
action: drop
- source_labels: [__meta_kubernetes_ingress_annotation_blackbox_severity_id]
separator: ;
regex: ^(.+)$
target_label: severity_id
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __probe_scheme
replacement: https
action: replace
- source_labels: [__meta_kubernetes_ingress_annotation_blackbox_scheme]
separator: ;
regex: ^(.+)$
target_label: __probe_scheme
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __probe_path
replacement: /-system/liveness
action: replace
- source_labels: [__meta_kubernetes_ingress_annotation_blackbox_path]
separator: ;
regex: ^(.+)$
target_label: __probe_path
replacement: $1
action: replace
- source_labels: [__probe_scheme, __param_target, __probe_path]
separator: ;
regex: (.+);http://(.+);(.+)
target_label: __param_target
replacement: ${1}://${2}${3}
action: replace
- source_labels: [__param_target]
separator: ;
regex: (.*)
target_label: instance
replacement: $1
action: replace
- separator: ;
regex: (.*)
target_label: __param_module
replacement: http_2xx
action: replace
- source_labels: [__meta_kubernetes_ingress_annotation_blackbox_module]
separator: ;
regex: ^(.+)$
target_label: __param_module
replacement: ${1}
action: replace
- separator: ;
regex: __meta_kubernetes_ingress_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__param_target]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: ingress
kubeconfig_file: "/home/foo/.kube/config"
follow_redirects: true
enable_http2: true
namespaces:
names:
- namespace1
- namespace2
selectors:
- role: ingress
label: "release=release1"Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response
Maybe related to #10257
What did you do?
What did you expect to see?
When the kubernetes_sd_configs is modified to produce an empty result, Prometheus should remove all of the targets that were discovered by the kubernetes_sd_configs before it was modified.
When an existing kubernetes_sd_configs is modified so that the targets discovered change without producing an empty result the targets in prometheus update correctly on reload, in both cases where more or less targets are discovered.
What did you see instead? Under which circumstances?
The discovered targets are still present and being scraped until prometheus is restarted
Maybe related: I have observed that if you (re)start Prometheus with a kubernetes_sd_configs that produces an empty result, nothing shows on the URLs /targets or /service-discovery related to this config.
System information
Linux 6.4.15-200.fc38.x86_64 (also seen on different setups local and k8s)
Prometheus version
Also seen on prometheus 2.45.0
Prometheus configuration file
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response