Skip to content

Commit 49e8aa5

Browse files
committed
fix(keda): pin job=kube-state-metrics in the ScaleStuck Pending join
The DatabaseAutoscalerScaleStuck Pending-intersection term selected kube_pod_status_phase and kube_pod_labels without job="kube-state-metrics", unlike every other selector in this file (the rule's own HPA terms, all of ScaleBlocked, AtMax). Each VMRule runs against a tenant's own vmselect; if that vmselect ever carries a second series for either metric under a different job (a tenant-run KSM alongside federated/remote-written series), the on(namespace,pod) match becomes ambiguous and the whole rule errors out silently while its siblings keep working. Pin the job label on both selectors so the join stays one-to-one. The alerts unittest now asserts the pin on both terms so a revert goes red. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 6fa79c7 commit 49e8aa5

2 files changed

Lines changed: 9 additions & 5 deletions

File tree

‎packages/system/keda/templates/alerts.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,9 @@ spec:
7070
)
7171
and on(namespace,cluster) (
7272
max by (namespace,cluster) (
73-
kube_pod_status_phase{phase="Pending"}
73+
kube_pod_status_phase{job="kube-state-metrics",phase="Pending"}
7474
* on(namespace,pod) group_left(cluster)
75-
label_replace(kube_pod_labels{label_cnpg_io_cluster!=""}, "cluster", "$1", "label_cnpg_io_cluster", "(.+)")
75+
label_replace(kube_pod_labels{job="kube-state-metrics",label_cnpg_io_cluster!=""}, "cluster", "$1", "label_cnpg_io_cluster", "(.+)")
7676
) > 0
7777
)
7878
for: 15m

‎packages/system/keda/tests/db_autoscaling_alerts_test.yaml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -105,15 +105,19 @@ tests:
105105
path: spec.groups[0].rules[4].expr
106106
pattern: count\(kube_horizontalpodautoscaler_status_current_replicas\{.*keda-hpa
107107
# ScaleStuck must intersect with a Pending pod so a healthy long basebackup
108-
# (Running pod, desired > current for >15m) does not false-fire
108+
# (Running pod, desired > current for >15m) does not false-fire. Both selectors
109+
# in the Pending term MUST pin job="kube-state-metrics" like every other selector
110+
# in the file: a tenant vmselect carrying a second KSM series (federated/remote
111+
# write, or a tenant-run KSM) makes the on(namespace,pod) match ambiguous and the
112+
# whole rule errors out silently. Dropping the job pin re-opens that, so this goes red.
109113
- matchRegex:
110114
path: spec.groups[0].rules[0].expr
111-
pattern: kube_pod_status_phase\{phase="Pending"\}
115+
pattern: kube_pod_status_phase\{job="kube-state-metrics",phase="Pending"\}
112116
# ...and that Pending term is scoped to CNPG pods (label_cnpg_io_cluster) so an
113117
# unrelated Pending pod in the tenant namespace cannot trip the alert
114118
- matchRegex:
115119
path: spec.groups[0].rules[0].expr
116-
pattern: kube_pod_labels\{label_cnpg_io_cluster!=
120+
pattern: kube_pod_labels\{job="kube-state-metrics",label_cnpg_io_cluster!=
117121

118122
- it: can be disabled
119123
set:

0 commit comments

Comments
 (0)