Skip to content

Commit 1f2ff31

Browse files
committed
fix(postgres): pin job=kube-state-metrics on the autoscaler read-load query
The read-load trigger query joined on kube_pod_labels without pinning job="kube-state-metrics", while the sibling DatabaseAutoscaler* alert rules already pin it. A second exporter emitting a kube_pod_labels series (or a duplicate scrape) would multiply the join and distort the desired-count arithmetic. Pin the job on every kube_pod_labels selector in the query for parity with the alerts. Follow-up from cozystack#3954 (cozystack#4187). Signed-off-by: Alexey Artamonov <[email protected]>
1 parent afc003c commit 1f2ff31

2 files changed

Lines changed: 11 additions & 5 deletions

File tree

‎packages/apps/postgres/templates/_autoscaling.tpl‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ provisioning so it no longer errors there; it stays opt-in / not-live-validated.
174174
{{- $ns := .Release.Namespace -}}
175175
{{- $rel := .Release.Name -}}
176176
{{- $target := .Values.autoscaling.target -}}
177-
{{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}}
177+
{{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}}
178178
{{- /* Read-load term Σ. The floor is GATED on the existence of ANY instance pod
179179
(instance_role=~".+", primary OR replica), not replica-only: `or (vector(0) and
180180
on() (count(...instance_role=~".+") > 0))` returns 0 whenever the cluster has a
@@ -187,7 +187,7 @@ provisioning so it no longer errors there; it stays opt-in / not-live-validated.
187187
basebackup that routinely exceeds the alert's 15m window. The primary always exists
188188
during provisioning, so gating on any instance role floors correctly there. This is
189189
the fail-safe separation of no-load from no-series. */ -}}
190-
{{- $idleFloor := printf "(vector(0) and on() (count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) > 0))" $ns $rel -}}
190+
{{- $idleFloor := printf "(vector(0) and on() (count(kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) > 0))" $ns $rel -}}
191191
{{- $load := "" -}}
192192
{{- if eq .Values.autoscaling.metric "ReadCPUUtilization" -}}
193193
{{- $load = printf "((sum(rate(container_cpu_usage_seconds_total{namespace=%q,container=\"postgres\"}[5m]) %s) * 1000) or %s)" $ns $joinReplica $idleFloor -}}
@@ -200,7 +200,7 @@ provisioning so it no longer errors there; it stays opt-in / not-live-validated.
200200
{{- $base -}}
201201
{{- else -}}
202202
{{- /* Cluster-wide join (all instances) for the lag and WAL-write terms. */ -}}
203-
{{- $joinCluster := printf "* on(namespace,pod) group_left() kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q}" $ns $rel -}}
203+
{{- $joinCluster := printf "* on(namespace,pod) group_left() kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q}" $ns $rel -}}
204204
{{- $cooldown := "5m" -}}
205205
{{- /* Each gate is individually floored with `or vector(0)`: if its source series
206206
is absent (a CNPG that does not emit cnpg_pg_replication_lag /
@@ -218,7 +218,7 @@ provisioning so it no longer errors there; it stays opt-in / not-live-validated.
218218
carry jobRole, not instanceRole), and kube-state-metrics reports them until
219219
GC. Without this filter a lingering join Job pod inflates the frozen count
220220
while the brake is engaged, nudging desired up instead of holding it. */ -}}
221-
{{- $frozen := printf "(count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}}
221+
{{- $frozen := printf "(count(kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}}
222222
{{- /* base when not braking, frozen (= currentInstances * target) when braking.
223223
The frozen summand carries a defensive `or vector(0)`, but note it is a no-op
224224
given the surrounding expression, NOT a floor that ever changes the result: the

‎packages/apps/postgres/tests/autoscaling_test.yaml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,12 @@ tests:
182182
- matchRegex:
183183
path: spec.triggers[0].metadata.query
184184
pattern: label_cnpg_io_instance_role="replica"
185+
# the kube_pod_labels join pins job="kube-state-metrics", parity with the alert
186+
# rules, so a duplicate kube_pod_labels series from another exporter cannot
187+
# multiply the join.
188+
- matchRegex:
189+
path: spec.triggers[0].metadata.query
190+
pattern: kube_pod_labels\{job="kube-state-metrics"
185191
# ...and scoped to THIS cluster by release name, so a namespace with multiple
186192
# postgres clusters does not cross-count. The dashboard read-load panel mirrors
187193
# this per-cluster scoping (label_cnpg_io_cluster).
@@ -560,7 +566,7 @@ tests:
560566
# (which carry cnpg.io/cluster but jobRole, not instanceRole) don't inflate it
561567
- matchRegex:
562568
path: spec.triggers[0].metadata.query
563-
pattern: count\(kube_pod_labels\{namespace="tenant-acme",label_cnpg_io_cluster="postgres-db",label_cnpg_io_instance_role=~".\+"\}\) \* 150
569+
pattern: count\(kube_pod_labels\{job="kube-state-metrics",namespace="tenant-acme",label_cnpg_io_cluster="postgres-db",label_cnpg_io_instance_role=~".\+"\}\) \* 150
564570
# the frozen summand is floored with `or vector(0)` so a kube-state-metrics
565571
# scrape gap (empty count) cannot empty the whole `+` sum and defeat the
566572
# base term's own floor

0 commit comments

Comments
 (0)