feat(monitoring): alert when pods of one workload share a node - #4993

What this PR does

On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant etcd with two of its three members on one node, a tenant Redis with two of its three Sentinels on the node that also ran the master, and several VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect, vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one of those nodes would have cost the etcd or Sentinel quorum, or both copies of a storage tier. Spread constraints keep new pods apart once they are in place (#4986 for etcd, #4989 for monitoring, #4990 for Redis), but a placement that already exists, or that comes from a soft rule or a drain, stays as it is, because the scheduler does not move running pods.

This adds PodReplicasColocated, a warning in a new workload-placement group under packages/system/monitoring-agents/alerts/. It fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster or StrimziPodSet have been on one node for an hour while the cluster has at least as many schedulable nodes as the workload has running pods, so five replicas on three nodes stay quiet. The alert carries cluster, namespace, owner_kind, owner_name and node, plus the severity, exported_instance and service labels the other platform alerts set, and its value is the number of pods on that node. DaemonSets, Jobs and every other owner kind are not in the list, so a DaemonSet surge pod next to its predecessor does not count.

etcd-operator gives every member pod its own EtcdMember as controller owner (etcdmember_controller.go#L546), so grouping by owner would see one pod per group. Members get their names from GenerateName: <cluster>- (etcdcluster_controller.go#L434, #L839), and members adopted from the old StatefulSet keep names like etcd-0 (adopt.go#L103). The rule drops the last dash-separated part of the member name and reports the group as EtcdCluster <name>. With the etcd package that is one group per tenant namespace, and two etcd clusters in one namespace would still be told apart. CNPG instance pods carry a controller reference to their Cluster (cluster_create.go#L1580), and Strimzi pods one to their StrimziPodSet (StrimziPodSetController.java#L369).

Owner and node both come from kube_pod_info. kube-state-metrics fills created_by_kind and created_by_name from the controller owner reference and node from spec.nodeName (pod.go#L638-L653). kube_pod_owner and kube_pod_status_phase have no node label of their own, and the kube-state-metrics scrape adds node from __meta_kubernetes_pod_node_name with honorLabels: true (templates/kube-state-metrics-scrape.yaml), so on those series node is the node of the kube-state-metrics pod. The test data keeps that label on the phase series. A pod counts while kube_pod_status_phase{phase="Running"} is 1, so a Failed or Pending pod on the same node does not. kube_pod_labels does carry every pod label here (--metric-labels-allowlist=pods=[*]), but the rule does not depend on it.

A node counts as schedulable unless it has a NoSchedule or NoExecute taint. A cordon shows up as the node.kubernetes.io/unschedulable:NoSchedule taint (node_lifecycle_controller.go#L549-L555), and not ready and unreachable nodes get theirs from their conditions (#L87-L99), so one clause covers all three, and a tainted control plane does not count either. Pods that a drain had to put together on the remaining nodes therefore stay quiet while the node is out, and the hour starts once it is back.

The alert waits an hour. A replacement pod runs next to its terminating predecessor only for the predecessor's grace period, and node maintenance takes the drained node out of the count, while a placement that has held for an hour does not fix itself. The alert is about a risk rather than an outage, so the wait costs nothing. The VMAlert sets both remoteWrite and remoteRead, so the pending state survives a vmalert restart (vmalert docs).

Not covered are workloads that run one pod per owner, such as ClickHouse with one StatefulSet per replica (patterns.go#L45-L53) and the KubeVirt VMs behind a tenant Kubernetes node group. They would need grouping by pod labels, which this rule avoids. The node count is cluster-wide, so a workload that a node selector or affinity restricts to fewer nodes than it has pods fires too.

hack/workload-placement-alerts.bats takes .spec of the VMRule and replays it through vmalert-tool v1.136.0 pinned by digest, the release vm-operator v0.68.4 deploys by default (config.go#L38) and the same image #4963 and #4985 use. hack/testdata/workload-placement-alerts/unittest.yaml has 13 cases with 24 checks. Two of three StatefulSet pods on one node fire at 61 minutes and not at 59, with labels and annotations pinned. Spread pods, five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of one Job, a Failed pod next to a running one, and one pod each in two clusters that share workload and node names stay quiet. etcd members fire as one EtcdCluster per namespace, for generated and adopted member names. A replacement pod next to its terminating predecessor for twenty minutes stays quiet, and the same overlap held past the hour fires and resolves when it ends. A cordoned node keeps the alert quiet until it is uncordoned, and a node with unreachable taints keeps it quiet throughout. ReplicaSet, CNPG Cluster and StrimziPodSet pods fire with their owner kind.

$ bats hack/workload-placement-alerts.bats
1..1
ok 1 workload placement alerts pass their vmalert-tool cases

With the expression replaced by one that never fires, the suite prints not ok 1 and vmalert-tool reports 5 of the 24 checks, which are all the checks that expect an alert. On main, where the rule file does not exist, the suite fails as well. Each of these changes to the rule fails at least one check, with the number of failing checks in brackets. for: 15m (5), no phase filter (1), no node count (6), no cluster label (6), DaemonSet and Job in the kind list (3), no taint clause (4), < instead of <= (5), etcd members grouped per member (1).

With Bats 1.14.0 at the commit CI pins, the suite passes together with alert-severity-contract, bats-strict-setup, bats-no-negated-assert, bats-no-exit-trap, bats-runner-coverage, bats-runner-contract, image-pin-consistency and check-commit-trailers, 109 of 109, and the severity contract lists the new alert as warning. helm-unittest 1.1.2, the version CI pins, passes the monitoring-agents chart with 19 of 19 tests in 6 suites, and helm template with Helm 3.17.3 renders the VMRule the same as the file. Like the migration suites the new suite needs docker, and only the image pull touches the network. I have not run the rule in vmalert on a live cluster.

On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts template, the VMAlert and vm-operator v0.68.4 are the same, and etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member names. The new files apply there without changes. That backport is a separate PR with one more commit that drops the load test_helper line, because release-1.6 runs hack/*.bats through hack/cozytest.sh.

Screenshots

Not a UI change.

Downstream repositories

A new alert file under packages/system/monitoring-agents/alerts/, a Bats suite under hack/ and its test data. No values, schema, README, image or ApplicationDefinition change, and the website does not list platform alerts by name, so no trigger map entry applies.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them.

Summary by CodeRabbit

• New Features

  • Added monitoring for workloads whose running pod replicas are colocated on the same node.
  • Alerts include workload, node, cluster, and severity details, along with quorum-loss guidance.
  • Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets, and EtcdMembers.

• Tests

  • Added validation for workload-placement alerts across scheduling, availability, cluster-isolation, and pod-replacement scenarios.