fix(monitoring): spread monitoring replicas over nodes - #4989

What this PR does

The monitoring chart renders its VMCluster, VLCluster, VTCluster and VMAlertmanager without affinity or topologySpreadConstraints, and vm-operator v0.68.4, which main and release-1.6 ship, copies both fields onto the pods as written and adds none of its own (build/statefulset.go#L11-L36, build/deployment.go#L11-L36). Nothing keeps two replicas of one tier off the same node.

Seen on three three-node clusters, on v1.6.4 and v1.6.5, which both render these resources without a placement rule and ship the same operator. In the root tenant of one of them both vmstorage replicas of longterm and both vmselect replicas of shortterm shared a node, with their volumes on the node-local local class. On another both vlstorage replicas shared a node, on replicated. On all three, two of the three Alertmanager replicas shared a node. With replicationFactor: 2 both copies of every longterm series sit on that node, and vmalert evaluates its rules through the vmselect of the first metrics storage, shortterm by default, so losing that node also stops alert evaluation. A StatefulSet pod on a lost node is not replaced until its Pod or Node object is deleted, so this holds for the Alertmanager replicas too, which have no volume.

This PR gives every tier a per-node spread keyed to the app.kubernetes.io/name and app.kubernetes.io/instance labels the operator puts on that component's pods (ClusterSelectorLabels, VMAlertmanager).

• The tiers that run as StatefulSets, vmstorage, vmselect, vlstorage, vtstorage and the Alertmanager, get maxSkew: 1 on kubernetes.io/hostname with whenUnsatisfiable: DoNotSchedule and nodeTaintsPolicy: Ignore, as fix(etcd): require an even spread of members over nodes by default #4986 proposes for etcd members and fix(seaweedfs): keep the masters on separate nodes through a drain #4947 for the SeaweedFS masters, for the reasons given there. A preference only adds to a node's score and fix(etcd): require an even spread of members over nodes by default #4986 shows other scores outweighing it, while a replica placed wrong on a node-local volume stays there. More replicas than nodes still start, and a replica evicted by a drain goes to a node without a peer or waits for its own node instead of joining a peer. As in fix(etcd): require an even spread of members over nodes by default #4986, a node the pods can never run on, such as a tainted control plane, still counts as an empty node, so next to such a node the three Alertmanagers need three schedulable nodes.
• The tiers that run as Deployments, vminsert, vlinsert, vlselect, vtinsert and vtselect, get the same spread with ScheduleAnyway. A Deployment replaces their pods on another node by itself, and a requirement would only add Pending pods during drains on small clusters.
• When _cluster.scheduling.globalAppTopologySpreadConstraints is set, every tier gets that policy instead, with the tier's selector added to each constraint that has none and a selector the policy sets itself left as written, the rule the keycloak chart applies to its database constraints. A value that does not parse or holds no non-empty topologySpreadConstraints list fails the render, as an empty logsStorages already does, instead of silently leaving the policy out. The traces-central VMAuth keeps the spread it already has.

On upgrade

The new pod templates make the operator update every tier once. StatefulSets default to OnDelete in the operator (build/defaults.go#L84-L87), and it replaces their pods itself, one at a time because the chart sets no rollingUpdateStrategyBehavior, in ordinal order through the Eviction API, waiting for each new pod to become Ready (reconcile/statefulset.go#L156-L176, #L215-L359). The wait is VM_PODWAITREADYTIMEOUT, 80s by default (config.go#L578-L587), which the victoria-metrics-operator package leaves alone. Deployments roll through the Deployment controller (reconcile/deploy.go#L26-L80). Within a VMCluster the order is vmstorage, vmselect, vminsert (vmcluster.go#L81-L143), within a VLCluster vlstorage, vlselect, vlinsert (vlcluster.go#L65-L74).

A replica without a volume, like the Alertmanager, or with a volume that can be attached on another node, like vlstorage on replicated, is placed again under the new rule during that rollout, so given enough nodes those tiers end up spread with nothing else to do.

A replica on a node-local volume cannot leave its node, and local is the default class for the metrics storage in the documented layout. If a peer of the same tier runs on that node while another node has none, the new pod cannot be scheduled, since the volume allows only that node and the spread refuses it, so it stays Pending. The operator gives up the wait after 80s and keeps the resource expanding, because a wait timeout counts as retryable (reconcile.go#L141-L143, controllers.go#L343-L353), and tries again on the next reconcile. The later tiers of that VMCluster or VLCluster are not updated meanwhile, and the peer keeps serving. For a VLCluster the monitoring HelmRelease waits on that status through its healthCheckExprs, so the release upgrade times out as well.

Such a replica does not move by itself, and this PR does not move it. Moving it means recreating its volume on another node. For vmselect that drops a cache. For vmstorage it drops that replica's copy, and with replicationFactor: 2 the series written before the move keep one copy until they age out of retention. vlinsert shards logs over vlstorage rather than replicating them, so for vlstorage it drops the logs on that volume. A platform that wants to defer the move can set globalAppTopologySpreadConstraints to a ScheduleAnyway policy, which the other charts that read it pick up too.

#4947 gives the same tiers a preferred pod anti-affinity in its second commit and argues against a required rule because of exactly this case. Since then the Alertmanager layout showed up on a third cluster, and #4986 found tenant etcd members on one node despite a ScheduleAnyway spread on the same storage classes, so this PR takes the other side for the tiers that keep state and states the cost. The two touch neighbouring lines but merge cleanly, and the monitoring suite passes on the merge, 27 suites and 327 tests. If you prefer this approach, the monitoring commit of #4947 is no longer needed.

Tests

tests/topology_spread_test.yaml pins the whole constraint list of every tier for the default, for a second VMCluster name, for an empty platform policy, for a zone policy, for a policy with two constraints, for a policy constraint that brings its own selector, and for a constraint without a selector after and before one with its own, with matchLabels and with matchExpressions. tests/topology_spread_invalid_policy_test.yaml pins the failure for a policy that does not parse, a bare list, an empty list and a map. Against the templates on main all 18 cases fail, and against 0dbbb3b only the two new ones. The suites also catch each of 19 mutations, which are a soft default for every tier, no nodeTaintsPolicy, Honor, maxSkew: 2, a soft vmselect, vlstorage, vtstorage or Alertmanager, a hard vminsert, ignoring the platform policy, adding the selector to the last platform constraint only, keying vmstorage to the release name, dropping the failure, dropping either of its two conditions, overwriting a selector the policy brings, and merging the tier's selector into the shared selector map, into the parsed policy constraint or into every constraint. The first of those three is the merge order of the first commit. With a policy that mixes both kinds of constraints it wrote the policy's own selector into the selector of every constraint without one, and only the two new cases catch it.

$ docker run --rm -v "$PWD":/apps -w /apps/packages/system/monitoring helmunittest/helm-unittest:4.2.3-1.1.2 .
Charts:      1 passed, 1 total
Test Suites: 26 passed, 26 total
Tests:       322 passed, 322 total

On main the same command gives 24 suites and 304 tests, with helm-unittest 1.1.2 as CI pins it, and packages/extra/monitoring stays at 4 suites and 31 tests. With default values helm template differs from main only in the new blocks and in the alerta key and Grafana password the chart generates at random, and an empty globalAppTopologySpreadConstraints renders the same as none. I also decoded the rendered VMCluster, VLCluster, VTCluster and VMAlertmanager specs strictly into the operator's v0.68.4 API types, which rejects unknown fields, and checked each selector against the labels the operator's PodLabels gives its component. Each matches its own component and no other one in the release. With a policy that mixes both kinds of constraints the same holds for every selector the chart adds, and the policy's own selector stays as written. I have not run the e2e suite or rolled this out on a cluster.

vmcluster.yaml and vlogs.yaml are the same on release-1.6, and alerta.yaml differs from main in one unrelated line. release-1.6 has no VictoriaTraces, so the backport leaves those lines and their tests out. That backport is a separate PR.

Screenshots

Not a UI change.

Downstream repositories

Only templates and tests of packages/system/monitoring change. No values, schema, README, ApplicationDefinition, platform values or tooling change, and the website shows the monitoring PVCs and pods of an install but documents no placement workaround, so no trigger map entry applies.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

Release note

fix(monitoring): the vmstorage, vmselect, vlstorage, vtstorage and Alertmanager replicas of the monitoring stack now require separate nodes, and vminsert, vlinsert, vlselect, vtinsert and vtselect prefer them. A platform-wide scheduling.globalAppTopologySpreadConstraints replaces both. On upgrade, a replica on a node-local volume whose peer of the same tier shares its node stays Pending and holds up the rollout of its cluster until that replica is moved.

Summary by CodeRabbit

• New Features

  • Improved monitoring workload distribution across hosts and zones to increase resilience.
  • Added topology-aware scheduling for metrics, logs, traces, and alerting components.
  • Supports configurable platform scheduling policies, including required and preferred placement rules.
  • Provides sensible hostname-based defaults when no custom policy is configured.

• Bug Fixes

  • Ensures scheduling constraints consistently target the correct monitoring components, including when custom constraints are supplied.
  • Added validation for malformed or unsupported topology scheduling policies.