fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves by yankawai · Pull Request #4963 · cozystack/cozystack

europrinter (yankawai) · 2026-10-19T16:42:54Z

What this PR does

linstorSatelliteErrorRate fires for every node that has any old LINSTOR error report whenever the LINSTOR controller pod moves to another node, and it keeps firing for 15 minutes. linstor_error_reports_count is exported by the controller, so its series carry the controller pod's instance and controller_node, and a moved pod starts new series that hold the existing counts. VictoriaMetrics' increase() starts from the last sample before the window, and when a series has none, rollupDelta counts it from zero whenever its first value is below ten times the next change plus one. That is the case the docs describe as increase ignoring "the first value in a series if it is too big" under increase_pure, and report counts are never that big, so every new series counts all of its reports as new. The up guard from #2265 matches on(instance, job), and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the upgrade the controller pod moved to another node, and the alert fired for all three nodes with values 1, 1 and 2 for 15 minutes, while linstor error-reports list --since 1h was empty and the four reports that exist were weeks old. The day before, it fired the same way for about 15 minutes when only the cluster external label of vmagent changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3 the controller asks every satellite for its reports on each scrape and leaves out any satellite that is not connected (prepareErrRequestApi), so after a node reboot or any other disconnect longer than five minutes the satellite's old count is counted again.

Both rules now take the count per hostname, hold the last value for an hour with last_over_time, and run increase() over that as a one-minute subquery, whose inner points VictoriaMetrics aligns to that step. Keyed on hostname, a moved controller or a changed label continues the same series, and the hold bridges a disconnect or a slow controller start of up to an hour. The up guard matches on(), since the aggregated result has no instance any more. A new report still fires with the same value and stays for 15 minutes, at most a minute later than before because of the one-minute step. The alerts now carry hostname and severity, and no longer the controller's instance, job and controller_node, the module, or the external labels of the scrape.

linstorControllerErrorRate gets the same form. Its hostname is the controller pod name and the piraeus controller keeps its reports in an emptyDir, so a new pod starts with no reports and a move alone does not trip it, but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts over, and as before the up guard holds the alert back only if the controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with read-only queries to vmselect. Over the upgrade the current expression returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second step, the same interval as vmalert's ALERTS series, and over the label change from 20:27:30 to 20:42:00. The new expression returns nothing over both windows, and without > 0 it returns each of the three nodes with 0 at every step, 88 and 121 points. The same storage has six gaps of 8 to 113 minutes over one night with no failed scrape recorded in them. Replaying vmalert's instant query of the increase() part every 30 seconds in the 154 evaluations after those gaps, the current expression returns at least one node in 99, the hostname key without the hold in 84, and the new expression in 30, all of them after the 113-minute gap. Retention there does not reach back to a real new report, so the firing side is covered by the tests.

hack/linstor-error-report-alerts.bats replays both rules through vmalert-tool v1.136.0, pinned by digest, which is the VictoriaMetrics release victoria-metrics-operator v0.68.4 deploys by default, with 30-second samples and 15-second evaluations like the platform's vmagent and vmalert. The suite needs docker, like the migration suites, and only the image pull touches the network. hack/testdata/linstor-error-report-alerts/unittest.yaml has 20 cases in five groups, which cover a controller move followed by a new report, a changed external label for both rules, one satellite away for 40 minutes and one that comes back with a new report, a new report and a first report on a node for both rules, and a controller that cannot be scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of them because old reports fire again after the move, the label change or the disconnect, and the other 4 on the label set alone, since the current alerts carry the controller's labels. Each of seven mutations of the new expressions turns at least one case red, namely no hold, a 15-minute hold, the guard on instance, no guard, the key on hostname and instance, the key on hostname and cluster, and the controller rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite with alert-severity-contract, bats-strict-setup, bats-no-negated-assert, bats-no-exit-trap, bats-runner-coverage and image-pin-consistency passes 74 of 74, and helm template renders the same 15 rules as main. I have not run the new rules in vmalert on a live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes there. On release-1.6 it applies without conflicts, but that branch runs hack/*.bats through hack/cozytest.sh and has no hack/test_helper.bash, so the suite there has to drop its load test_helper line.

Screenshots

Not a UI change.

Downstream repositories

A rule file under packages/system/piraeus-operator/alerts/, a new Bats suite under hack/ and its test data. No values, schema, README, image, make target or ApplicationDefinition change, and the website does not mention these rules or the metric, so no trigger map entry applies.

Release note

fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.

Summary by CodeRabbit

Bug Fixes
- Improved LINSTOR controller and satellite error alerts to account for recent error reports over a 15-minute window, even when metrics are intermittent or scrape labels change.
- Updated alert matching to better handle controller moves between nodes and satellites that disappear from metrics while disconnected.

…n the controller pod moves linstor_error_reports_count comes from the LINSTOR controller, so its series carry the controller pod's instance and controller_node labels. When the pod moves to another node, or an external label of the scrape changes, the existing report counts come back as new series. VictoriaMetrics' increase() counts the first samples of a series from zero when they are small, and report counts are small, so linstorSatelliteErrorRate fires for every node with an old report and keeps firing for 15 minutes. The up guard matches on instance and lets the new series through. The controller also leaves a satellite out of the metric while it is not connected, so a satellite that comes back after more than five minutes has its old count counted again in the same way. Keyed on hostname, a moved controller or a changed label continues the same series, and holding the last value for an hour bridges a disconnect or a slow controller restart. The up guard matches on() because the result no longer has an instance label. The controller rule takes the same form. Its hostname is the pod name and a new pod starts with no reports, but a label change hit it the same way. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>

coderabbitai · 2026-10-19T16:49:44Z

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Source mode: Auto
Repository: cozystack/cozystack
Branch: main
File: .coderabbit.yaml
Commit: 56224a1
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: 93f0a77b-0cbc-41d0-98d1-f8e4d2c275cd

Commits

Reviewing files that changed from the base of the PR and between 8502629 and f1067fd.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The LINSTOR satellite and controller error-rate alert expressions now use recent samples aggregated by hostname. New unit scenarios test alert evaluations, and a Bats test runs those scenarios with vmalert-tool.

Changes: LINSTOR error-report alert evaluation

Layer / File(s)	Summary
Update error-rate expressions `packages/system/piraeus-operator/alerts/piraeus-datastore.yaml`	The satellite and controller expressions use the maximum of samples from the previous hour, then check for increases over 15 minutes at one-minute resolution. The satellite availability condition matches without labels.
Test alert evaluation `hack/testdata/linstor-error-report-alerts/unittest.yaml`, `hack/linstor-error-report-alerts.bats`	Unit scenarios cover controller movement, a changed cluster label, disconnected satellites, new reports, and failed scrapes. The Bats test runs the rules and scenarios using a pinned `vmalert-tool` image.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Suggested reviewers: sasha-sup

Merge Risk: Minimal · previous review 85026

The alert changes are mergeable after normal checks. The questioned long-outage behavior has not been shown to result from this change.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: preventing LINSTOR error-report alerts from firing when the controller pod moves.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (2 skipped: 2 unsupported.)

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: `db6989b5-8ef7-4184-b1f8-f5d246ae7df1`

📥 Commits

Reviewing files that changed from the base of the PR and between `aed7fa8` and `8502629`.

📒 Files selected for processing: 3

`hack/linstor-error-report-alerts.bats`

`hack/testdata/linstor-error-report-alerts/unittest.yaml`

`packages/system/piraeus-operator/alerts/piraeus-datastore.yaml`

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

IvanHunters

Used vmalert-tool v1.136.0 as the oracle (VM increase semantics differ from Prometheus, so promtool would be wrong here) and ran the new bats suite (20 cases green), including the true-positive case where a single new error report still fires. Rendered 15 rules; both expressions render verbatim and keep severity: warning. No Alertmanager route or inhibit references the dropped labels, so routing is unaffected. Merge with current main is clean. No blocking issues.

Non-blocking:

piraeus-datastore.yaml:32,42: a narrow over-suppression edge. While a stale and a fresh series with the same hostname coexist (up to the last_over_time[1h] window), max by (hostname) takes the larger, so if reports were deleted on the fresh series and a new report arrives below the retained stale value, it can be masked until the stale series expires. Requires move plus delete plus new report within an hour. The old code had a much worse false positive on every controller move, so this is a clear improvement.
The native hostname label on linstor_error_reports_count cannot be confirmed offline (the metric comes from the external LINSTOR controller). Indirectly supported by the podscrape relabeling, the pre-PR annotation, and the dashboard.

…ts (#4985) ## What this PR does The tenant etcd chart has its own copy of `etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every `grpc_code` other than `OK` as a failure. `grpc_server_handled_total` counts a Watch stream once, when it ends, and the streams of a kube-apiserver that goes away end with `Canceled`. So whenever an API server on a tenant etcd stops, the Watch method of every member reads as 100 percent failed and the critical rule fires on all of them. Both rules now count only `Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`, the list the etcd mixin has used since etcd-io/etcd#13127 and the one the platform copy in `packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The grouping `BY (job, instance, grpc_service, grpc_method)`, the thresholds, `for` and the annotations stay as they are, so the alerts keep their labels. Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was created and deleted a few minutes later. vmalert had the critical alert firing for `grpc_method="Watch"` on all three members with the value 100. At that point `rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for Watch with `Canceled`, and the only other non-OK codes were `FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is one call each. The same query an hour earlier returned nothing. The platform copy runs in the same vmalert over the same series, and none of the firing alerts came from it. Each alert also showed up seven times there, once per tenant copy of these rules, because the root tenant's vmalert loads the rules of every tenant namespace. This PR does not change that. `hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with `helm template` and replays it through `vmalert-tool` v1.136.0 pinned by digest, the same image #4963 uses. `hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with seven checks. Watch streams that end only with `Canceled` must not fire at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25 percent must fire the critical rule at 7 minutes and both rules at 12, with the labels and the summary pinned. Both together must fire only for `Range`. Against the rules on main the suite fails 4 of the 7 checks, all of them on Watch alerts with the value 100. Dropping `Unavailable` from the list, or grouping by `job` and `instance` only, fails 3 checks each. ```console $ bats hack/etcd-grpc-alerts.bats 1..1 ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases ``` The same run on the rules from main prints `not ok 1` and the four failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes together with `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`, `alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like the migration suites it needs docker, and only the image pull touches the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run the new rules in vmalert on a live cluster. The rules file is the same on release-1.6 and release-1.7. The commit cherry-picks onto release-1.6 without conflicts, and that backport is a separate PR with one more commit that drops the `load test_helper` line, because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`. ### Screenshots Not a UI change. ### Downstream repositories A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and its test data. No values, schema, README, image or `ApplicationDefinition` change, and the website does not mention these rules, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member. ```  ## Summary by CodeRabbit - **Bug Fixes** - Refined etcd gRPC failure alerts to target specific failure status codes, reducing alerts for non-failure responses while preserving existing thresholds and timing. - Added coverage for canceled watch streams, unavailable range requests, and combined failure scenarios, including warning and critical alert evaluations. - **Tests** - Added integration validation for rendering etcd alert rules and running them through the alert-testing tool, including failure handling when prerequisites are unavailable.

…ts (#4985) ## What this PR does The tenant etcd chart has its own copy of `etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every `grpc_code` other than `OK` as a failure. `grpc_server_handled_total` counts a Watch stream once, when it ends, and the streams of a kube-apiserver that goes away end with `Canceled`. So whenever an API server on a tenant etcd stops, the Watch method of every member reads as 100 percent failed and the critical rule fires on all of them. Both rules now count only `Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`, the list the etcd mixin has used since etcd-io/etcd#13127 and the one the platform copy in `packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The grouping `BY (job, instance, grpc_service, grpc_method)`, the thresholds, `for` and the annotations stay as they are, so the alerts keep their labels. Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was created and deleted a few minutes later. vmalert had the critical alert firing for `grpc_method="Watch"` on all three members with the value 100. At that point `rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for Watch with `Canceled`, and the only other non-OK codes were `FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is one call each. The same query an hour earlier returned nothing. The platform copy runs in the same vmalert over the same series, and none of the firing alerts came from it. Each alert also showed up seven times there, once per tenant copy of these rules, because the root tenant's vmalert loads the rules of every tenant namespace. This PR does not change that. `hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with `helm template` and replays it through `vmalert-tool` v1.136.0 pinned by digest, the same image #4963 uses. `hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with seven checks. Watch streams that end only with `Canceled` must not fire at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25 percent must fire the critical rule at 7 minutes and both rules at 12, with the labels and the summary pinned. Both together must fire only for `Range`. Against the rules on main the suite fails 4 of the 7 checks, all of them on Watch alerts with the value 100. Dropping `Unavailable` from the list, or grouping by `job` and `instance` only, fails 3 checks each. ```console $ bats hack/etcd-grpc-alerts.bats 1..1 ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases ``` The same run on the rules from main prints `not ok 1` and the four failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes together with `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`, `alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like the migration suites it needs docker, and only the image pull touches the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run the new rules in vmalert on a live cluster. The rules file is the same on release-1.6 and release-1.7. The commit cherry-picks onto release-1.6 without conflicts, and that backport is a separate PR with one more commit that drops the `load test_helper` line, because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`. ### Screenshots Not a UI change. ### Downstream repositories A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and its test data. No values, schema, README, image or `ApplicationDefinition` change, and the website does not mention these rules, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member. ```  ## Summary by CodeRabbit - **Bug Fixes** - Refined etcd gRPC failure alerts to target specific failure status codes, reducing alerts for non-failure responses while preserving existing thresholds and timing. - Added coverage for canceled watch streams, unavailable range requests, and combined failure scenarios, including warning and critical alert evaluations. - **Tests** - Added integration validation for rendering etcd alert rules and running them through the alert-testing tool, including failure handling when prerequisites are unavailable.  Backported-from: 432d02e

Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>

Aleksei Sviridkin (lexfrei) · 2026-11-13T11:21:46Z

europrinter (@yankawai) I merged main into your branch (f1067fd) so CI runs against the current base. Your commits are unchanged.

## What this PR does On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant etcd with two of its three members on one node, a tenant Redis with two of its three Sentinels on the node that also ran the master, and several VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect, vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one of those nodes would have cost the etcd or Sentinel quorum, or both copies of a storage tier. Spread constraints keep new pods apart once they are in place (#4986 for etcd, #4989 for monitoring, #4990 for Redis), but a placement that already exists, or that comes from a soft rule or a drain, stays as it is, because the scheduler does not move running pods. This adds `PodReplicasColocated`, a warning in a new `workload-placement` group under `packages/system/monitoring-agents/alerts/`. It fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or `StrimziPodSet` have been on one node for an hour while the cluster has at least as many schedulable nodes as the workload has running pods, so five replicas on three nodes stay quiet. The alert carries `cluster`, `namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`, `exported_instance` and `service` labels the other platform alerts set, and its value is the number of pods on that node. DaemonSets, Jobs and every other owner kind are not in the list, so a DaemonSet surge pod next to its predecessor does not count. etcd-operator gives every member pod its own `EtcdMember` as controller owner ([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)), so grouping by owner would see one pod per group. Members get their names from `GenerateName: <cluster>-` ([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434), [#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)), and members adopted from the old StatefulSet keep names like `etcd-0` ([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)). The rule drops the last dash-separated part of the member name and reports the group as `EtcdCluster <name>`. With the etcd package that is one group per tenant namespace, and two etcd clusters in one namespace would still be told apart. CNPG instance pods carry a controller reference to their `Cluster` ([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)), and Strimzi pods one to their `StrimziPodSet` ([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)). Owner and node both come from `kube_pod_info`. kube-state-metrics fills `created_by_kind` and `created_by_name` from the controller owner reference and `node` from `spec.nodeName` ([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)). `kube_pod_owner` and `kube_pod_status_phase` have no node label of their own, and the kube-state-metrics scrape adds `node` from `__meta_kubernetes_pod_node_name` with `honorLabels: true` (`templates/kube-state-metrics-scrape.yaml`), so on those series `node` is the node of the kube-state-metrics pod. The test data keeps that label on the phase series. A pod counts while `kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending pod on the same node does not. `kube_pod_labels` does carry every pod label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not depend on it. A node counts as schedulable unless it has a `NoSchedule` or `NoExecute` taint. A cordon shows up as the `node.kubernetes.io/unschedulable:NoSchedule` taint ([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)), and not ready and unreachable nodes get theirs from their conditions ([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)), so one clause covers all three, and a tainted control plane does not count either. Pods that a drain had to put together on the remaining nodes therefore stay quiet while the node is out, and the hour starts once it is back. The alert waits an hour. A replacement pod runs next to its terminating predecessor only for the predecessor's grace period, and node maintenance takes the drained node out of the count, while a placement that has held for an hour does not fix itself. The alert is about a risk rather than an outage, so the wait costs nothing. The VMAlert sets both `remoteWrite` and `remoteRead`, so the pending state survives a vmalert restart ([vmalert docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)). Not covered are workloads that run one pod per owner, such as ClickHouse with one StatefulSet per replica ([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53)) and the KubeVirt VMs behind a tenant Kubernetes node group. They would need grouping by pod labels, which this rule avoids. The node count is cluster-wide, so a workload that a node selector or affinity restricts to fewer nodes than it has pods fires too. `hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and replays it through `vmalert-tool` v1.136.0 pinned by digest, the release vm-operator v0.68.4 deploys by default ([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38)) and the same image #4963 and #4985 use. `hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases with 24 checks. Two of three StatefulSet pods on one node fire at 61 minutes and not at 59, with labels and annotations pinned. Spread pods, five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of one Job, a Failed pod next to a running one, and one pod each in two clusters that share workload and node names stay quiet. etcd members fire as one `EtcdCluster` per namespace, for generated and adopted member names. A replacement pod next to its terminating predecessor for twenty minutes stays quiet, and the same overlap held past the hour fires and resolves when it ends. A cordoned node keeps the alert quiet until it is uncordoned, and a node with unreachable taints keeps it quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods fire with their owner kind. ```console $ bats hack/workload-placement-alerts.bats 1..1 ok 1 workload placement alerts pass their vmalert-tool cases ``` With the expression replaced by one that never fires, the suite prints `not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all the checks that expect an alert. On main, where the rule file does not exist, the suite fails as well. Each of these changes to the rule fails at least one check, with the number of failing checks in brackets. `for: 15m` (5), no phase filter (1), no node count (6), no `cluster` label (6), DaemonSet and Job in the kind list (3), no taint clause (4), `<` instead of `<=` (5), etcd members grouped per member (1). With Bats 1.14.0 at the commit CI pins, the suite passes together with `alert-severity-contract`, `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`, `image-pin-consistency` and `check-commit-trailers`, 109 of 109, and the severity contract lists the new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm template` with Helm 3.17.3 renders the VMRule the same as the file. Like the migration suites the new suite needs docker, and only the image pull touches the network. I have not run the rule in vmalert on a live cluster. On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts template, the VMAlert and vm-operator v0.68.4 are the same, and etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member names. The new files apply there without changes. That backport is a separate PR with one more commit that drops the `load test_helper` line, because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`. ### Screenshots Not a UI change. ### Downstream repositories A new alert file under `packages/system/monitoring-agents/alerts/`, a Bats suite under `hack/` and its test data. No values, schema, README, image or `ApplicationDefinition` change, and the website does not list platform alerts by name, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them. ```  ## Summary by CodeRabbit * **New Features** * Added monitoring for workloads whose running pod replicas are colocated on the same node. * Alerts include workload, node, cluster, and severity details, along with quorum-loss guidance. * Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets, and EtcdMembers. * **Tests** * Added validation for workload-placement alerts across scheduling, availability, cluster-isolation, and pod-replacement scenarios.

Aleksei Sviridkin (lexfrei)

europrinter (@yankawai) LGTM, CI and E2E are green on this head.

europrinter (yankawai) requested review from IvanHunters, Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), mattia-eleuteri, myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners October 19, 2026 16:42

github-actions Bot added area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Oct 19, 2026

IvanHunters approved these changes Oct 23, 2026

View reviewed changes

Merge branch 'main' into fix/linstor-error-report-alerts-controller-move

f1067fd

Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>

Aleksei Sviridkin (lexfrei) approved these changes Nov 13, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) merged commit 51282a1 into cozystack:main Nov 13, 2026
24 checks passed

Navigation Menu

Repository navigation

fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves - #4963

fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves#4963
Aleksei Sviridkin (lexfrei) merged 2 commits into
cozystack:mainfrom
yankawai:fix/linstor-error-report-alerts-controller-move

europrinter (yankawai) commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading

Review skipped

Uh oh!

IvanHunters left a comment

Uh oh!

Aleksei Sviridkin (lexfrei) commented Nov 13, 2026

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

europrinter (yankawai) commented Oct 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Screenshots

Downstream repositories

Release note

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

IvanHunters left a comment

Choose a reason for hiding this comment

Uh oh!

Aleksei Sviridkin (lexfrei) commented Nov 13, 2026

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

europrinter (yankawai) commented Oct 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 19, 2026 •

edited

Loading