Repository navigation
[piraeus-operator] Fix LINSTOR satellite alert labels and scrape flapping false positives - #2265
Conversation
…ping false positives Signed-off-by: sasha-sup <[email protected]>
dosubot
Bot
added
the
size/S
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses two issues within the Piraeus Operator's LINSTOR alert definitions. It refines the alert messages for satellite-related issues by using the correct label for host identification and significantly reduces false positive alerts for satellite error rates by ensuring the LINSTOR controller's stability is considered in the alert's evaluation logic. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
📝 WalkthroughWalkthroughReplaced the controller reachability alert with a deployment-availability based Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates LINSTOR alert definitions, specifically changing label references from name to hostname and modifying the linstorSatelliteErrorRate expression. The review highlights a critical bug in the linstorSatelliteErrorRate alert expression where mismatched job labels (linstor-controller vs. linstor-satellite) will prevent it from firing. Additionally, it suggests improving the operator experience by re-adding --since 15m to the suggested linstor error-reports list command in the alert description.
| increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0 | ||
| and on(instance, job) | ||
| min_over_time(up{job="linstor-controller"}[15m]) == 1 |
There was a problem hiding this comment.
This alert expression appears to have a bug that will prevent it from ever firing. The job labels on both sides of the and operator do not match.
According to packages/system/linstor/templates/podscrape.yaml, metrics from the linstor-controller (like linstor_error_reports_count) get the job="linstor-satellite" label. To check for controller scrape stability, you should therefore use up{job="linstor-satellite"}.
The current expression uses up{job="linstor-controller"}, which refers to satellite scrapes. This mismatch will cause the and on(instance, job) to fail.
increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0
and on(instance, job)
min_over_time(up{job="linstor-satellite"}[15m]) == 1| Use "linstor error-reports list --nodes {{ $labels.name }} --since 15minutes" to see them. | ||
| expr: increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0 | ||
| LINSTOR Satellite "{{ $labels.hostname }}" reports {{ $value }} errors in the last 15 minutes. | ||
| Use "linstor error-reports list --nodes {{ $labels.hostname }}" to inspect the reports. |
There was a problem hiding this comment.
For a better operator experience, consider adding --since 15m to the suggested command. The alert triggers on errors within the last 15 minutes, so aligning the command with this timeframe will help in quickly identifying the relevant error reports.
Use "linstor error-reports list --nodes {{ $labels.hostname }} --since 15m" to inspect the reports.Signed-off-by: sasha-sup <[email protected]>
…alerts Signed-off-by: sasha-sup <[email protected]>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/system/piraeus-operator/alerts/piraeus-datastore.yaml`:
- Line 14: The alert expression for linstorControllerUnavailable currently only
fires when
kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"}
< 1, which misses the case where the metric/series is absent; change the rule
expression to use the standard pattern
"absent(kube_deployment_status_replicas_available{namespace=\"cozy-linstor\",deployment=\"linstor-controller\"})
or
kube_deployment_status_replicas_available{namespace=\"cozy-linstor\",deployment=\"linstor-controller\"}
< 1" so the alert fires when the metric is missing or when available replicas
are below 1.
- Line 22: Update the Prometheus alert expressions to detect both scrape
failures and missing targets by combining the existing equality check with an
absent() check; specifically modify the expr for the alert referencing
up{job="linstor-controller"} (and likewise the expr for the drbdReactorOffline
alert) so it fires when up{job="linstor-controller"} == 0 OR when
absent(up{job="linstor-controller"}), ensuring both failed scrapes and
disappeared targets trigger the alert.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c67556a8-61d0-49dd-befa-3ad05bc7928b
📒 Files selected for processing (1)
packages/system/piraeus-operator/alerts/piraeus-datastore.yaml
| LINSTOR Controller is not reachable. | ||
| expr: up{job="linstor-controller"} == 0 | ||
| LINSTOR Controller deployment has no available replicas. | ||
| expr: kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1 |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
In PromQL, what does kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1return when the matching series is absent, and isabsent(...) or metric < 1 the standard pattern to alert on a missing Deployment metric?
💡 Result:
In PromQL, the query kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1 returns an empty result (no series) when the matching series is absent. An empty vector result in an alerting rule is interpreted by Prometheus as "everything is fine," so no alert fires. This is standard PromQL behavior for selectors that match no time series. Yes, absent(kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"}) or kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1 is the standard pattern to alert on both a missing Deployment metric (via absent) and when available replicas are low. The absent function returns a series with value 1 (with derived labels) if no matching series exist, or empty if any exist; the or combines it with the low-replicas condition to trigger an alert in either case. This handles cases where the metric disappears (e.g., kube-state-metrics issues or Deployment deletion) without silently failing.
Citations:
- 1: https://promlabs.com/blog/2023/09/13/dealing-with-missing-time-series-in-prometheus/
- 2: Signal to Prometheus that a query should match something prometheus/prometheus#11339
- 3: https://prometheus.io/docs/prometheus/latest/querying/operators/
- 4: https://grafana.com/docs/grafana/latest/alerting/best-practices/missing-data
- 5: https://stackoverflow.com/questions/51358390/prometheus-check-if-a-metric-is-present
- 6: https://prometheus.io/docs/prometheus/latest/querying/functions/
- 7: https://groups.google.com/g/prometheus-users/c/TMPlzYWJ7b4
- 8: https://zenn.dev/nissy_dev/articles/kube-state-metrics-alert?locale=en
linstorControllerUnavailable still has a missing-series blind spot.
If the goal is "controller gone or unavailable," Line 14 only covers the case where kube_deployment_status_replicas_available exists and is < 1. If the Deployment disappears entirely or that series stops being exposed (e.g., kube-state-metrics issues), the expression returns an empty vector and Prometheus interprets this as "everything is fine"—so this critical alert stays silent.
Suggested fix using the standard pattern
- expr: kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1
+ expr: |
+ absent(kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"})
+ or kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1Use absent(...) or metric < 1 to trigger an alert in either case: when the metric disappears or when available replicas are low.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| expr: kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1 | |
| expr: | | |
| absent(kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"}) | |
| or kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"} < 1 |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/system/piraeus-operator/alerts/piraeus-datastore.yaml` at line 14,
The alert expression for linstorControllerUnavailable currently only fires when
kube_deployment_status_replicas_available{namespace="cozy-linstor",deployment="linstor-controller"}
< 1, which misses the case where the metric/series is absent; change the rule
expression to use the standard pattern
"absent(kube_deployment_status_replicas_available{namespace=\"cozy-linstor\",deployment=\"linstor-controller\"})
or
kube_deployment_status_replicas_available{namespace=\"cozy-linstor\",deployment=\"linstor-controller\"}
< 1" so the alert fires when the metric is missing or when available replicas
are below 1.
| annotations: | ||
| description: | | ||
| LINSTOR Controller metrics endpoint is not being scraped successfully. | ||
| expr: up{job="linstor-controller"} == 0 |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
In Prometheus, does up{job="linstor-controller"} == 0fire when a target disappears from service discovery, or does theup series become absent? What alert expression is usually recommended to catch both failed scrapes and missing targets?
💡 Result:
In Prometheus, the expression up{job="linstor-controller"} == 0 does not fire when a target disappears from service discovery. Instead, the up series for that target becomes absent because Prometheus stops scraping it and no longer generates the up metric for the missing target. The up metric is only present for targets that Prometheus attempts to scrape, set to 1 on successful scrapes and 0 on failed scrapes. When a target is removed from service discovery, no scrape attempt occurs, so no up series exists. The usually recommended alert expression to catch both failed scrapes (up==0) and missing targets is absent(up{job="linstor-controller"}). This fires if no up series match the selector, indicating the entire job has no targets (all disappeared). For individual missing targets within a job that still has some up targets, use up{job="linstor-controller"} == 0 or absent(up{job="linstor-controller"}), but absent(up{job="linstor-controller"}) alone covers the missing case while up == 0 covers scrape failures on existing targets. Common practice includes separate alerts: one for up == 0 for: 5m (target down) and one for absent(up{job="..."}) (job/targets missing).
Citations:
- 1: https://chejinying.com/tech/posts/prometheus/up/
- 2: https://stackoverflow.com/questions/55162188/prometheus-how-up-metrics-works
- 3: https://prometheus.io/docs/concepts/jobs_instances
- 4: https://www.robustperception.io/absent-alerting-for-jobs
🏁 Script executed:
# First, check if the file exists and read the relevant context around line 22
if [ -f "packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" ]; then
echo "=== File content around line 22 ==="
sed -n '15,30p' packages/system/piraeus-operator/alerts/piraeus-datastore.yaml | cat -n
else
echo "File not found at packages/system/piraeus-operator/alerts/piraeus-datastore.yaml"
# Try to find the file
find . -name "piraeus-datastore.yaml" -type f
fiRepository: cozystack/cozystack
Length of output: 888
🏁 Script executed:
# Check for other similar up metric expressions in the file
if [ -f "packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" ]; then
echo "=== All up metric expressions in the file ==="
grep -n "up{" packages/system/piraeus-operator/alerts/piraeus-datastore.yaml
fiRepository: cozystack/cozystack
Length of output: 300
🏁 Script executed:
# Get the full alert structure for context
if [ -f "packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" ]; then
echo "=== Full YAML file content ==="
cat -n packages/system/piraeus-operator/alerts/piraeus-datastore.yaml
fiRepository: cozystack/cozystack
Length of output: 6735
Add absent() to catch missing targets in addition to scrape failures.
The expression up{job="linstor-controller"} == 0 only fires when a scrape fails on an existing target. When a target disappears entirely from service discovery, Prometheus stops attempting scrapes and the up metric becomes absent rather than 0—so this alert won't trigger even though metrics are unavailable.
Use the following expression to catch both conditions:
Suggested fix
- expr: up{job="linstor-controller"} == 0
+ expr: absent(up{job="linstor-controller"}) or up{job="linstor-controller"} == 0Note: The same issue exists in the drbdReactorOffline alert at line 73.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| expr: up{job="linstor-controller"} == 0 | |
| expr: absent(up{job="linstor-controller"}) or up{job="linstor-controller"} == 0 |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/system/piraeus-operator/alerts/piraeus-datastore.yaml` at line 22,
Update the Prometheus alert expressions to detect both scrape failures and
missing targets by combining the existing equality check with an absent() check;
specifically modify the expr for the alert referencing
up{job="linstor-controller"} (and likewise the expr for the drbdReactorOffline
alert) so it fires when up{job="linstor-controller"} == 0 OR when
absent(up{job="linstor-controller"}), ensuring both failed scrapes and
disappeared targets trigger the alert.
|
This PR depends on #2264 being merged first. Currently in |
Kirill Ilin (sircthulhu)
added
kind/backport
24078dc
into
cozystack:main
|
Successfully created backport PR for |
|
Successfully created backport PR for |
…n the controller pod moves (#4963) ## What this PR does `linstorSatelliteErrorRate` fires for every node that has any old LINSTOR error report whenever the LINSTOR controller pod moves to another node, and it keeps firing for 15 minutes. `linstor_error_reports_count` is exported by the controller, so its series carry the controller pod's `instance` and `controller_node`, and a moved pod starts new series that hold the existing counts. VictoriaMetrics' [`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase) starts from the last sample before the window, and when a series has none, [`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901) counts it from zero whenever its first value is below ten times the next change plus one. That is the case the docs describe as `increase` ignoring "the first value in a series if it is too big" under [`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure), and report counts are never that big, so every new series counts all of its reports as new. The `up` guard from #2265 matches `on(instance, job)`, and the new controller is up, so it lets them through. Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the upgrade the controller pod moved to another node, and the alert fired for all three nodes with values 1, 1 and 2 for 15 minutes, while `linstor error-reports list --since 1h` was empty and the four reports that exist were weeks old. The day before, it fired the same way for about 15 minutes when only the `cluster` external label of vmagent changed. A satellite coming back to the metric does the same. In LINSTOR v1.33.3 the controller asks every satellite for its reports on each scrape and leaves out any satellite that is not connected ([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)), so after a node reboot or any other disconnect longer than five minutes the satellite's old count is counted again. Both rules now take the count per `hostname`, hold the last value for an hour with `last_over_time`, and run `increase()` over that as a one-minute [subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries), whose inner points VictoriaMetrics aligns to that step. Keyed on `hostname`, a moved controller or a changed label continues the same series, and the hold bridges a disconnect or a slow controller start of up to an hour. The `up` guard matches `on()`, since the aggregated result has no `instance` any more. A new report still fires with the same value and stays for 15 minutes, at most a minute later than before because of the one-minute step. The alerts now carry `hostname` and `severity`, and no longer the controller's `instance`, `job` and `controller_node`, the `module`, or the external labels of the scrape. `linstorControllerErrorRate` gets the same form. Its `hostname` is the controller pod name and the piraeus controller keeps its reports in an [`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112), so a new pod starts with no reports and a move alone does not trip it, but a label change does whenever the controller has reports. A gap longer than the hour is not covered. After it the series starts over, and as before the `up` guard holds the alert back only if the controller target recorded failed scrapes during the gap. I checked it against the stored metrics of the cluster above with read-only queries to vmselect. Over the upgrade the current expression returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second step, the same interval as vmalert's `ALERTS` series, and over the label change from 20:27:30 to 20:42:00. The new expression returns nothing over both windows, and without `> 0` it returns each of the three nodes with 0 at every step, 88 and 121 points. The same storage has six gaps of 8 to 113 minutes over one night with no failed scrape recorded in them. Replaying vmalert's instant query of the `increase()` part every 30 seconds in the 154 evaluations after those gaps, the current expression returns at least one node in 99, the `hostname` key without the hold in 84, and the new expression in 30, all of them after the 113-minute gap. Retention there does not reach back to a real new report, so the firing side is covered by the tests. `hack/linstor-error-report-alerts.bats` replays both rules through `vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics release victoria-metrics-operator v0.68.4 [deploys by default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38), with 30-second samples and 15-second evaluations like the platform's vmagent and vmalert. The suite needs docker, like the migration suites, and only the image pull touches the network. `hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases in five groups, which cover a controller move followed by a new report, a changed external label for both rules, one satellite away for 40 minutes and one that comes back with a new report, a new report and a first report on a node for both rules, and a controller that cannot be scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of them because old reports fire again after the move, the label change or the disconnect, and the other 4 on the label set alone, since the current alerts carry the controller's labels. Each of seven mutations of the new expressions turns at least one case red, namely no hold, a 15-minute hold, the guard on `instance`, no guard, the key on `hostname` and `instance`, the key on `hostname` and `cluster`, and the controller rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite with `alert-severity-contract`, `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `image-pin-consistency` passes 74 of 74, and `helm template` renders the same 15 rules as main. I have not run the new rules in vmalert on a live cluster. The commit cherry-picks onto release-1.7 as is, and the suite passes there. On release-1.6 it applies without conflicts, but that branch runs `hack/*.bats` through `hack/cozytest.sh` and has no `hack/test_helper.bash`, so the suite there has to drop its `load test_helper` line. ### Screenshots Not a UI change. ### Downstream repositories A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats suite under `hack/` and its test data. No values, schema, README, image, make target or `ApplicationDefinition` change, and the website does not mention these rules or the metric, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved LINSTOR controller and satellite error alerts to account for recent error reports over a 15-minute window, even when metrics are intermittent or scrape labels change. * Updated alert matching to better handle controller moves between nodes and satellites that disappear from metrics while disconnected. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Summary
This fixes two issues in the LINSTOR alerts shipped by
cozy-piraeus-operator:linstorSatelliteErrorRateuses a non-existentnamelabel in annotations, which results inSatellite ""in notifications.linstorSatelliteErrorRatecan produce false positives when thelinstor-controllerscrape flaps and historicallinstor_error_reports_countcounters reappear inside the alert window.Test plan
{{ $labels.name }}to{{ $labels.hostname }}for LINSTOR satellite alerts.linstorSatelliteErrorRateexpression now requires stableup{job="linstor-controller"}for the full 15 minute window.linstorSatelliteErrorRatealerts stopped firing once controller scrape recovered.Release note
Summary by CodeRabbit
Improvements
New Features