fix: alert state is not restored when Prometheus restarts#18364
Closed
denysvitali wants to merge 1 commit into
Closed
fix: alert state is not restored when Prometheus restarts#18364denysvitali wants to merge 1 commit into
denysvitali wants to merge 1 commit into
Conversation
roidelapluie
added a commit
to roidelapluie/prometheus
that referenced
this pull request
Mar 26, 2026
QueryForStateSeries built Select matchers from the raw rule labels,
which can contain Go template expressions such as
`instance_{{ $labels.instance }}`. The stored ALERTS_FOR_STATE series
carry the per-instance evaluated values (e.g. `instance_0`), so the
unevaluated template string never matched, leaving seriesByLabels empty
and silently skipping restoration for every active alert.
Fix by omitting any label whose value contains `{{` from the matcher
list. Static labels (including `__name__` and `alertname`) are never
templated and continue to scope the query to the correct rule. The
in-memory lookup against evaluated alert labels that follows is
unaffected, so the single-query-per-rule optimisation introduced in
prometheus#13980 is fully preserved.
Fixes prometheus#16883
Ref prometheus#13980
Ref prometheus#18364
Signed-off-by: Julien Pivotto <[email protected]>
Member
|
What do you think of #18375 |
Author
|
Seems like a very good solution @roidelapluie ! Thanks! |
Author
|
Closing as #18375 is merged. Thank you so much @roidelapluie - you have no idea on how frustrating this bug was and how long it took us to find the root cause of our AlertManager flood on each Thanos restart! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #13980 an optimization in the
ALERTS_FOR_STATElogic caused the alert restoration to fail when template variables are used.This PR intentionally doesn't solve this (yet) because the fix might cause performance issues that the problematic PR tried to address.
I strongly believe, as shown from the test here, that the performance fix never really worked - and this bug got unnoticed for a long time.
This investigation is the result of multiple months of debugging this behavior at a previous company I worked at together with @verejoel - and it's one of the reasons why he gave this talk at PromCon 2025 (although he didn't mention this issue there).
The intent of this PR is to get acknowledgment on the problem, and understand together what is the best way forward to solve it. Ideally a revert of the problematic PR would be enough, but that would solve the performance issues that @gotjosh tried to solve in #13980.
This PR will be used to fix #16883
cc/ @bwplotka