What did you do?
When an AlertingRule is defined using labels that contain a Go template (e.g: instance_ext=instance_{{ $labels.instance }}, storing the state works correctly (label is templated), but retrieving the state is broken as it queries for the un-evaluated template.
This leads to refirings when the ruler is restarted as the previous alert state isn't considered anymore.
What did you expect to see?
The state is restored correctly, by querying for the right labels
What did you see instead? Under which circumstances?
The store is queried with "instance_{{ $labels.instance }}" - which is wrong.
This happens always when at least one template is defined in the alert - which breaks its state.
This bug was introduced in 1dd0bff (#13980) (v2.53.0 / 0.53.0). This propagated to Thanos with version v0.36.0-rc.0 (last known good version is v0.35.1).
By reverting the commit - I was able to confirm that the state is restored correctly when templated labels are used.
In my fork (branch feature/add-test-restore-label-template) I've created a commit / test to show this behavior:
https://github.com/denysvitali/prometheus/actions/runs/16318662310/job/46090585204
=== RUN TestForStateRestoreWithTemplate/queryOffset_0s/normal_restore_(alerts_were_not_firing)
manager_test.go:678: {__name__="ALERTS_FOR_STATE", alertname="HTTPRequestRateLow", group="canary", instance="0", instance_ext="instance_0", job="app-server", severity="critical"} => 0 @[0]
manager_test.go:722: unexpected Select calls (-want +got):
[]storage.SelectCall{
{
SortSeries: false,
Hints: nil,
Matchers: []*labels.Matcher{
&{Name: "__name__", Value: "ALERTS_FOR_STATE"},
&{Name: "alertname", Value: "HTTPRequestRateLow"},
&{
Type: s"=",
Name: "instance_ext",
- Value: "instance_0",
+ Value: "instance_{{ $labels.instance }}",
... // 1 ignored field
},
&{Name: "severity", Value: "critical"},
},
},
}
manager_test.go:760:
Error Trace: /home/dvitali/Documents/git/prometheus/rules/manager_test.go:760
Error: Not equal:
expected: 0
actual : 300
Test: TestForStateRestoreWithTemplate/queryOffset_0s/normal_restore_(alerts_were_not_firing)
Messages: 'for' state restored time is wrong
--- FAIL: TestForStateRestoreWithTemplate/queryOffset_0s/normal_restore_(alerts_were_not_firing) (0.00s)
Expected :0
Actual :300
System information
No response
Prometheus version
Any version after and including v2.53.0.
If you really need a version, I was still able to reproduce this from main (a9438f0 / v3.5.0)
What did you do?
When an
AlertingRuleis defined using labels that contain a Go template (e.g:instance_ext=instance_{{ $labels.instance }}, storing the state works correctly (label is templated), but retrieving the state is broken as it queries for the un-evaluated template.This leads to refirings when the ruler is restarted as the previous alert state isn't considered anymore.
What did you expect to see?
The state is restored correctly, by querying for the right labels
What did you see instead? Under which circumstances?
The store is queried with
"instance_{{ $labels.instance }}"- which is wrong.This happens always when at least one template is defined in the alert - which breaks its state.
This bug was introduced in 1dd0bff (#13980) (v2.53.0 / 0.53.0). This propagated to Thanos with version
v0.36.0-rc.0(last known good version isv0.35.1).By reverting the commit - I was able to confirm that the state is restored correctly when templated labels are used.
In my fork (branch
feature/add-test-restore-label-template) I've created a commit / test to show this behavior:https://github.com/denysvitali/prometheus/actions/runs/16318662310/job/46090585204
System information
No response
Prometheus version
Any version after and including v2.53.0.
If you really need a version, I was still able to reproduce this from
main(a9438f0 / v3.5.0)