Repository navigation
[Backport release-1.6] fix(monitoring): send alerts to every Alertmanager replica - #4890
Merged
myasnikovdaniil merged 1 commit intoOct 14, 2026
Conversation
vmalert had a single notifier, the headless vmalertmanager-alertmanager Service. It resolves to all three pods, but each alert reached one replica at a time and moved to another when vmalert reconnected. On clusters running 1.6.4 all active alerts sat on one replica while the other two had none, and on one of them the alert count of each replica changed about 170 times a day. The Alertmanager cluster deduplicates notifications only when every replica receives every alert. The replica an alert left let it expire and sent a resolved notification, and the replica it reached sent a new firing one. vmalert now lists each Alertmanager pod as its own notifier, using the <name>-<i>.<service>.<namespace>.svc address the operator itself gives StatefulSet pods. The replica count comes from one helper that also sets spec.replicaCount of the VMAlertmanager, so the two cannot drift. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]> (cherry picked from commit 9d4221b)
github-actions
Bot
added
the
size/M
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/monitoring
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Backport of #4889 to
release-1.6. The commit is cherry-picked from 9d4221b. The only conflict was intemplates/_helpers.tpl, which is shorter on this branch, so the newmonitoring.alertmanager.replicashelper is appended to the release-1.6 file and nothing else from main comes along.The vmalert and VMAlertmanager templates here are the same as on main before the fix, a single notifier on the shared Service and
replicaCount: 3, so the change applies as is. The monitoring chart tests pass on this branch, 7 suites and 50 tests, the new suite included.Screenshots
Not a UI change.
Downstream repositories
Same diff as #4889, two templates, the helpers and a test file of
packages/system/monitoring, so no trigger map entry applies.Release note