Skip to content

Change discovery manager syncCh to buffered channel#13677

Closed
Spaceman1701 wants to merge 4 commits intoprometheus:mainfrom
Spaceman1701:fix/alertmanager-sd
Closed

Change discovery manager syncCh to buffered channel#13677
Spaceman1701 wants to merge 4 commits intoprometheus:mainfrom
Spaceman1701:fix/alertmanager-sd

Conversation

@Spaceman1701
Copy link
Copy Markdown
Contributor

This is intended to improve the behavior of Alertmanager service discovery when the notifier is under high load. In normal conditions, buffering this channel doesn't make much of a difference as both the discovery manager and the receiver have plenty of chances to write/read the channel. When the receiver is under heavy load, the buffered channel guarantees it will see the results of a recent discovery manager iteration. I don't think this has any unforeseen impacts. I've opened #13676 with details of the problem I'm trying to fix.

Fixes #13676

@Spaceman1701
Copy link
Copy Markdown
Contributor Author

I've updated this so it buffers both the new discovery manager and the legacy discovery manager since they both seem to have the same problem.

Signed-off-by: Ethan Hunter <[email protected]>
Copy link
Copy Markdown
Member

@jesusvazquez jesusvazquez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for raising and looking at this issue.

For future reviewers, most of the conversation around this PR is happening in the issue linked. We left a comment there #13676 (comment)

@bwplotka
Copy link
Copy Markdown
Member

Hello from the bug scrub!

Looks like #14174 fixed it already. If there's anything else to do let us know on Slack or separate issue, thanks! 💪🏽

@bwplotka bwplotka closed this Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alertmanager service discovery does not update under heavy load

3 participants