notifier: stop queue filling due to single failed AM#14099
Closed
krajorama wants to merge 1 commit intoprometheus:mainfrom
Closed
notifier: stop queue filling due to single failed AM#14099krajorama wants to merge 1 commit intoprometheus:mainfrom
krajorama wants to merge 1 commit intoprometheus:mainfrom
Conversation
Ref: prometheus#7676 Signed-off-by: György Krajcsovits <[email protected]>
|
Nice work on the unit test. Principally I'd prefer 2nd, separate queues. This would allow completely independent processing and overflowing of queues. |
|
an alternative 3rd solution with a shared queue I was considering was having a ring buffer, where there's a go routine for every alert manager that has a pointer into the ring buffer. Whenever the insert operation into the ring buffer catches up with an alert manager pointer, it would move that pointer forward, effectively dropping the oldest alert. |
Member
|
Thanks for this! (I always appreciate creative tests) In an ideal word, An SD is to be used for those AM instead of static config, so the faulty ones are excluded and the notifier doesn't have to worry about that. I'm afraid a "quarantine logic" would look like re-implementing the SD logic... |
Member
|
Hello from the bug-scrub! @krajorama do you think you will come back to this? |
Member
Author
Probably not :( Bigger project than what I have bandwidth for. |
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independant Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configurarion reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independant Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configurarion reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configurarion reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Mar 31, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 1, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 7, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 7, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 8, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 8, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 8, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 8, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Apr 22, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
May 1, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
May 1, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
May 1, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
May 2, 2025
Independent Alertmanager queues avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The buffered queues are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Aug 29, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 11, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 12, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 12, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 22, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 25, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 25, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 25, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Nov 28, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Dec 2, 2025
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Jan 12, 2026
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Jan 12, 2026
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
siavashs
added a commit
to siavashs/prometheus
that referenced
this pull request
Jan 13, 2026
Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]>
Naman-B-Parlecha
pushed a commit
to Naman-B-Parlecha/prometheus
that referenced
this pull request
Jan 20, 2026
* notifier: unit test for dropping throughput on stuck AM Ref: prometheus#7676 Signed-off-by: György Krajcsovits <[email protected]> Signed-off-by: Siavash Safi <[email protected]> * chore(notifier): remove year from copyrights Signed-off-by: Siavash Safi <[email protected]> * feat(notifier): independent alertmanager sendloops Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from prometheus#14099 Closes prometheus#7676 Signed-off-by: machine424 <[email protected]> Signed-off-by: Siavash Safi <[email protected]> --------- Signed-off-by: György Krajcsovits <[email protected]> Signed-off-by: Siavash Safi <[email protected]> Signed-off-by: machine424 <[email protected]> Co-authored-by: György Krajcsovits <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Adds a unit test to emulate the problem of throughput dropping in #7676
Solution ideas (not implemented yet):
Fixes: #7676