Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Alert admins if containers are crashing/restarting #9793

Closed
emidoots opened this issue Apr 13, 2020 · 3 comments · Fixed by #17239
Closed

Alert admins if containers are crashing/restarting #9793

emidoots opened this issue Apr 13, 2020 · 3 comments · Fixed by #17239
Assignees
@emidoots
Labels
container-metrics monitoring monitoring-long-tail
Milestone
Backlog

Comments

@emidoots
Copy link
Member

when containers restart, alerts should fire. Should work in all deployment environments.
@emidoots emidoots added this to the 3.15 milestone Apr 13, 2020
@emidoots emidoots self-assigned this Apr 13, 2020
@beyang beyang mentioned this issue Apr 13, 2020
Closed
@uwedeportivo
Copy link
Contributor

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.15 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

@emidoots emidoots mentioned this issue Apr 15, 2020
Merged
@emidoots emidoots added the planned/3.15 Issues that were planned for the given milestone. Used by cmd/tracking-issue. label Apr 15, 2020
@emidoots emidoots modified the milestones: 3.15, 3.16 Apr 15, 2020
@emidoots emidoots added container-metrics planned/3.16 Issues that were planned for the given milestone. Used by cmd/tracking-issue. and removed planned/3.15 Issues that were planned for the given milestone. Used by cmd/tracking-issue. labels Apr 16, 2020
@beyang beyang mentioned this issue Apr 21, 2020
Closed
47 tasks
@tsenart tsenart mentioned this issue Apr 23, 2020
Closed
20 tasks
@emidoots emidoots removed this from the 3.16 milestone Apr 24, 2020
@emidoots emidoots mentioned this issue Apr 24, 2020
Closed
22 tasks
@emidoots emidoots removed the planned/3.16 Issues that were planned for the given milestone. Used by cmd/tracking-issue. label Apr 24, 2020
@bobheadxi
Copy link
Member

bobheadxi commented May 29, 2020 •
edited
Loading

This should be covered in all environments except server deployments (since https://github.com/sourcegraph/sourcegraph/issues/9791) by the cadvisor_container_restart_count metric and the associated alert, right?

@emidoots
Copy link
Member Author

I think the only thing left to do here is confirm this works in k8s and docker-compose deployments. Then, we should promote the alert to a critical one (so admins would be paged) and ensure it won't fire on sourcegraph/server deployments.

@emidoots emidoots changed the title monitoring for container restarts Alert admins when containers crash/restart Jun 23, 2020
@emidoots emidoots changed the title Alert admins when containers crash/restart Alert admins if containers are crashing/restarting Jun 23, 2020
@emidoots emidoots added this to the Backlog milestone Jul 1, 2020
@bobheadxi bobheadxi mentioned this issue Jan 13, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@emidoots emidoots

Labels
container-metrics monitoring monitoring-long-tail
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

monitoring: cadvisor observables review sourcegraph/sourcegraph-public-snapshot
3 participants
@uwedeportivo @emidoots @bobheadxi