Repository navigation
Commit 5632596
committed
feat(monitoring): alert when pods of one workload share a node
Two etcd members, two Redis Sentinels or both replicas of a storage
tier can end up on one node through a soft spread rule, a drain, or a
placement made before a spread rule existed. Nothing reported it, and
the scheduler does not move running pods, so it stays like that until
a pod is recreated. One node failure then costs the quorum or every
copy at once.
etcd-operator owns each member pod by its own EtcdMember, so members
are grouped by the EtcdCluster name their names start with. Owner and
node are read from kube_pod_info because kube_pod_owner and
kube_pod_status_phase get the node of the kube-state-metrics pod as a
target label. Nodes with a NoSchedule or NoExecute taint do not count,
which covers cordoned, not ready and unreachable nodes. The alert waits
an hour, longer than a replacement pod runs next to its terminating
predecessor or a node takes to drain and come back.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>1 parent d1d4e92 commit 5632596
3 files changed
Lines changed: 679 additions & 0 deletions
File tree
- hack
- testdata/workload-placement-alerts
- packages/system/monitoring-agents/alerts
0 commit comments