Prerequisites
Bug Description
If a node has multiple pods running, the node drainer add multiple events with different combinations of the same pod as seen below:
NodeDraining AwaitingPodCompletion 99s (x2 over 2m19s) nvsentinel-node-drainer Waiting for following pods to finish: [lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5 alloy/alloy-metrics-55d6fbdbd-vrp2h alloy/dgxc-alloy-gateway-0]
NodeDraining AwaitingPodCompletion 39s (x11 over 3m4s) nvsentinel-node-drainer Waiting for following pods to finish: [alloy/alloy-gateway-0 alloy/alloy-metrics-55d6fbdbd-vrp2h lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5]
NodeDraining AwaitingPodCompletion 19s (x9 over 2m54s) nvsentinel-node-drainer Waiting for following pods to finish: [lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5 alloy/alloy-gateway-0 alloy/dgxc-alloy-metrics-55d6fbdbd-vrp2h]
Ideally the node drainer should only add a single event with the list ordered
Component
Fault Management
Steps to Reproduce
- Run multiple pods on a node in different namespaces
- Configure the node drainer with allow completion
- Trigger a fatal health event
Environment
- NVSentinel version: v0.3.0
- Kubernetes version: any
- Deployment method: ArgoCD
Logs/Output
NodeDraining AwaitingPodCompletion 99s (x2 over 2m19s) nvsentinel-node-drainer Waiting for following pods to finish: [lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5 alloy/alloy-metrics-55d6fbdbd-vrp2h alloy/dgxc-alloy-gateway-0]
NodeDraining AwaitingPodCompletion 39s (x11 over 3m4s) nvsentinel-node-drainer Waiting for following pods to finish: [alloy/alloy-gateway-0 alloy/alloy-metrics-55d6fbdbd-vrp2h lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5]
NodeDraining AwaitingPodCompletion 19s (x9 over 2m54s) nvsentinel-node-drainer Waiting for following pods to finish: [lalitadithya/ubuntu-gpu-deployment-64c54f5b4c-2c4c5 alloy/alloy-gateway-0 alloy/dgxc-alloy-metrics-55d6fbdbd-vrp2h]
Prerequisites
Bug Description
If a node has multiple pods running, the node drainer add multiple events with different combinations of the same pod as seen below:
Ideally the node drainer should only add a single event with the list ordered
Component
Fault Management
Steps to Reproduce
Environment
Logs/Output