Skip to content

Commit ac248de

Browse files
committed
fix(kubernetes): stop the Proxmox CCM outliving the node it runs on
The chart tolerated every taint with a blanket `operator: Exists`, which reads as "run anywhere" and does something else as well: it matches node.kubernetes.io/unreachable:NoExecute, so the DefaultTolerationSeconds admission plugin adds no 300s bound and the Pod is never evicted from a Node that has died. That turns a machine replacement into a deadlock, because the controller that removes a departed Node is this one. The stale Node keeps the Pod, the Pod keeps the controller off the live node, the new node keeps its uninitialized taint, and recovery needs an operator to delete the Node by hand. Tolerating only the uninitialized taint — which is all that was ever needed to land on a fresh node — lets admission add the bounded NoExecute tolerations back. Verified by deleting the Machine: the Pod moved to a live node on its own, the departed Node was cleaned up without help, and the replacement reached Running in about three minutes. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent 0f11a37 commit ac248de

1 file changed

Lines changed: 12 additions & 1 deletion

File tree

‎packages/system/proxmox-ccm/values.yaml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,19 @@ proxmox-cloud-controller-manager:
1818
# Runs on tenant worker nodes, which are the only nodes a Kamaji tenant has.
1919
affinity: {}
2020
nodeSelector: {}
21+
# Precisely the taint a fresh node carries, and nothing more. A blanket
22+
# `operator: Exists` looks equivalent and is not: it also tolerates
23+
# node.kubernetes.io/unreachable:NoExecute with no tolerationSeconds, which
24+
# stops the DefaultTolerationSeconds admission plugin from adding the 300s
25+
# one. The Pod then survives the death of the Node it runs on — and since the
26+
# controller that removes a departed Node is this one, a single-worker pool
27+
# deadlocks on machine replacement: the stale Node keeps the Pod, the new node
28+
# keeps the uninitialized taint, and nothing moves until an operator deletes
29+
# the Node by hand.
2130
tolerations:
22-
- operator: Exists
31+
- effect: NoSchedule
32+
key: node.cloudprovider.kubernetes.io/uninitialized
33+
operator: Exists
2334

2435
# cloud-node applies the topology labels and the providerID; the lifecycle
2536
# controller removes Nodes whose VM is gone. Both are wanted here; the

0 commit comments

Comments
 (0)