Repository navigation
[Backport release-1.6] fix(kubernetes): bring up a replacement worker before draining the old one - #5154
Conversation
…d one A worker pool rollout (instance type, image or version change) ran with maxUnavailable 1, so CAPI drained and deleted an old Machine as soon as its replacement had been created rather than once it had joined. A one-node pool was left with no node for the few minutes the new VM took to boot and register, and every pool lost one node of capacity for each step of the rollout. With maxUnavailable 0 a node is only drained once its replacement is up. The surge stays at the pool ceiling, with a floor of 1 because CAPI refuses a rolling update where both values are 0 and maxReplicas may be 0. On this branch the in-chart nodeGroups of the kubernetes app render the same MachineDeployment strategy, so it gets the same change and a test of its own; the kubernetes-nodes render snapshot does not exist here. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit 55066b9)
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
github-actions
Bot
added
area/kubernetes
A replacement VM in a pool with passed-through GPUs may need the very GPU the old node holds. With maxUnavailable 0 it would stay Pending and the rollout would never finish, so those pools keep removing a node first. Assisted-by: LLM (cherry picked from commit 392bbb3) Signed-off-by: Mattia Eleuteri <[email protected]>
|
CI note: the remaining red checks come from this PR being opened from a fork, not from its content. |
What this PR does
Backport of #5153 to
release-1.6. The commit is cherry-picked and keeps its cherry-pick reference. Worker pool MachineDeployments switch frommaxUnavailable: 1tomaxUnavailable: 0, withmaxSurgeat the pool ceiling and a floor of 1, so a rollout brings each replacement node up before draining the old one instead of leaving a one-node pool empty for several minutes.Two differences from main. On this branch the Kubernetes app still renders its own
nodeGroupsMachineDeployments inpackages/apps/kubernetes/templates/cluster.yamlwith the same strategy, so that template gets the same change and a test suite of its own (tests/nodegroup_rollout_strategy_test.yaml); thekubernetes-nodesrender snapshot, which was the cherry-pick conflict, does not exist here and is dropped.On this branch
helm unittest packages/apps/kubernetes packages/apps/kubernetes-nodespasses 211 of 211. With the release-1.6 templates the new non-GPU cases of each suite fail.Exception: a pool with
gpuskeepsmaxUnavailable: 1. Its replacement VM may need the very passed-through GPU the old node holds, and withmaxUnavailable: 0it would stay Pending and the rollout would never finish. A Proxmox pool that passes GPUs through its template carries nogpusvalue, so it is not recognised and getsmaxUnavailable: 0.Trade-off, as on main: a rollout needs room for one extra VM, and a tenant at its quota stalls the rollout instead of temporarily shrinking the pool.
Screenshots
Downstream repositories
Release note