Skip to content

Commit 55066b9

Browse files
fix(kubernetes): bring up a replacement worker before draining the old one
A worker pool rollout (instance type, image or version change) ran with maxUnavailable 1, so CAPI drained and deleted an old Machine as soon as its replacement had been created rather than once it had joined. A one-node pool was left with no node for the few minutes the new VM took to boot and register, and every pool lost one node of capacity for each step of the rollout. With maxUnavailable 0 a node is only drained once its replacement is up. The surge stays at the pool ceiling, with a floor of 1 because CAPI refuses a rolling update where both values are 0 and maxReplicas may be 0. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 048c029 commit 55066b9

3 files changed

Lines changed: 77 additions & 5 deletions

File tree

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1024,10 +1024,15 @@ spec:
10241024
# action until the autoscaler notices unschedulable Pods and
10251025
# scales back up — a real disruption window on every routine
10261026
# platform bump.
1027+
{{- /* maxUnavailable 0 makes a rollout (instance type, image, version)
1028+
wait for each replacement Machine to be up before draining an old
1029+
one; with 1, a one-node pool lost its only node until the new one
1030+
joined. CAPI refuses maxSurge and maxUnavailable both at 0, and
1031+
maxReplicas may be 0, hence the floor of 1. */}}
10271032
strategy:
10281033
rollingUpdate:
1029-
maxSurge: {{ $group.maxReplicas }}
1030-
maxUnavailable: 1
1034+
maxSurge: {{ max 1 ($group.maxReplicas | int) }}
1035+
maxUnavailable: 0
10311036
type: RollingUpdate
10321037
selector:
10331038
matchLabels:

‎packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ matches the golden render of a GPU pool:
9797
strategy:
9898
rollingUpdate:
9999
maxSurge: 3
100-
maxUnavailable: 1
100+
maxUnavailable: 0
101101
type: RollingUpdate
102102
template:
103103
metadata:
@@ -250,7 +250,7 @@ matches the golden render of a kubelet-override pool:
250250
strategy:
251251
rollingUpdate:
252252
maxSurge: 3
253-
maxUnavailable: 1
253+
maxUnavailable: 0
254254
type: RollingUpdate
255255
template:
256256
metadata:
@@ -405,7 +405,7 @@ matches the golden render of the pool objects:
405405
strategy:
406406
rollingUpdate:
407407
maxSurge: 3
408-
maxUnavailable: 1
408+
maxUnavailable: 0
409409
type: RollingUpdate
410410
template:
411411
metadata:
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
suite: worker pool rollout keeps capacity
2+
templates:
3+
- templates/nodegroup.yaml
4+
release:
5+
name: kubernetes-nodes-myk8s-md0
6+
namespace: tenant-test
7+
set:
8+
cluster: myk8s
9+
_cluster:
10+
cluster-domain: cozy.local
11+
version: "v1.35"
12+
minReplicas: 1
13+
maxReplicas: 1
14+
instanceType: ""
15+
diskSize: 20Gi
16+
storageClass: replicated
17+
resources:
18+
cpu: "2"
19+
memory: 4Gi
20+
tests:
21+
# A rollout replaces every Machine. With maxUnavailable 1 a one-node pool
22+
# drained and deleted its only node as soon as the replacement Machine was
23+
# created, leaving the pool empty until the new node joined.
24+
- it: never drains a node before its replacement is ready
25+
documentSelector:
26+
path: kind
27+
value: MachineDeployment
28+
asserts:
29+
- equal:
30+
path: spec.strategy.type
31+
value: RollingUpdate
32+
- equal:
33+
path: spec.strategy.rollingUpdate.maxUnavailable
34+
value: 0
35+
- equal:
36+
path: spec.strategy.rollingUpdate.maxSurge
37+
value: 1
38+
39+
- it: surges up to the pool ceiling
40+
set:
41+
maxReplicas: 5
42+
documentSelector:
43+
path: kind
44+
value: MachineDeployment
45+
asserts:
46+
- equal:
47+
path: spec.strategy.rollingUpdate.maxSurge
48+
value: 5
49+
- equal:
50+
path: spec.strategy.rollingUpdate.maxUnavailable
51+
value: 0
52+
53+
# CAPI rejects a RollingUpdate with maxSurge and maxUnavailable both 0.
54+
- it: keeps a surge of one on a pool capped at zero
55+
set:
56+
minReplicas: 0
57+
maxReplicas: 0
58+
documentSelector:
59+
path: kind
60+
value: MachineDeployment
61+
asserts:
62+
- equal:
63+
path: spec.strategy.rollingUpdate.maxSurge
64+
value: 1
65+
- equal:
66+
path: spec.strategy.rollingUpdate.maxUnavailable
67+
value: 0

0 commit comments

Comments
 (0)