Skip to content

Commit f1f30b3

Browse files
committed
fix(etcd): require an even spread of members over nodes by default
The default spread of the etcd members was ScheduleAnyway, which only adds to the score of a node, and other scores can outweigh it. A new etcd on a three-node cluster got its first two members on one node. On a node-local StorageClass a member cannot leave its node, so losing that node takes two of the three members and the quorum. DoNotSchedule with maxSkew 1 still lets an etcd with more members than nodes start, because the skew is counted between the nodes that exist. One node takes all three members and two nodes take two and one. nodeTaintsPolicy is Ignore, so a cordoned node still counts and a member waits for it instead of landing next to another member. The operator copies the constraints into an EtcdMember when it creates it and builds the Pod from that copy, so members that exist keep the old setting and only members created later get the new one. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent aed7fa8 commit f1f30b3

2 files changed

Lines changed: 37 additions & 12 deletions

File tree

‎packages/extra/etcd/templates/etcd-cluster.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,8 @@ spec:
6969
topologySpreadConstraints:
7070
- maxSkew: 1
7171
topologyKey: "kubernetes.io/hostname"
72-
whenUnsatisfiable: ScheduleAnyway
72+
whenUnsatisfiable: DoNotSchedule
73+
nodeTaintsPolicy: Ignore
7374
labelSelector:
7475
matchLabels:
7576
app.kubernetes.io/instance: etcd

‎packages/extra/etcd/tests/topology-spread_test.yaml‎

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,25 +8,49 @@ templates:
88
- templates/etcd-cluster.yaml
99

1010
tests:
11-
- it: uses a soft per-host spread by default
11+
- it: uses a strict per-host spread by default
1212
set:
1313
_cluster: {}
1414
documentSelector:
1515
path: kind
1616
value: EtcdCluster
1717
asserts:
1818
- equal:
19-
path: spec.topologySpreadConstraints[0].maxSkew
20-
value: 1
21-
- equal:
22-
path: spec.topologySpreadConstraints[0].topologyKey
23-
value: kubernetes.io/hostname
24-
- equal:
25-
path: spec.topologySpreadConstraints[0].whenUnsatisfiable
26-
value: ScheduleAnyway
19+
path: spec.topologySpreadConstraints[0]
20+
value:
21+
maxSkew: 1
22+
topologyKey: kubernetes.io/hostname
23+
whenUnsatisfiable: DoNotSchedule
24+
nodeTaintsPolicy: Ignore
25+
labelSelector:
26+
matchLabels:
27+
app.kubernetes.io/instance: etcd
28+
- lengthEqual:
29+
path: spec.topologySpreadConstraints
30+
count: 1
31+
32+
- it: renders a soft cluster-wide spread policy as written
33+
set:
34+
_cluster:
35+
scheduling:
36+
globalAppTopologySpreadConstraints: |
37+
topologySpreadConstraints:
38+
- maxSkew: 1
39+
topologyKey: kubernetes.io/hostname
40+
whenUnsatisfiable: ScheduleAnyway
41+
documentSelector:
42+
path: kind
43+
value: EtcdCluster
44+
asserts:
2745
- equal:
28-
path: spec.topologySpreadConstraints[0].labelSelector.matchLabels["app.kubernetes.io/instance"]
29-
value: etcd
46+
path: spec.topologySpreadConstraints[0]
47+
value:
48+
maxSkew: 1
49+
topologyKey: kubernetes.io/hostname
50+
whenUnsatisfiable: ScheduleAnyway
51+
labelSelector:
52+
matchLabels:
53+
app.kubernetes.io/instance: etcd
3054
- lengthEqual:
3155
path: spec.topologySpreadConstraints
3256
count: 1

0 commit comments

Comments
 (0)