Skip to content

Commit 902051e

Browse files
committed
fix(redis, valkey, harbor): spread sentinels and data pods over nodes
The RedisFailover charts left placement to the operator's default preferred pod anti-affinity. A preference only decides where a pod starts, so a drain or a rollout of the sentinel Deployment can leave two of the three sentinels on one node, and nothing moves them back. Losing that node leaves one sentinel, below the quorum of two, so no failover can happen while it is down. On a node-local StorageClass a data pod that starts next to another one stays there for good. Both now get a hostname spread with whenUnsatisfiable DoNotSchedule and maxSkew 1. An instance with more pods than nodes still starts, because the skew is counted between the nodes that exist. nodeTaintsPolicy is Ignore, so a cordoned node still counts and a pod waits for it instead of starting next to another one. The sentinel constraint also matches on pod-template-hash, because a rollout adds each new sentinel while the old ones still run, and counting those would let two new sentinels share a node once the old ones are gone. A cluster-wide globalAppTopologySpreadConstraints replaces the default, as in the other charts that read it. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent d1d4e92 commit 902051e

7 files changed

Lines changed: 375 additions & 0 deletions

File tree

‎packages/apps/redis/templates/redisfailover.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
{{- /* One image for the data pods and the Sentinels, resolved once so they cannot drift apart. */}}
22
{{- $redisImage := printf "redis:%s" (include "redis.versionMap" $) }}
33
{{- $tlsEnabled := (include "redis.tls.enabled" .) | eq "true" -}}
4+
{{- $spreadPolicy := dig "scheduling" "globalAppTopologySpreadConstraints" "" (.Values._cluster | default dict) -}}
45
{{- /*
56
A Sentinel holds only its configuration and failover state, so the Redis
67
sizing over-provisions it. An empty sentinel block still falls back to the
@@ -66,11 +67,47 @@ spec:
6667
# on every start, so a chart rollback to an older engine is safe here.
6768
image: {{ $redisImage | quote }}
6869
replicas: 3
70+
{{- if $spreadPolicy }}
71+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
72+
labelSelector:
73+
matchLabels:
74+
app.kubernetes.io/name: {{ .Release.Name }}
75+
app.kubernetes.io/component: sentinel
76+
{{- else }}
77+
topologySpreadConstraints:
78+
- maxSkew: 1
79+
topologyKey: kubernetes.io/hostname
80+
whenUnsatisfiable: DoNotSchedule
81+
nodeTaintsPolicy: Ignore
82+
matchLabelKeys:
83+
- pod-template-hash
84+
labelSelector:
85+
matchLabels:
86+
app.kubernetes.io/name: {{ .Release.Name }}
87+
app.kubernetes.io/component: sentinel
88+
{{- end }}
6989
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list $sentinelPreset $sentinelResources $) | nindent 6 }}
7090
redis:
7191
image: {{ $redisImage | quote }}
7292
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | nindent 6 }}
7393
replicas: {{ .Values.replicas }}
94+
{{- if $spreadPolicy }}
95+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
96+
labelSelector:
97+
matchLabels:
98+
app.kubernetes.io/name: {{ .Release.Name }}
99+
app.kubernetes.io/component: redis
100+
{{- else }}
101+
topologySpreadConstraints:
102+
- maxSkew: 1
103+
topologyKey: kubernetes.io/hostname
104+
whenUnsatisfiable: DoNotSchedule
105+
nodeTaintsPolicy: Ignore
106+
labelSelector:
107+
matchLabels:
108+
app.kubernetes.io/name: {{ .Release.Name }}
109+
app.kubernetes.io/component: redis
110+
{{- end }}
74111
{{- with .Values.size }}
75112
storage:
76113
persistentVolumeClaim:
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
suite: Sentinels and Redis pods spread over nodes
2+
3+
release:
4+
name: my-redis
5+
namespace: tenant-test
6+
7+
set:
8+
_cluster:
9+
cluster-domain: cluster.local
10+
_namespace:
11+
host: example.com
12+
13+
templates:
14+
- templates/redisfailover.yaml
15+
16+
tests:
17+
- it: requires one sentinel and one Redis pod per node by default
18+
documentSelector:
19+
path: kind
20+
value: RedisFailover
21+
asserts:
22+
- equal:
23+
path: spec.sentinel.topologySpreadConstraints
24+
value:
25+
- maxSkew: 1
26+
topologyKey: kubernetes.io/hostname
27+
whenUnsatisfiable: DoNotSchedule
28+
nodeTaintsPolicy: Ignore
29+
matchLabelKeys:
30+
- pod-template-hash
31+
labelSelector:
32+
matchLabels:
33+
app.kubernetes.io/name: my-redis
34+
app.kubernetes.io/component: sentinel
35+
- equal:
36+
path: spec.redis.topologySpreadConstraints
37+
value:
38+
- maxSkew: 1
39+
topologyKey: kubernetes.io/hostname
40+
whenUnsatisfiable: DoNotSchedule
41+
nodeTaintsPolicy: Ignore
42+
labelSelector:
43+
matchLabels:
44+
app.kubernetes.io/name: my-redis
45+
app.kubernetes.io/component: redis
46+
- notExists:
47+
path: spec.sentinel.affinity
48+
- notExists:
49+
path: spec.redis.affinity
50+
51+
- it: keeps the default when the cluster-wide policy is empty
52+
set:
53+
_cluster:
54+
scheduling:
55+
globalAppTopologySpreadConstraints: ""
56+
documentSelector:
57+
path: kind
58+
value: RedisFailover
59+
asserts:
60+
- equal:
61+
path: spec.sentinel.topologySpreadConstraints[0].whenUnsatisfiable
62+
value: DoNotSchedule
63+
- equal:
64+
path: spec.redis.topologySpreadConstraints[0].whenUnsatisfiable
65+
value: DoNotSchedule
66+
67+
- it: renders a cluster-wide spread policy as written
68+
set:
69+
_cluster:
70+
scheduling:
71+
globalAppTopologySpreadConstraints: |
72+
topologySpreadConstraints:
73+
- maxSkew: 1
74+
topologyKey: topology.kubernetes.io/zone
75+
whenUnsatisfiable: ScheduleAnyway
76+
documentSelector:
77+
path: kind
78+
value: RedisFailover
79+
asserts:
80+
- equal:
81+
path: spec.sentinel.topologySpreadConstraints
82+
value:
83+
- maxSkew: 1
84+
topologyKey: topology.kubernetes.io/zone
85+
whenUnsatisfiable: ScheduleAnyway
86+
labelSelector:
87+
matchLabels:
88+
app.kubernetes.io/name: my-redis
89+
app.kubernetes.io/component: sentinel
90+
- equal:
91+
path: spec.redis.topologySpreadConstraints
92+
value:
93+
- maxSkew: 1
94+
topologyKey: topology.kubernetes.io/zone
95+
whenUnsatisfiable: ScheduleAnyway
96+
labelSelector:
97+
matchLabels:
98+
app.kubernetes.io/name: my-redis
99+
app.kubernetes.io/component: redis

‎packages/apps/valkey/Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
include ../../../hack/package.mk
22

3+
.PHONY: test
4+
test:
5+
helm unittest .
6+
37
generate:
48
cozyvalues-gen -m 'valkey' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/valkey/types.go
59
../../../hack/update-crd.sh

‎packages/apps/valkey/templates/valkeyfailover.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
{{- $spreadPolicy := dig "scheduling" "globalAppTopologySpreadConstraints" "" (.Values._cluster | default dict) }}
12
{{- if .Values.authEnabled }}
23
{{- $existingPassword := lookup "v1" "Secret" .Release.Namespace (printf "%s-auth" .Release.Name) }}
34
{{- $password := randAlphaNum 32 | b64enc }}
@@ -31,11 +32,47 @@ spec:
3132
engine: Valkey
3233
sentinel:
3334
replicas: 3
35+
{{- if $spreadPolicy }}
36+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
37+
labelSelector:
38+
matchLabels:
39+
app.kubernetes.io/name: {{ .Release.Name }}
40+
app.kubernetes.io/component: sentinel
41+
{{- else }}
42+
topologySpreadConstraints:
43+
- maxSkew: 1
44+
topologyKey: kubernetes.io/hostname
45+
whenUnsatisfiable: DoNotSchedule
46+
nodeTaintsPolicy: Ignore
47+
matchLabelKeys:
48+
- pod-template-hash
49+
labelSelector:
50+
matchLabels:
51+
app.kubernetes.io/name: {{ .Release.Name }}
52+
app.kubernetes.io/component: sentinel
53+
{{- end }}
3454
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | nindent 6 }}
3555
redis:
3656
image: "valkey/valkey:{{ include "valkey.versionMap" $ }}"
3757
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | nindent 6 }}
3858
replicas: {{ .Values.replicas }}
59+
{{- if $spreadPolicy }}
60+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
61+
labelSelector:
62+
matchLabels:
63+
app.kubernetes.io/name: {{ .Release.Name }}
64+
app.kubernetes.io/component: redis
65+
{{- else }}
66+
topologySpreadConstraints:
67+
- maxSkew: 1
68+
topologyKey: kubernetes.io/hostname
69+
whenUnsatisfiable: DoNotSchedule
70+
nodeTaintsPolicy: Ignore
71+
labelSelector:
72+
matchLabels:
73+
app.kubernetes.io/name: {{ .Release.Name }}
74+
app.kubernetes.io/component: redis
75+
{{- end }}
3976
{{- with .Values.size }}
4077
storage:
4178
persistentVolumeClaim:
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
suite: Sentinels and Valkey pods spread over nodes
2+
3+
release:
4+
name: my-valkey
5+
namespace: tenant-test
6+
7+
set:
8+
_cluster: {}
9+
10+
templates:
11+
- templates/valkeyfailover.yaml
12+
13+
tests:
14+
- it: requires one sentinel and one Valkey pod per node by default
15+
documentSelector:
16+
path: kind
17+
value: RedisFailover
18+
asserts:
19+
- equal:
20+
path: spec.sentinel.topologySpreadConstraints
21+
value:
22+
- maxSkew: 1
23+
topologyKey: kubernetes.io/hostname
24+
whenUnsatisfiable: DoNotSchedule
25+
nodeTaintsPolicy: Ignore
26+
matchLabelKeys:
27+
- pod-template-hash
28+
labelSelector:
29+
matchLabels:
30+
app.kubernetes.io/name: my-valkey
31+
app.kubernetes.io/component: sentinel
32+
- equal:
33+
path: spec.redis.topologySpreadConstraints
34+
value:
35+
- maxSkew: 1
36+
topologyKey: kubernetes.io/hostname
37+
whenUnsatisfiable: DoNotSchedule
38+
nodeTaintsPolicy: Ignore
39+
labelSelector:
40+
matchLabels:
41+
app.kubernetes.io/name: my-valkey
42+
app.kubernetes.io/component: redis
43+
- notExists:
44+
path: spec.sentinel.affinity
45+
- notExists:
46+
path: spec.redis.affinity
47+
48+
- it: renders a cluster-wide spread policy as written
49+
set:
50+
_cluster:
51+
scheduling:
52+
globalAppTopologySpreadConstraints: |
53+
topologySpreadConstraints:
54+
- maxSkew: 1
55+
topologyKey: topology.kubernetes.io/zone
56+
whenUnsatisfiable: ScheduleAnyway
57+
documentSelector:
58+
path: kind
59+
value: RedisFailover
60+
asserts:
61+
- equal:
62+
path: spec.sentinel.topologySpreadConstraints
63+
value:
64+
- maxSkew: 1
65+
topologyKey: topology.kubernetes.io/zone
66+
whenUnsatisfiable: ScheduleAnyway
67+
labelSelector:
68+
matchLabels:
69+
app.kubernetes.io/name: my-valkey
70+
app.kubernetes.io/component: sentinel
71+
- equal:
72+
path: spec.redis.topologySpreadConstraints
73+
value:
74+
- maxSkew: 1
75+
topologyKey: topology.kubernetes.io/zone
76+
whenUnsatisfiable: ScheduleAnyway
77+
labelSelector:
78+
matchLabels:
79+
app.kubernetes.io/name: my-valkey
80+
app.kubernetes.io/component: redis

‎packages/system/harbor/templates/redis.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
{{- $spreadPolicy := dig "scheduling" "globalAppTopologySpreadConstraints" "" (.Values._cluster | default dict) }}
12
---
23
apiVersion: v1
34
kind: Secret
@@ -27,6 +28,25 @@ spec:
2728
# bump it as a deliberate, separate change.
2829
image: redis:6.2.6-alpine
2930
replicas: 3
31+
{{- if $spreadPolicy }}
32+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
33+
labelSelector:
34+
matchLabels:
35+
app.kubernetes.io/name: {{ .Values.harbor.fullnameOverride }}-redis
36+
app.kubernetes.io/component: sentinel
37+
{{- else }}
38+
topologySpreadConstraints:
39+
- maxSkew: 1
40+
topologyKey: kubernetes.io/hostname
41+
whenUnsatisfiable: DoNotSchedule
42+
nodeTaintsPolicy: Ignore
43+
matchLabelKeys:
44+
- pod-template-hash
45+
labelSelector:
46+
matchLabels:
47+
app.kubernetes.io/name: {{ .Values.harbor.fullnameOverride }}-redis
48+
app.kubernetes.io/component: sentinel
49+
{{- end }}
3050
resources:
3151
limits:
3252
cpu: 250m
@@ -39,6 +59,23 @@ spec:
3959
# Harbor's dump.rdb into a 7.x format a platform rollback can no longer load.
4060
image: redis:6.2.6-alpine
4161
replicas: {{ .Values.redis.replicas }}
62+
{{- if $spreadPolicy }}
63+
{{- $spreadPolicy | fromYaml | toYaml | nindent 4 }}
64+
labelSelector:
65+
matchLabels:
66+
app.kubernetes.io/name: {{ .Values.harbor.fullnameOverride }}-redis
67+
app.kubernetes.io/component: redis
68+
{{- else }}
69+
topologySpreadConstraints:
70+
- maxSkew: 1
71+
topologyKey: kubernetes.io/hostname
72+
whenUnsatisfiable: DoNotSchedule
73+
nodeTaintsPolicy: Ignore
74+
labelSelector:
75+
matchLabels:
76+
app.kubernetes.io/name: {{ .Values.harbor.fullnameOverride }}-redis
77+
app.kubernetes.io/component: redis
78+
{{- end }}
4279
resources:
4380
limits:
4481
cpu: "1"

0 commit comments

Comments
 (0)