Skip to content

Commit 7c1bd33

Browse files
fix(redis): delete the Sentinel PDB left behind on scale-down
Setting sentinel.disablePodDisruptionBudget only stops the operator from ensuring the PDB; it never deletes the one it created while three Sentinels ran. An existing instance scaled to one Redis pod therefore kept minAvailable 1 on its single Sentinel, which refuses every eviction. A post-upgrade hook deletes rfs-<release> in that case. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 48779cc commit 7c1bd33

5 files changed

Lines changed: 206 additions & 3 deletions

File tree

‎packages/apps/redis/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-pr
7979

8080
### sentinel
8181

82-
`sentinel.resources` and `sentinel.resourcesPreset` size the three Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed.
82+
`sentinel.resources` and `sentinel.resourcesPreset` size the Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed.
8383

8484
```yaml
8585
resourcesPreset: u1.small

‎packages/apps/redis/templates/_resources.tpl‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,7 @@ These presets are for basic testing and not meant to be used in production
4747
{{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}}
4848
{{- end -}}
4949
{{- end -}}
50+
51+
{{- define "redis.sentinelReplicas" -}}
52+
{{- ternary 1 3 (le (int .Values.replicas) 1) -}}
53+
{{- end -}}
Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
{{- /*
2+
An instance scaled down to one Redis pod keeps the Sentinel PDB the operator
3+
created while it ran three Sentinels: setting disablePodDisruptionBudget only
4+
stops the operator from ensuring it, never deletes it, and minAvailable 1 on
5+
a single Sentinel refuses every eviction. A fresh install with one replica
6+
never gets that PDB, so this runs on upgrade only.
7+
*/}}
8+
{{- if and .Release.IsUpgrade (eq (include "redis.sentinelReplicas" . | int) 1) }}
9+
{{- $name := printf "%s-sentinel-pdb" .Release.Name }}
10+
---
11+
apiVersion: v1
12+
kind: ServiceAccount
13+
metadata:
14+
name: {{ $name }}
15+
labels:
16+
app.kubernetes.io/instance: {{ .Release.Name }}
17+
annotations:
18+
helm.sh/hook: post-upgrade
19+
helm.sh/hook-weight: "0"
20+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
21+
---
22+
apiVersion: rbac.authorization.k8s.io/v1
23+
kind: Role
24+
metadata:
25+
name: {{ $name }}
26+
labels:
27+
app.kubernetes.io/instance: {{ .Release.Name }}
28+
annotations:
29+
helm.sh/hook: post-upgrade
30+
helm.sh/hook-weight: "0"
31+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
32+
rules:
33+
- apiGroups: ["policy"]
34+
resources: ["poddisruptionbudgets"]
35+
resourceNames: ["rfs-{{ .Release.Name }}"]
36+
verbs: ["get", "delete"]
37+
---
38+
apiVersion: rbac.authorization.k8s.io/v1
39+
kind: RoleBinding
40+
metadata:
41+
name: {{ $name }}
42+
labels:
43+
app.kubernetes.io/instance: {{ .Release.Name }}
44+
annotations:
45+
helm.sh/hook: post-upgrade
46+
helm.sh/hook-weight: "0"
47+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
48+
roleRef:
49+
apiGroup: rbac.authorization.k8s.io
50+
kind: Role
51+
name: {{ $name }}
52+
subjects:
53+
- kind: ServiceAccount
54+
name: {{ $name }}
55+
namespace: {{ .Release.Namespace }}
56+
---
57+
apiVersion: batch/v1
58+
kind: Job
59+
metadata:
60+
name: {{ $name }}
61+
labels:
62+
app.kubernetes.io/instance: {{ .Release.Name }}
63+
annotations:
64+
helm.sh/hook: post-upgrade
65+
helm.sh/hook-weight: "5"
66+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
67+
spec:
68+
ttlSecondsAfterFinished: 3600
69+
# Bounds the kubectl call too: a --request-timeout would make kubectl skip
70+
# its in-cluster credentials (hack/check-hook-request-timeout.bats).
71+
activeDeadlineSeconds: 120
72+
backoffLimit: 2
73+
template:
74+
metadata:
75+
labels:
76+
app.kubernetes.io/instance: {{ .Release.Name }}
77+
policy.cozystack.io/allow-to-apiserver: "true"
78+
spec:
79+
serviceAccountName: {{ $name }}
80+
restartPolicy: Never
81+
terminationGracePeriodSeconds: 10
82+
securityContext:
83+
runAsNonRoot: true
84+
runAsUser: 65534
85+
runAsGroup: 65534
86+
seccompProfile:
87+
type: RuntimeDefault
88+
containers:
89+
- name: delete-pdb
90+
image: '{{ include "cozy-lib.image" (list "clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c" $) }}'
91+
securityContext:
92+
allowPrivilegeEscalation: false
93+
readOnlyRootFilesystem: true
94+
capabilities:
95+
drop: ["ALL"]
96+
resources:
97+
requests:
98+
cpu: 10m
99+
memory: 64Mi
100+
limits:
101+
cpu: 200m
102+
memory: 128Mi
103+
env:
104+
- name: HOME
105+
value: /tmp
106+
command:
107+
- kubectl
108+
- --namespace={{ .Release.Namespace }}
109+
- delete
110+
- poddisruptionbudget
111+
- rfs-{{ .Release.Name }}
112+
- --ignore-not-found=true
113+
volumeMounts:
114+
- name: tmp
115+
mountPath: /tmp
116+
volumes:
117+
- name: tmp
118+
emptyDir: {}
119+
{{- end }}

‎packages/apps/redis/templates/redisfailover.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,11 @@
1414
watch it. One Sentinel still lets the operator find the master (its quorum is
1515
replicas/2+1, so 1). The operator gives the Sentinel Deployment a PDB with
1616
minAvailable 1 whenever Redis has two replicas or fewer, which on a single
17-
Sentinel would block every node drain, so that PDB is dropped with it.
17+
Sentinel would block every node drain, so that PDB is dropped with it; the
18+
operator does not delete a PDB it stops managing, hence
19+
hooks/sentinel-pdb.yaml.
1820
*/}}
19-
{{- $sentinelReplicas := ternary 1 3 (le (int .Values.replicas) 1) -}}
21+
{{- $sentinelReplicas := include "redis.sentinelReplicas" . | int -}}
2022
{{- $sentinelPreset := $sentinel.resourcesPreset | default .Values.resourcesPreset -}}
2123
{{- $sentinelResources := .Values.resources -}}
2224
{{- if or $sentinel.resources $sentinel.resourcesPreset -}}
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
suite: stale Sentinel PDB removed on scale-down to one Redis pod
2+
3+
# The operator only stops ensuring the Sentinel PDB once
4+
# disablePodDisruptionBudget is set; it never deletes the one it created while
5+
# three Sentinels ran. Left in place, its minAvailable 1 refuses every eviction
6+
# of the single Sentinel and blocks node drains.
7+
8+
templates:
9+
- templates/hooks/sentinel-pdb.yaml
10+
11+
release:
12+
name: my-redis
13+
namespace: tenant-test
14+
15+
set:
16+
_cluster:
17+
cluster-domain: cluster.local
18+
_namespace:
19+
host: example.com
20+
21+
tests:
22+
- it: deletes the operator's Sentinel PDB on an upgrade to one replica
23+
release:
24+
upgrade: true
25+
set:
26+
replicas: 1
27+
documentSelector:
28+
path: kind
29+
value: Job
30+
asserts:
31+
- equal:
32+
path: metadata.annotations["helm.sh/hook"]
33+
value: post-upgrade
34+
- equal:
35+
path: spec.template.spec.containers[0].command
36+
value:
37+
- kubectl
38+
- --namespace=tenant-test
39+
- delete
40+
- poddisruptionbudget
41+
- rfs-my-redis
42+
- --ignore-not-found=true
43+
- notMatchRegex:
44+
path: spec.template.spec.containers[0].command[4]
45+
pattern: request-timeout
46+
47+
- it: lets the hook delete only that PDB
48+
release:
49+
upgrade: true
50+
set:
51+
replicas: 1
52+
documentSelector:
53+
path: kind
54+
value: Role
55+
asserts:
56+
- equal:
57+
path: rules
58+
value:
59+
- apiGroups: ["policy"]
60+
resources: ["poddisruptionbudgets"]
61+
resourceNames: ["rfs-my-redis"]
62+
verbs: ["get", "delete"]
63+
64+
- it: renders nothing on install, where no such PDB exists
65+
set:
66+
replicas: 1
67+
asserts:
68+
- hasDocuments:
69+
count: 0
70+
71+
- it: renders nothing while three Sentinels run
72+
release:
73+
upgrade: true
74+
set:
75+
replicas: 2
76+
asserts:
77+
- hasDocuments:
78+
count: 0

0 commit comments

Comments
 (0)