Skip to content

Commit 33fa5e5

Browse files
committed
fix(etcd): fix etcd-operator 1.5->1.6 upgrade (immutable selector + OOM floor)
Two defects that break the etcd-operator on an in-cluster 1.5 -> 1.6 upgrade (#2859 swapped the upstream chart for the cozystack-authored one): - #3242: the controller Deployment's spec.selector.matchLabels changed ({instance,name} -> {name,control-plane}). spec.selector is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails ("field is immutable") -- the operator and the v1alpha2 CRDs it serves never come up, which blocks the etcd v1alpha2 adoption. Fresh installs use the new selector directly, so fresh-install CI does not catch it. Add a pre-upgrade hook that deletes the Deployment ONLY when its live selector is the pre-1.6 one (lacks control-plane=controller-manager), so Helm recreates it cleanly. No-op when the selector already matches (rc.1 -> later) and never runs on a fresh install (pre-upgrade only). Keeping the selector stable in the chart is not an option: rc.1 already shipped the new selector, so aligning it back would merely move the immutable break to rc.1 -> next. - Raise the manager's cold-start memory limit floor 128Mi -> 256Mi (and the VPA minAllowed to match). The steady-state working set is ~250Mi (the VPA's own recommendation); at 128Mi a Pod that starts before the VPA admission webhook rewrites it (e.g. a Deployment recreated out of band, or the webhook briefly unavailable during upgrade) OOMKills into a crash loop. Defense in depth so the operator never depends on VPA timing merely to avoid crashing; the VPA still scales it further under load up to maxAllowed. Verified live on a 1.5 -> 1.6 adoption: the delete-Deployment step plus the VPA re-applying 256Mi+ let the operator come up and the adoption complete. Refs: #3242, #3243 Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 2932ecc commit 33fa5e5

2 files changed

Lines changed: 169 additions & 4 deletions

File tree

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
{{- /*
2+
Pre-upgrade selector migration for the controller-manager Deployment.
3+
4+
1.6 replaced the upstream etcd-operator chart with the cozystack-authored one
5+
(#2859), changing the Deployment's spec.selector.matchLabels:
6+
7+
1.5: {app.kubernetes.io/instance: etcd-operator, app.kubernetes.io/name: etcd-operator}
8+
1.6: {app.kubernetes.io/name: etcd-operator, control-plane: controller-manager}
9+
10+
spec.selector is immutable, so `helm upgrade` cannot patch the existing
11+
Deployment and the whole HelmRelease upgrade fails ("field is immutable") — the
12+
operator (and the v1alpha2 CRDs it serves) never come up, which blocks the etcd
13+
v1alpha2 adoption. Fresh installs use the new selector directly and are
14+
unaffected, so fresh-install CI does not catch it. See cozystack/cozystack#3242.
15+
16+
This pre-upgrade hook deletes the Deployment ONLY when its live selector is the
17+
pre-1.6 one (i.e. lacks control-plane=controller-manager), so Helm recreates it
18+
cleanly with the new selector. It is a no-op when the selector already matches
19+
(rc.1 -> later upgrades) and never runs on a fresh install (pre-upgrade only).
20+
Keeping the selector stable in the chart is NOT an option: rc.1 already shipped
21+
the new selector, so aligning it back would merely move the immutable break to
22+
rc.1 -> next.
23+
*/}}
24+
{{- $name := "etcd-operator-selector-fix" }}
25+
apiVersion: v1
26+
kind: ServiceAccount
27+
metadata:
28+
name: {{ $name }}
29+
namespace: {{ .Release.Namespace }}
30+
labels:
31+
app.kubernetes.io/name: etcd-operator
32+
app.kubernetes.io/component: selector-fix-hook
33+
annotations:
34+
helm.sh/hook: pre-upgrade
35+
helm.sh/hook-weight: "-5"
36+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
37+
---
38+
apiVersion: rbac.authorization.k8s.io/v1
39+
kind: Role
40+
metadata:
41+
name: {{ $name }}
42+
namespace: {{ .Release.Namespace }}
43+
labels:
44+
app.kubernetes.io/name: etcd-operator
45+
app.kubernetes.io/component: selector-fix-hook
46+
annotations:
47+
helm.sh/hook: pre-upgrade
48+
helm.sh/hook-weight: "-5"
49+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
50+
rules:
51+
- apiGroups: ["apps"]
52+
resources: ["deployments"]
53+
verbs: ["get", "list", "delete"]
54+
---
55+
apiVersion: rbac.authorization.k8s.io/v1
56+
kind: RoleBinding
57+
metadata:
58+
name: {{ $name }}
59+
namespace: {{ .Release.Namespace }}
60+
labels:
61+
app.kubernetes.io/name: etcd-operator
62+
app.kubernetes.io/component: selector-fix-hook
63+
annotations:
64+
helm.sh/hook: pre-upgrade
65+
helm.sh/hook-weight: "-5"
66+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
67+
roleRef:
68+
apiGroup: rbac.authorization.k8s.io
69+
kind: Role
70+
name: {{ $name }}
71+
subjects:
72+
- kind: ServiceAccount
73+
name: {{ $name }}
74+
namespace: {{ .Release.Namespace }}
75+
---
76+
apiVersion: batch/v1
77+
kind: Job
78+
metadata:
79+
name: {{ $name }}
80+
namespace: {{ .Release.Namespace }}
81+
labels:
82+
app.kubernetes.io/name: etcd-operator
83+
app.kubernetes.io/component: selector-fix-hook
84+
annotations:
85+
helm.sh/hook: pre-upgrade
86+
helm.sh/hook-weight: "0"
87+
# Keep a failed Job's Pod around for `kubectl logs`; reap the whole hook
88+
# bundle on success. before-hook-creation clears a prior run first.
89+
helm.sh/hook-delete-policy: hook-succeeded,before-hook-creation
90+
spec:
91+
activeDeadlineSeconds: 120
92+
backoffLimit: 1
93+
ttlSecondsAfterFinished: 600
94+
template:
95+
metadata:
96+
labels:
97+
app.kubernetes.io/name: etcd-operator
98+
app.kubernetes.io/component: selector-fix-hook
99+
spec:
100+
serviceAccountName: {{ $name }}
101+
restartPolicy: Never
102+
securityContext:
103+
runAsNonRoot: true
104+
seccompProfile:
105+
type: RuntimeDefault
106+
# writable HOME so kubectl can create its discovery cache under
107+
# readOnlyRootFilesystem=true (kubectl 1.30+ aborts otherwise).
108+
volumes:
109+
- name: home
110+
emptyDir: {}
111+
containers:
112+
- name: selector-fix
113+
image: "{{ .Values.kubectlImage.repository }}:{{ .Values.kubectlImage.tag }}"
114+
imagePullPolicy: {{ .Values.kubectlImage.pullPolicy }}
115+
securityContext:
116+
allowPrivilegeEscalation: false
117+
readOnlyRootFilesystem: true
118+
capabilities:
119+
drop: ["ALL"]
120+
env:
121+
- name: HOME
122+
value: /tmp
123+
volumeMounts:
124+
- name: home
125+
mountPath: /tmp
126+
command:
127+
- /bin/sh
128+
- -c
129+
- |
130+
set -eu
131+
ns="{{ .Release.Namespace }}"
132+
name=etcd-operator-controller-manager
133+
if ! kubectl -n "$ns" get deployment "$name" >/dev/null 2>&1; then
134+
echo "[selector-fix] Deployment $name absent; nothing to do."
135+
exit 0
136+
fi
137+
cp="$(kubectl -n "$ns" get deployment "$name" \
138+
-o jsonpath='{.spec.selector.matchLabels.control-plane}' 2>/dev/null || true)"
139+
if [ "$cp" = "controller-manager" ]; then
140+
echo "[selector-fix] Deployment $name already has the v1.6 selector; nothing to do."
141+
exit 0
142+
fi
143+
echo "[selector-fix] Deployment $name has a pre-1.6 spec.selector (immutable); deleting so the chart recreates it with the current selector."
144+
kubectl -n "$ns" delete deployment "$name" --wait=true --timeout=90s
145+
echo "[selector-fix] Deleted; the chart will recreate $name."

‎packages/system/etcd-operator/values.yaml‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,38 @@ resources:
2020
memory: 64Mi
2121
limits:
2222
cpu: 500m
23-
memory: 128Mi
23+
# Cold-start floor. This is the limit a freshly-created Pod runs under until
24+
# the VPA admission webhook has had a chance to rewrite it (see vpa below).
25+
# The manager's steady-state working set is ~250Mi (the VPA's own
26+
# recommendation), so a 128Mi floor OOMKills the operator on any start where
27+
# the webhook does not mutate the Pod in time — e.g. a Deployment recreated
28+
# out of band, or the webhook briefly unavailable during an upgrade. Keep
29+
# this at or above the observed working set so the operator never depends on
30+
# VPA timing merely to avoid a crash loop; the VPA still scales it further
31+
# under real load up to maxAllowed.
32+
memory: 256Mi
2433

2534
vpa:
2635
enabled: true
2736
updateMode: Auto
2837
minAllowed:
2938
cpu: 100m
30-
memory: 128Mi
39+
# Matches the cold-start limit floor above so the VPA never recommends BELOW
40+
# the manager's ~250Mi working set.
41+
memory: 256Mi
3142
# maxAllowed is the VPA's scaling CEILING, deliberately well above the static
32-
# resources.limits above (128Mi): with updateMode: Auto the VPA rewrites the
43+
# resources.limits above (256Mi): with updateMode: Auto the VPA rewrites the
3344
# Pod's request/limit up to this bound under real load. Do NOT "align" it down
34-
# to the 128Mi limit — that would cap the autoscaler's headroom.
45+
# to the 256Mi limit — that would cap the autoscaler's headroom.
3546
maxAllowed:
3647
cpu: 1000m
3748
memory: 1Gi
49+
50+
# Image for the pre-upgrade selector-migration hook Job (see
51+
# templates/pre-upgrade-selector-fix.yaml). Only needs a kubectl binary. Pinned
52+
# by digest for reproducibility; clastix/kubectl is already used elsewhere in
53+
# cozystack (Kamaji).
54+
kubectlImage:
55+
repository: docker.io/clastix/kubectl
56+
tag: v1.32
57+
pullPolicy: IfNotPresent

0 commit comments

Comments
 (0)