Skip to content

Commit 5d0bb38

Browse files
fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first
Disabling the Monitoring Agents addon of a tenant cluster removes this release, the VictoriaMetrics operator and the VPA CRDs together, and Flux deletes them in parallel without honouring dependsOn. When the operator goes first, nothing clears apps.victoriametrics.com/finalizer on the VMAgent and the other VictoriaMetrics objects this chart created, so they stay Terminating and the release sits in UninstallFailed for good. A pre-delete hook now deletes the objects the release owns, gives a live operator a bounded minute to finalize them, and clears the finalizers itself when it has not. Objects are selected by Helm's ownership annotations so the rules under alerts/ are covered too, and the hook always exits 0 so it can never be what blocks the uninstall. The VPA is also rendered only when its CRD is served, so a release manifest never records an object Helm could not map at uninstall. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 048c029 commit 5d0bb38

4 files changed

Lines changed: 323 additions & 0 deletions

File tree

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
{{- /*
2+
The VictoriaMetrics operator puts its finalizer on the objects this chart
3+
creates and is the only thing that ever removes it. The operator ships as a
4+
separate release that is gated on the same addon flag, so disabling the addon
5+
removes both at once and Flux does not order deletions: when the operator goes
6+
first, every one of these objects stays Terminating and the uninstall never
7+
finishes.
8+
9+
So delete them before Helm does, give a live operator a bounded chance to
10+
finalize them itself, and only then clear the finalizers by hand. The VMAgent's
11+
Deployment, Service and Secrets carry owner references and are garbage
12+
collected once it is gone.
13+
14+
Cluster-scoped RBAC because the objects span two namespaces, and a Role in the
15+
operator's namespace could not be created if that namespace were already
16+
gone. The hook always exits 0: a cleanup that cannot complete must not be what
17+
keeps the release from uninstalling.
18+
*/ -}}
19+
{{- $name := "vm-objects-cleanup" }}
20+
{{- $img := printf "%s:%s" .Values.kubectlImage.repository .Values.kubectlImage.tag }}
21+
{{- with .Values.kubectlImage.digest }}{{- $img = printf "%s@%s" $img . }}{{- end }}
22+
{{- $kinds := list "vmagents" "vmrules" "vmservicescrapes" "vmpodscrapes" "vmnodescrapes" }}
23+
apiVersion: v1
24+
kind: ServiceAccount
25+
metadata:
26+
name: {{ $name }}
27+
namespace: {{ .Release.Namespace }}
28+
annotations:
29+
helm.sh/hook: pre-delete
30+
helm.sh/hook-weight: "-10"
31+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
32+
---
33+
apiVersion: rbac.authorization.k8s.io/v1
34+
kind: ClusterRole
35+
metadata:
36+
name: {{ .Release.Namespace }}-{{ $name }}
37+
annotations:
38+
helm.sh/hook: pre-delete
39+
helm.sh/hook-weight: "-10"
40+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
41+
rules:
42+
- apiGroups: ["operator.victoriametrics.com"]
43+
resources:
44+
{{- toYaml $kinds | nindent 6 }}
45+
verbs: ["get", "list", "delete", "patch"]
46+
---
47+
apiVersion: rbac.authorization.k8s.io/v1
48+
kind: ClusterRoleBinding
49+
metadata:
50+
name: {{ .Release.Namespace }}-{{ $name }}
51+
annotations:
52+
helm.sh/hook: pre-delete
53+
helm.sh/hook-weight: "-10"
54+
helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation
55+
roleRef:
56+
apiGroup: rbac.authorization.k8s.io
57+
kind: ClusterRole
58+
name: {{ .Release.Namespace }}-{{ $name }}
59+
subjects:
60+
- kind: ServiceAccount
61+
name: {{ $name }}
62+
namespace: {{ .Release.Namespace }}
63+
---
64+
apiVersion: batch/v1
65+
kind: Job
66+
metadata:
67+
name: {{ $name }}
68+
namespace: {{ .Release.Namespace }}
69+
annotations:
70+
helm.sh/hook: pre-delete
71+
helm.sh/hook-weight: "0"
72+
helm.sh/hook-delete-policy: hook-succeeded,before-hook-creation
73+
spec:
74+
# Well inside the HelmRelease's 10m uninstall budget, and above the 60s wait
75+
# plus one call per object on either side of it.
76+
activeDeadlineSeconds: 180
77+
backoffLimit: 0
78+
ttlSecondsAfterFinished: 600
79+
template:
80+
spec:
81+
serviceAccountName: {{ $name }}
82+
restartPolicy: Never
83+
securityContext:
84+
runAsNonRoot: true
85+
# The image user is the non-numeric `nonroot`, which the kubelet cannot
86+
# check against runAsNonRoot.
87+
runAsUser: 65532
88+
seccompProfile:
89+
type: RuntimeDefault
90+
volumes:
91+
- name: home
92+
emptyDir: {}
93+
containers:
94+
- name: cleanup
95+
image: {{ $img | quote }}
96+
imagePullPolicy: {{ .Values.kubectlImage.pullPolicy }}
97+
resources:
98+
requests:
99+
cpu: 10m
100+
memory: 32Mi
101+
limits:
102+
cpu: 100m
103+
memory: 64Mi
104+
securityContext:
105+
allowPrivilegeEscalation: false
106+
readOnlyRootFilesystem: true
107+
capabilities:
108+
drop: ["ALL"]
109+
# kubectl writes its discovery cache under $HOME.
110+
env:
111+
- name: HOME
112+
value: /tmp
113+
volumeMounts:
114+
- name: home
115+
mountPath: /tmp
116+
command:
117+
- /bin/sh
118+
- -c
119+
- |
120+
rel={{ .Release.Name | quote }}
121+
relns={{ .Release.Namespace | quote }}
122+
kinds={{ join " " $kinds | quote }}
123+
124+
# Selected by Helm's ownership annotations rather than by name, so
125+
# the alert rules added under alerts/ are covered without a list
126+
# to keep in step. A kind whose CRD is gone just prints an error.
127+
owned() {
128+
for k in $kinds; do
129+
kubectl get "$k.operator.victoriametrics.com" -A --request-timeout=15s \
130+
-o jsonpath='{range .items[*]}{.metadata.annotations.meta\.helm\.sh/release-name}{"|"}{.metadata.annotations.meta\.helm\.sh/release-namespace}{"|"}{.metadata.namespace}{"|"}{.metadata.name}{"\n"}{end}' |
131+
while IFS='|' read -r r rns ns n; do
132+
if [ "$r" = "$rel" ] && [ "$rns" = "$relns" ]; then
133+
echo "$k.operator.victoriametrics.com $ns $n"
134+
fi
135+
done
136+
done
137+
}
138+
139+
objs=$(owned)
140+
if [ -z "$objs" ]; then
141+
echo "no VictoriaMetrics objects left for $relns/$rel"
142+
exit 0
143+
fi
144+
145+
echo "$objs" | while read -r k ns n; do
146+
kubectl -n "$ns" delete "$k" "$n" --wait=false --request-timeout=15s
147+
done
148+
149+
deadline=$(( $(date +%s) + 60 ))
150+
while [ "$(date +%s)" -lt "$deadline" ]; do
151+
sleep 2
152+
objs=$(owned)
153+
if [ -z "$objs" ]; then
154+
echo "the operator finalized every object"
155+
exit 0
156+
fi
157+
done
158+
159+
echo "finalizers still held after 60s; clearing them by hand"
160+
echo "$objs" | while read -r k ns n; do
161+
kubectl -n "$ns" patch "$k" "$n" --type=merge \
162+
-p '{"metadata":{"finalizers":null}}' --request-timeout=15s
163+
done
164+
exit 0

‎packages/system/monitoring-agents/templates/vpa.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,11 @@
66
Initial sets requests at creation without evicting, removing the trigger.
77
Do not switch back to Auto without fixing the underlying Cilium IP-reuse race.
88
*/ -}}
9+
{{- /*
10+
Rendered only when the CRD is served: a release manifest that records a VPA
11+
the cluster cannot map is one Helm can no longer uninstall.
12+
*/ -}}
13+
{{- if .Capabilities.APIVersions.Has "autoscaling.k8s.io/v1/VerticalPodAutoscaler" }}
914
apiVersion: autoscaling.k8s.io/v1
1015
kind: VerticalPodAutoscaler
1116
metadata:
@@ -34,3 +39,4 @@ spec:
3439
maxAllowed:
3540
cpu: 4000m
3641
memory: 6G
42+
{{- end }}
Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
suite: the release uninstalls whatever order its dependencies go in
2+
3+
# Disabling the addon removes this release, the VictoriaMetrics operator and the
4+
# VPA CRDs together, and Flux deletes them in parallel. When the operator goes
5+
# first, nothing clears its finalizer on the objects this chart created and the
6+
# uninstall hangs; when the VPA CRD goes first, a release manifest recording a
7+
# VPA cannot be built and the uninstall fails.
8+
release:
9+
name: cozy-monitoring-agents
10+
namespace: cozy-monitoring
11+
templates:
12+
- templates/vm-cleanup-hook.yaml
13+
- templates/vpa.yaml
14+
tests:
15+
- it: runs before Helm deletes anything, bounded and with one attempt
16+
template: templates/vm-cleanup-hook.yaml
17+
documentSelector:
18+
path: kind
19+
value: Job
20+
asserts:
21+
- equal:
22+
path: metadata.annotations["helm.sh/hook"]
23+
value: pre-delete
24+
- equal:
25+
path: metadata.namespace
26+
value: cozy-monitoring
27+
- equal:
28+
path: spec.activeDeadlineSeconds
29+
value: 180
30+
- equal:
31+
path: spec.backoffLimit
32+
value: 0
33+
- matchRegex:
34+
path: spec.template.spec.containers[0].image
35+
pattern: ^docker\.io/clastix/kubectl:v[0-9.]+@sha256:[0-9a-f]{64}$
36+
37+
- it: its RBAC exists only for the hook and is removed with it
38+
template: templates/vm-cleanup-hook.yaml
39+
documentSelector:
40+
path: kind
41+
value: ClusterRole
42+
asserts:
43+
- equal:
44+
path: metadata.annotations["helm.sh/hook"]
45+
value: pre-delete
46+
- equal:
47+
path: metadata.annotations["helm.sh/hook-weight"]
48+
value: "-10"
49+
- matchRegex:
50+
path: metadata.annotations["helm.sh/hook-delete-policy"]
51+
pattern: hook-succeeded
52+
53+
- it: may delete and clear finalizers on every VictoriaMetrics kind the chart renders
54+
template: templates/vm-cleanup-hook.yaml
55+
documentSelector:
56+
path: kind
57+
value: ClusterRole
58+
asserts:
59+
- equal:
60+
path: rules[0].apiGroups
61+
value: ["operator.victoriametrics.com"]
62+
- contains:
63+
path: rules[0].resources
64+
content: vmagents
65+
- contains:
66+
path: rules[0].resources
67+
content: vmrules
68+
- contains:
69+
path: rules[0].resources
70+
content: vmservicescrapes
71+
- contains:
72+
path: rules[0].resources
73+
content: vmpodscrapes
74+
- contains:
75+
path: rules[0].resources
76+
content: vmnodescrapes
77+
- equal:
78+
path: rules[0].verbs
79+
value: ["get", "list", "delete", "patch"]
80+
- lengthEqual:
81+
path: rules
82+
count: 1
83+
84+
- it: grants that role to the hook's own service account
85+
template: templates/vm-cleanup-hook.yaml
86+
documentSelector:
87+
path: kind
88+
value: ClusterRoleBinding
89+
asserts:
90+
- equal:
91+
path: roleRef.name
92+
value: cozy-monitoring-vm-objects-cleanup
93+
- equal:
94+
path: subjects[0]
95+
value:
96+
kind: ServiceAccount
97+
name: vm-objects-cleanup
98+
namespace: cozy-monitoring
99+
100+
- it: touches only objects this release owns, and clears their finalizers
101+
template: templates/vm-cleanup-hook.yaml
102+
documentSelector:
103+
path: kind
104+
value: Job
105+
asserts:
106+
- matchRegex:
107+
path: spec.template.spec.containers[0].command[2]
108+
pattern: rel="cozy-monitoring-agents"
109+
- matchRegex:
110+
path: spec.template.spec.containers[0].command[2]
111+
pattern: relns="cozy-monitoring"
112+
- matchRegex:
113+
path: spec.template.spec.containers[0].command[2]
114+
pattern: meta\\\.helm\\\.sh/release-name
115+
- matchRegex:
116+
path: spec.template.spec.containers[0].command[2]
117+
pattern: 'delete "\$k" "\$n" --wait=false'
118+
- matchRegex:
119+
path: spec.template.spec.containers[0].command[2]
120+
pattern: '"finalizers":null'
121+
# A pre-delete hook that fails aborts the uninstall it exists to unblock.
122+
- matchRegex:
123+
path: spec.template.spec.containers[0].command[2]
124+
pattern: exit 0\s*$
125+
126+
- it: the VPA is rendered when its CRD is served
127+
template: templates/vpa.yaml
128+
capabilities:
129+
apiVersions:
130+
- autoscaling.k8s.io/v1/VerticalPodAutoscaler
131+
asserts:
132+
- hasDocuments:
133+
count: 1
134+
- isKind:
135+
of: VerticalPodAutoscaler
136+
137+
- it: the VPA stays out of the release manifest when its CRD is not served
138+
template: templates/vpa.yaml
139+
asserts:
140+
- hasDocuments:
141+
count: 0

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -552,3 +552,15 @@ prometheus-node-exporter:
552552
memory: 32Mi
553553
limits:
554554
memory: 256Mi
555+
556+
# Image for the pre-delete hook that releases the VictoriaMetrics objects (see
557+
# templates/vm-cleanup-hook.yaml). Only needs a kubectl binary. Digest-pinned so
558+
# an upstream retag does not change what runs across the fleet; refresh with
559+
# `docker manifest inspect docker.io/clastix/kubectl:v1.32`. Same pin as the
560+
# etcd-operator hook.
561+
# renovate: datasource=docker depName=docker.io/clastix/kubectl
562+
kubectlImage:
563+
repository: docker.io/clastix/kubectl
564+
tag: v1.32
565+
digest: sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
566+
pullPolicy: IfNotPresent

0 commit comments

Comments
 (0)