Skip to content

Commit 7b4739b

Browse files
committed
fix(monitoring): make cleanup deletes non-blocking and guard delete scope
Address review on #3094: - Add --wait=false to both kubectl delete calls so PVC finalizers (pvc-protection on pods still terminating) can never block the post-delete hook and stall the HelmRelease teardown. - Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop. - Tests: assert the non-blocking flag, plus negative assertions that the PVC delete stays label-scoped (no --all) and the secret delete stays name-scoped (no -l/--all). deletecollection RBAC verb intentionally not added: kubectl delete -l lists then deletes individually, so the existing delete verb suffices. BREAKING CHANGE: disabling monitoring now permanently removes its VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold. Signed-off-by: Алексей Артамонов <[email protected]> Assisted-By: Claude <[email protected]>
1 parent 2a2d158 commit 7b4739b

2 files changed

Lines changed: 21 additions & 6 deletions

File tree

‎packages/extra/monitoring/templates/hooks/cleanup.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ metadata:
1010
"helm.sh/hook-weight": "10"
1111
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
1212
spec:
13+
# Backstop: the deletes are best-effort and non-blocking (--wait=false), but
14+
# cap the Job so it can never stall the HelmRelease teardown.
15+
activeDeadlineSeconds: 120
1316
template:
1417
metadata:
1518
labels:
@@ -47,9 +50,9 @@ spec:
4750
- /bin/sh
4851
- -c
4952
- |
50-
kubectl delete pvc -n {{ .Release.Namespace }} -l managed-by=vm-operator --ignore-not-found \
53+
kubectl delete pvc -n {{ .Release.Namespace }} -l managed-by=vm-operator --ignore-not-found --wait=false \
5154
|| echo "WARNING: VM storage PVC cleanup failed for {{ .Release.Name }}; orphaned PVCs may remain" >&2
52-
kubectl delete secret -n {{ .Release.Namespace }} alerta-tls grafana-ingress-tls --ignore-not-found \
55+
kubectl delete secret -n {{ .Release.Namespace }} alerta-tls grafana-ingress-tls --ignore-not-found --wait=false \
5356
|| echo "WARNING: TLS secret cleanup failed for {{ .Release.Name }}; orphaned secrets may remain" >&2
5457
volumeMounts:
5558
- name: tmp

‎packages/extra/monitoring/tests/cleanup_test.yaml‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,10 @@ tests:
2525
- equal:
2626
path: metadata.annotations["helm.sh/hook-delete-policy"]
2727
value: before-hook-creation,hook-succeeded
28+
# Backstop so a stuck delete can never stall the HelmRelease teardown.
29+
- equal:
30+
path: spec.activeDeadlineSeconds
31+
value: 120
2832
- equal:
2933
path: spec.template.metadata.labels["policy.cozystack.io/allow-to-apiserver"]
3034
value: "true"
@@ -53,23 +57,31 @@ tests:
5357
path: spec.template.spec.containers[0].securityContext.capabilities.drop
5458
content: ALL
5559

56-
- it: deletes the vm-operator storage PVCs by label
60+
- it: deletes the vm-operator storage PVCs by label without blocking teardown
5761
documentSelector:
5862
path: kind
5963
value: Job
6064
asserts:
6165
- matchRegex:
6266
path: spec.template.spec.containers[0].command[2]
63-
pattern: kubectl delete pvc -n tenant-root -l managed-by=vm-operator --ignore-not-found
67+
pattern: kubectl delete pvc -n tenant-root -l managed-by=vm-operator --ignore-not-found --wait=false
68+
# Never widen the PVC sweep beyond the vm-operator label scope.
69+
- notMatchRegex:
70+
path: spec.template.spec.containers[0].command[2]
71+
pattern: kubectl delete pvc[^\n]*--all
6472

65-
- it: deletes the cert-manager TLS secrets by name
73+
- it: deletes the cert-manager TLS secrets by name without blocking teardown
6674
documentSelector:
6775
path: kind
6876
value: Job
6977
asserts:
7078
- matchRegex:
7179
path: spec.template.spec.containers[0].command[2]
72-
pattern: kubectl delete secret -n tenant-root alerta-tls grafana-ingress-tls --ignore-not-found
80+
pattern: kubectl delete secret -n tenant-root alerta-tls grafana-ingress-tls --ignore-not-found --wait=false
81+
# The secret deletion must stay name-scoped — never a label or --all sweep.
82+
- notMatchRegex:
83+
path: spec.template.spec.containers[0].command[2]
84+
pattern: kubectl delete secret[^\n]*(-l |--all)
7385

7486
- it: creates the cleanup ServiceAccount as a post-delete hook
7587
documentSelector:

0 commit comments

Comments
 (0)