Skip to content

Commit 6c69c00

Browse files
authored
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does When the `monitoring` tenant module is disabled, the VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left orphaned in the tenant namespace. This adds a `post-delete` cleanup hook (`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that: - deletes the VM/VL storage PVCs scoped to this release via `-l apps.cozystack.io/application.name={{ .Release.Name }}-system`. That label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the selector matches and stays scoped to this monitoring release. `managed-by=vm-operator` is intentionally **not** used — vm-operator does not reliably stamp it onto the storage PVCs; - deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name (cert-manager TLS secrets carry only the cluster-wide `controller.cert-manager.io/fao=true` label, so a label sweep would be unsafe — they are removed by name). CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally **not** targeted — CNPG cleans those up itself, and they do not carry `apps.cozystack.io/application.name`, so the selector leaves them untouched. **Upgrade path:** the claim-template label only lands on PVCs created after the change, so **migration 50** (`packages/core/platform/images/migrations/migrations/50`) backfills `apps.cozystack.io/application.name` onto the pre-existing VM/VL storage PVCs of every cozystack-managed monitoring cluster, so the cleanup reclaims storage on already-deployed clusters too. A `test` Makefile target and helm-unittest suites are added for both the cleanup hook and the claim-template label. The cleanup Job is hardened: non-root, read-only root fs, dropped capabilities, `seccompProfile: RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`, `backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC get/list/delete and the two named secrets. Failures are logged but never block teardown. > [!WARNING] > **Disabling the monitoring module now permanently deletes its VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold.** This is intentional and fixes the storage leak. Back up before disabling if the data matters. Fixes #3091. ### Release note ```release-note fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Helm **post-delete** monitoring cleanup job to best-effort remove managed PVCs and two specific TLS secrets. * Included dedicated cleanup RBAC and hardened pod security settings for the cleanup job. * **Tests** * Added Helm chart tests validating hook weight/policies, job settings, exact deletion commands, and negative cases. * Added a `make test` target to run Helm unit tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents a28ef9b + ff037c0 commit 6c69c00

9 files changed

Lines changed: 633 additions & 1 deletion

File tree

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for packages/core/platform/images/migrations/migrations/51
4+
#
5+
# Migration 51 backfills apps.cozystack.io/application.name onto the pre-existing
6+
# VictoriaMetrics/VictoriaLogs storage PVCs of the monitoring module so the
7+
# post-delete cleanup hook (PR #3094) can reclaim storage on already-deployed
8+
# clusters. Its correctness property is that it must relabel EXACTLY the
9+
# monitoring cluster's vmstorage/vmselect/vlstorage PVCs — never a user's own
10+
# VMCluster, never the CNPG db PVCs — and stamp the version on a clean run.
11+
# helm-unittest cannot check this (it inspects rendered manifests, not the
12+
# operator's runtime PVCs); a mis-scoped or no-op selector is exactly the class
13+
# of bug this PR iterated on, so these tests pin the runtime behaviour.
14+
#
15+
# We put a fake `kubectl` on PATH that logs every call and simulates one tenant
16+
# namespace holding: a monitoring VMCluster (shortterm) + VLCluster (generic)
17+
# whose spec.managedMetadata carries application.name=monitoring-system; a user's
18+
# own VMCluster (custom) with no such label; plus the operator storage PVCs and
19+
# the CNPG db PVCs. We assert the label log:
20+
# - relabels the shortterm vmstorage/vmselect and generic vlstorage PVCs;
21+
# - never touches the CNPG db PVCs (grafana-db/alerta-db);
22+
# - never touches the user's `custom` VMCluster PVCs (no managedMetadata label);
23+
# - stamps the version (kubectl apply) on a clean run;
24+
# - is a safe no-op that still stamps when the CRDs are absent.
25+
#
26+
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
27+
# own line (column 0); assertions are direct shell tests, and the fake-kubectl
28+
# heredoc keeps every `}` off column 0. Each @test cleans up inline (no
29+
# EXIT/RETURN trap — see docs/agents/e2e-testing.md §3).
30+
#
31+
# Run with: hack/cozytest.sh hack/monitoring-pvc-backfill-migration.bats
32+
# (or `bats hack/monitoring-pvc-backfill-migration.bats`)
33+
# -----------------------------------------------------------------------------
34+
35+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
36+
REPO_ROOT="$(cd "$HACK_DIR/.." && pwd)"
37+
MIGRATION="$REPO_ROOT/packages/core/platform/images/migrations/migrations/51"
38+
39+
# write_fake_kubectl <dir>
40+
# Emits a fake `kubectl` that logs every call to $KLOG and, when $MOCK_ABSENT=1,
41+
# reports the VM/VL CRDs as absent. Ordering matters: the CR appname read is
42+
# matched by *managedMetadata* before the generic *vmclusters*/*vlclusters* list.
43+
# No line below is a bare column-0 `}`, so cozytest.sh's parser leaves it intact.
44+
write_fake_kubectl() {
45+
cat > "$1/kubectl" <<'KEOF'
46+
#!/bin/sh
47+
echo "$*" >> "$KLOG"
48+
case "$*" in
49+
*"get crd"*)
50+
[ "${MOCK_ABSENT:-}" = 1 ] && exit 1
51+
exit 0
52+
;;
53+
*label*pvc*)
54+
exit 0
55+
;;
56+
*apply*)
57+
cat >/dev/null
58+
exit 0
59+
;;
60+
*managedMetadata*)
61+
case "$*" in
62+
*shortterm*|*generic*) printf 'monitoring-system' ;;
63+
*) : ;;
64+
esac
65+
exit 0
66+
;;
67+
*vmclusters*)
68+
printf 'tenant-test shortterm\ntenant-test custom\n'
69+
exit 0
70+
;;
71+
*vlclusters*)
72+
printf 'tenant-test generic\n'
73+
exit 0
74+
;;
75+
*"get pvc"*"items[*]"*)
76+
printf 'vmstorage-db-vmstorage-shortterm-0\n'
77+
printf 'vmselect-cachedir-vmselect-shortterm-0\n'
78+
printf 'vmstorage-db-vmstorage-custom-0\n'
79+
printf 'vlstorage-db-vlstorage-generic-0\n'
80+
printf 'grafana-db-1\n'
81+
printf 'alerta-db-1\n'
82+
exit 0
83+
;;
84+
*"get pvc"*)
85+
exit 0
86+
;;
87+
*)
88+
exit 0
89+
;;
90+
esac
91+
KEOF
92+
chmod +x "$1/kubectl"
93+
}
94+
95+
@test "migration 51 relabels only the monitoring VM/VL storage PVCs and stamps the version" {
96+
tmp=$(mktemp -d)
97+
export KLOG="$tmp/kubectl.log"
98+
: > "$KLOG"
99+
write_fake_kubectl "$tmp"
100+
101+
if ! KLOG="$KLOG" PATH="$tmp:$PATH" "$MIGRATION" >"$tmp/out" 2>&1; then
102+
echo "expected a clean migration run to exit zero; output:" >&2
103+
cat "$tmp/out" >&2
104+
rm -rf "$tmp"
105+
exit 1
106+
fi
107+
108+
# The monitoring cluster's storage PVCs must be relabeled with the exact value.
109+
for pvc in vmstorage-db-vmstorage-shortterm-0 vmselect-cachedir-vmselect-shortterm-0 vlstorage-db-vlstorage-generic-0; do
110+
if ! grep -Eq "label pvc -n tenant-test $pvc apps.cozystack.io/application.name=monitoring-system --overwrite" "$KLOG"; then
111+
echo "expected $pvc to be relabeled with application.name=monitoring-system" >&2
112+
cat "$KLOG" >&2
113+
rm -rf "$tmp"
114+
exit 1
115+
fi
116+
done
117+
118+
# The CNPG db PVCs must never be touched (they are not monitoring storage).
119+
if grep -Eq "label pvc.*(grafana-db|alerta-db)" "$KLOG"; then
120+
echo "regression: cleanup relabeled a CNPG db PVC" >&2
121+
cat "$KLOG" >&2
122+
rm -rf "$tmp"
123+
exit 1
124+
fi
125+
126+
# A user's own VMCluster (no managedMetadata label) must be skipped entirely.
127+
if grep -Eq "label pvc.*custom" "$KLOG"; then
128+
echo "regression: relabeled a non-cozystack VMCluster's PVC" >&2
129+
cat "$KLOG" >&2
130+
rm -rf "$tmp"
131+
exit 1
132+
fi
133+
134+
# Positive control: the version stamp (kubectl apply) ran on a clean pass.
135+
if ! grep -q apply "$KLOG"; then
136+
echo "expected the version stamp (kubectl apply) to run on a clean migration" >&2
137+
cat "$KLOG" >&2
138+
rm -rf "$tmp"
139+
exit 1
140+
fi
141+
rm -rf "$tmp"
142+
}
143+
144+
@test "migration 51 is a safe no-op that still stamps when the VM/VL CRDs are absent" {
145+
tmp=$(mktemp -d)
146+
export KLOG="$tmp/kubectl.log"
147+
: > "$KLOG"
148+
write_fake_kubectl "$tmp"
149+
150+
if ! KLOG="$KLOG" MOCK_ABSENT=1 PATH="$tmp:$PATH" "$MIGRATION" >"$tmp/out" 2>&1; then
151+
echo "expected the CRD-absent path to exit zero; output:" >&2
152+
cat "$tmp/out" >&2
153+
rm -rf "$tmp"
154+
exit 1
155+
fi
156+
157+
# No CRD -> no PVC is relabeled ...
158+
if grep -q "label pvc" "$KLOG"; then
159+
echo "expected no relabel when the VM/VL CRDs are absent" >&2
160+
cat "$KLOG" >&2
161+
rm -rf "$tmp"
162+
exit 1
163+
fi
164+
165+
# ... but the version stamp must still advance so the upgrade completes.
166+
if ! grep -q apply "$KLOG"; then
167+
echo "expected the version stamp to run even when the CRDs are absent" >&2
168+
cat "$KLOG" >&2
169+
rm -rf "$tmp"
170+
exit 1
171+
fi
172+
rm -rf "$tmp"
173+
}
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
#!/bin/bash
2+
# Migration 51 --> 52
3+
# Backfill the apps.cozystack.io/application.name release label onto pre-existing
4+
# VictoriaMetrics / VictoriaLogs storage PVCs of the monitoring module.
5+
#
6+
# PR #3094 adds apps.cozystack.io/application.name to the VMCluster/VLCluster
7+
# storage volumeClaimTemplates (packages/system/monitoring) and a post-delete
8+
# hook that garbage-collects PVCs carrying that label. But vm-operator only
9+
# stamps volumeClaimTemplate.metadata.labels onto PVCs it creates *after* the
10+
# change; the Kubernetes StatefulSet controller applies claim-template labels
11+
# only at PVC creation and never re-labels existing PVCs, and vm-operator's PVC
12+
# reconciler is resize-only. Storage provisioned before this upgrade therefore
13+
# stays unlabeled, the cleanup hook's selector misses it, and it keeps leaking on
14+
# delete (issue #3091). This migration relabels those PVCs once so the hook can
15+
# reclaim them.
16+
#
17+
# The label value is read from each CR's spec.managedMetadata so we relabel only
18+
# cozystack-managed monitoring clusters (a user's own VMCluster carries no such
19+
# label and is left untouched) and always stamp the exact value the hook selects.
20+
#
21+
# Best-effort by design: the worst case if a relabel is skipped is the
22+
# pre-existing leak this PR set out to fix, which is strictly no worse than the
23+
# status quo. A transient apiserver hiccup must therefore NOT abort the whole
24+
# platform upgrade, so the CR list and per-PVC reads tolerate failure (|| true)
25+
# rather than exiting non-zero.
26+
27+
set -euo pipefail
28+
29+
# shellcheck source=lib/cozystack-version.sh
30+
. "$(dirname "$0")/lib/cozystack-version.sh"
31+
32+
label_key="apps.cozystack.io/application.name"
33+
# jsonpath escapes the dots/slash in the label key.
34+
label_path='{.spec.managedMetadata.labels.apps\.cozystack\.io/application\.name}'
35+
36+
# relabel_storage <crd> <pvc-name-regex>
37+
# For every CR of <crd> that carries the cozystack application.name label, relabel
38+
# its storage PVCs (matched by the StatefulSet-derived <pvc-name-regex>, with the
39+
# CR name spliced in) to the same value.
40+
relabel_storage() {
41+
crd="$1"
42+
pvc_regex_tmpl="$2"
43+
44+
# This migration runs platform-wide (targetVersion bump), including on clusters
45+
# that never installed monitoring. Skip when the CRD is absent. The `|| true`
46+
# on the list additionally keeps a transient apiserver error from aborting the
47+
# upgrade — there is nothing safety-critical to gate on.
48+
if ! kubectl get crd "$crd" >/dev/null 2>&1; then
49+
echo "$crd not present; skipping monitoring PVC label backfill"
50+
return 0
51+
fi
52+
53+
crs=$(kubectl get "$crd" -A \
54+
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' || true)
55+
56+
echo "$crs" | while read -r ns name; do
57+
[ -n "${ns:-}" ] || continue
58+
59+
appname=$(kubectl get "$crd" -n "$ns" "$name" -o jsonpath="$label_path" 2>/dev/null || true)
60+
if [ -z "$appname" ]; then
61+
echo "$crd $ns/$name has no $label_key; not a cozystack-managed cluster, skipping"
62+
continue
63+
fi
64+
65+
# Splice the CR name into the PVC regex (e.g. "(vmstorage|vmselect)-NAME-[0-9]+$").
66+
pvc_regex=$(printf '%s' "$pvc_regex_tmpl" | sed "s/NAME/${name}/g")
67+
68+
# grep exits 1 when a namespace has no matching PVCs; tolerate it so pipefail
69+
# does not abort the whole migration on an empty match.
70+
pvcs=$(kubectl get pvc -n "$ns" \
71+
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null \
72+
| grep -E "$pvc_regex" || true)
73+
74+
for pvc in $pvcs; do
75+
[ -n "${pvc:-}" ] || continue
76+
current=$(kubectl get pvc -n "$ns" "$pvc" \
77+
-o jsonpath="{.metadata.labels.apps\.cozystack\.io/application\.name}" 2>/dev/null || true)
78+
if [ "$current" = "$appname" ]; then
79+
echo "PVC $ns/$pvc already labeled $label_key=$appname; skipping"
80+
continue
81+
fi
82+
echo "Labeling PVC $ns/$pvc with $label_key=$appname"
83+
kubectl label pvc -n "$ns" "$pvc" "$label_key=$appname" --overwrite
84+
done
85+
done
86+
}
87+
88+
# vm-operator storage PVCs are named "<claim-template>-<sts>-<ordinal>" and the
89+
# StatefulSet is "<component>-<cr>", so every PVC ends with
90+
# "<component>-<cr>-<ordinal>". The regex suffix-matches that tail, so it is
91+
# robust to the claim-template prefix (e.g. vmstorage-db-vmstorage-<cr>-0 and
92+
# vmselect-cachedir-vmselect-<cr>-0 both match on the vmselect/vmstorage arm).
93+
relabel_storage vmclusters.operator.victoriametrics.com '(vmstorage|vmselect)-NAME-[0-9]+$'
94+
# vlstorage STS PVCs are "<claim-template>-vlstorage-<cr>-<ordinal>".
95+
relabel_storage vlclusters.operator.victoriametrics.com 'vlstorage-NAME-[0-9]+$'
96+
97+
# Stamp version via the shared helper so the cozystack-version ConfigMap keeps
98+
# the platform.cozystack.io/no-delete label.
99+
stamp_cozystack_version 52

‎packages/core/platform/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ migrations:
1414
# refuses to advance past a migration file missing from the image (exit 1), so
1515
# a stale pin fails loudly rather than silently skipping the etcd adoption.
1616
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.5.0@sha256:8bf61f17e99eb4a3365394e3d97ac52a1d68cc84aa104894ecb09de25721dc1d
17-
targetVersion: 51
17+
targetVersion: 52
1818
# Bundle deployment configuration
1919
bundles:
2020
system:

‎packages/extra/monitoring/Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ NAME=monitoring
55
include ../../../hack/common-envs.mk
66
include ../../../hack/package.mk
77

8+
.PHONY: test
9+
test:
10+
helm unittest .
11+
812
generate:
913
cozyvalues-gen -v values.yaml -s values.schema.json -r README.md
1014
../../../hack/update-crd.sh

0 commit comments

Comments
 (0)