Skip to content

Commit 0892c5a

Browse files
authored
fix(qdrant)!: clean up orphaned data PVC on delete (#3059) (#3075)
## What this PR does After deleting a `Qdrant` app CR, the StatefulSet data PVC (`qdrant-storage-<release>-0`) was left orphaned in the tenant namespace, consuming storage permanently. StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion by default. The vendored upstream Qdrant chart does **not** support `persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml` never renders that field), so the OpenBAO-style value passthrough is not possible here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the MariaDB hook) that deletes the release's PVCs by label. The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by every Qdrant in a namespace), so to select PVCs safely per release this also passes `persistence.additionalLabels.app.kubernetes.io/instance: <release>` through the HelmRelease — giving the data PVC a release-unique label the hook selects on. The cleanup Job is hardened: runs as non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped to `persistentvolumeclaims`. A real cleanup failure is logged (not silently swallowed) but never blocks namespace teardown. > [!WARNING] > **Deleting a Qdrant app now permanently removes its data PVC and the vector data it holds.** > This is intentional and fixes the storage leak; the hook only fires on full > release deletion (not on scale-down). Back up before deleting. Fixes #3059. ### Release note ```release-note fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a post-delete cleanup hook for Qdrant to remove leftover StatefulSet data PVCs after deletion. * Included hardened Kubernetes settings for the cleanup job to improve runtime safety. * **Tests** * Added Helm chart tests that validate the post-delete cleanup job and its RBAC permissions. * Enhanced end-to-end verification to confirm the data PVC is fully removed after deleting a Qdrant instance. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents bf14a73 + ae7c232 commit 0892c5a

3 files changed

Lines changed: 205 additions & 0 deletions

File tree

‎hack/e2e-apps/qdrant.bats‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,4 +27,22 @@ EOF
2727
timeout 60 sh -ec "until kubectl -n tenant-test get pvc qdrant-storage-qdrant-$name-0 >/dev/null 2>&1; do sleep 2; done"
2828
kubectl -n tenant-test wait pvc qdrant-storage-qdrant-$name-0 --timeout=50s --for=jsonpath='{.status.phase}'=Bound
2929
kubectl -n tenant-test delete qdrant.apps.cozystack.io $name
30+
# Issue #3059: Kubernetes never garbage-collects StatefulSet-templated PVCs,
31+
# so the post-delete cleanup hook must reclaim the data PVC. The hook Job runs
32+
# asynchronously during helm uninstall — wait for the PVC to actually be gone.
33+
# Gate on a NotFound result specifically: a transient API/auth error returns
34+
# non-zero too, and must not be misread as "PVC reclaimed".
35+
pvc="qdrant-storage-qdrant-$name-0"
36+
deleted=false
37+
for _ in $(seq 1 60); do
38+
if err=$(kubectl -n tenant-test get pvc "$pvc" 2>&1 >/dev/null); then
39+
sleep 2; continue # PVC still exists
40+
fi
41+
case "$err" in
42+
*NotFound*) deleted=true; break ;;
43+
*) echo "transient error querying $pvc, retrying: $err" >&2; sleep 2 ;;
44+
esac
45+
done
46+
[ "$deleted" = true ] \
47+
|| { echo "❌ data PVC $pvc not reclaimed after Qdrant delete (issue #3059 regression)"; kubectl -n tenant-test get pvc -o wide; kubectl -n tenant-test get jobs,pods -l app.kubernetes.io/instance=qdrant-$name -o wide; false; }
3048
}
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: {{ .Release.Name }}-qdrant-cleanup
6+
labels:
7+
app.kubernetes.io/instance: {{ .Release.Name }}
8+
annotations:
9+
"helm.sh/hook": post-delete
10+
"helm.sh/hook-weight": "10"
11+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
12+
spec:
13+
template:
14+
metadata:
15+
labels:
16+
app.kubernetes.io/instance: {{ .Release.Name }}
17+
policy.cozystack.io/allow-to-apiserver: "true"
18+
spec:
19+
serviceAccountName: {{ .Release.Name }}-qdrant-cleanup
20+
restartPolicy: Never
21+
securityContext:
22+
runAsNonRoot: true
23+
runAsUser: 65534
24+
runAsGroup: 65534
25+
seccompProfile:
26+
type: RuntimeDefault
27+
containers:
28+
- name: cleanup
29+
image: docker.io/clastix/kubectl:v1.32
30+
securityContext:
31+
allowPrivilegeEscalation: false
32+
readOnlyRootFilesystem: true
33+
capabilities:
34+
drop: ["ALL"]
35+
env:
36+
- name: HOME
37+
value: /tmp
38+
resources:
39+
requests:
40+
cpu: 10m
41+
memory: 64Mi
42+
limits:
43+
cpu: 200m
44+
memory: 128Mi
45+
command:
46+
- /bin/sh
47+
- -c
48+
- |
49+
# StatefulSet-templated data PVCs are named
50+
# qdrant-storage-<release>-<ordinal> (volumeClaimTemplate name
51+
# "qdrant-storage" + StatefulSet name <release> + pod ordinal).
52+
# Select by that deterministic name rather than a label: the
53+
# upstream chart stamps volumeClaimTemplates labels, which are
54+
# immutable on an existing StatefulSet, so a label-based scheme
55+
# cannot be applied to already-deployed instances. Name selection
56+
# needs no StatefulSet change and also reclaims PVCs created before
57+
# this hook existed. The ordinal is anchored ("-[0-9]+$") so a
58+
# release whose name is a prefix of a sibling (e.g. "qdrant-test"
59+
# vs "qdrant-test-2") never matches the sibling's PVCs and deletes
60+
# its data. See #3059.
61+
echo "Deleting orphaned data PVCs for release {{ .Release.Name }}..."
62+
pvcs=$(kubectl get pvc -n {{ .Release.Namespace }} -o name \
63+
| grep -E "^persistentvolumeclaim/qdrant-storage-{{ .Release.Name }}-[0-9]+$" || true)
64+
if [ -z "$pvcs" ]; then
65+
echo "No data PVCs found for release {{ .Release.Name }}; nothing to delete."
66+
else
67+
echo "$pvcs" | xargs kubectl delete -n {{ .Release.Namespace }} --ignore-not-found \
68+
|| echo "WARNING: PVC cleanup failed for {{ .Release.Name }}; orphaned PVCs may remain" >&2
69+
fi
70+
echo "PVC cleanup complete."
71+
volumeMounts:
72+
- name: tmp
73+
mountPath: /tmp
74+
volumes:
75+
- name: tmp
76+
emptyDir: {}
77+
---
78+
apiVersion: v1
79+
kind: ServiceAccount
80+
metadata:
81+
name: {{ .Release.Name }}-qdrant-cleanup
82+
labels:
83+
app.kubernetes.io/instance: {{ .Release.Name }}
84+
annotations:
85+
"helm.sh/hook": post-delete
86+
helm.sh/hook-weight: "0"
87+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
88+
---
89+
apiVersion: rbac.authorization.k8s.io/v1
90+
kind: Role
91+
metadata:
92+
name: {{ .Release.Name }}-qdrant-cleanup
93+
labels:
94+
app.kubernetes.io/instance: {{ .Release.Name }}
95+
annotations:
96+
"helm.sh/hook": post-delete
97+
"helm.sh/hook-weight": "5"
98+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
99+
rules:
100+
- apiGroups: [""]
101+
resources: ["persistentvolumeclaims"]
102+
verbs: ["get", "list", "delete"]
103+
---
104+
apiVersion: rbac.authorization.k8s.io/v1
105+
kind: RoleBinding
106+
metadata:
107+
name: {{ .Release.Name }}-qdrant-cleanup
108+
labels:
109+
app.kubernetes.io/instance: {{ .Release.Name }}
110+
annotations:
111+
"helm.sh/hook": post-delete
112+
helm.sh/hook-weight: "5"
113+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
114+
roleRef:
115+
apiGroup: rbac.authorization.k8s.io
116+
kind: Role
117+
name: {{ .Release.Name }}-qdrant-cleanup
118+
subjects:
119+
- kind: ServiceAccount
120+
name: {{ .Release.Name }}-qdrant-cleanup
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
suite: qdrant data PVC cleanup on delete
2+
3+
# Guards the fix for issue #3059: the upstream Qdrant chart does NOT support
4+
# persistentVolumeClaimRetentionPolicy, so its StatefulSet-templated data PVC
5+
# (qdrant-storage-<release>-0) is orphaned when the StatefulSet is deleted.
6+
#
7+
# A post-delete hook reclaims the data by deleting PVCs whose name matches the
8+
# deterministic StatefulSet template prefix (qdrant-storage-<release>-). A
9+
# name-prefix scheme is used instead of a label because volumeClaimTemplates
10+
# labels are immutable on an existing StatefulSet — routing a release-unique
11+
# label through the subchart would break reconciliation for already-deployed
12+
# instances and still would not match their pre-existing PVCs.
13+
14+
release:
15+
name: qdrant-test
16+
namespace: tenant-test
17+
18+
tests:
19+
#####################################################
20+
# post-delete cleanup hook #
21+
#####################################################
22+
23+
- it: "renders a post-delete Job that deletes the data PVC by anchored name"
24+
templates:
25+
- templates/hooks/cleanup-pvc.yaml
26+
documentSelector:
27+
path: kind
28+
value: Job
29+
asserts:
30+
- isKind:
31+
of: Job
32+
- equal:
33+
path: metadata.annotations["helm.sh/hook"]
34+
value: post-delete
35+
- equal:
36+
path: spec.template.spec.serviceAccountName
37+
value: qdrant-test-qdrant-cleanup
38+
- matchRegex:
39+
path: spec.template.spec.containers[0].command[2]
40+
pattern: "kubectl get pvc -n tenant-test -o name"
41+
# The ordinal MUST be anchored ("-[0-9]+$"). Without it, deleting release
42+
# "qdrant-test" would also match a sibling "qdrant-test-2"'s PVCs
43+
# (qdrant-storage-qdrant-test-2-0) and irreversibly delete its data.
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].command[2]
46+
pattern: 'grep -E "\^persistentvolumeclaim/qdrant-storage-qdrant-test-\[0-9\]\+\$"'
47+
# Guard against regressing to the unanchored prefix (plain grep, no $).
48+
- notMatchRegex:
49+
path: spec.template.spec.containers[0].command[2]
50+
pattern: 'grep "\^persistentvolumeclaim/qdrant-storage-qdrant-test-"'
51+
52+
- it: "creates qdrant-scoped RBAC for the cleanup hook"
53+
templates:
54+
- templates/hooks/cleanup-pvc.yaml
55+
documentSelector:
56+
path: kind
57+
value: Role
58+
asserts:
59+
- equal:
60+
path: metadata.name
61+
value: qdrant-test-qdrant-cleanup
62+
- contains:
63+
path: rules
64+
content:
65+
apiGroups: [""]
66+
resources: ["persistentvolumeclaims"]
67+
verbs: ["get", "list", "delete"]

0 commit comments

Comments
 (0)