Skip to content

Commit 628ebce

Browse files
feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy (#2650)
## What this PR does Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical platform objects labeled `platform.cozystack.io/no-delete=true`. Native, in-process CEL evaluation — no webhook DaemonSet, no Service, no TLS, no image to maintain. Requires Kubernetes 1.30+. This is an alternative to the closed #2402 (admission webhook approach). ### Protected objects (labeled in this PR) | Resource | Kind | Source | |---|---|---| | `cozy-system` | Namespace | `packages/core/installer/templates/cozy-system-{namespace,labels}.yaml` | | `tenant-root` | Namespace | `packages/system/cozystack-basics/templates/tenant-root.yaml` | | `tenant-root` | HelmRelease (ns: tenant-root) | same | | `cozystack-version` | ConfigMap (ns: cozy-system) | `packages/core/platform/templates/cozystack-version.yaml` | | `cozystack-packages` | OCIRepository (ns: cozy-system) | `packages/core/platform/templates/repository.yaml` | | `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` | ClusterIssuer | `packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` | | `linstorcluster` | LinstorCluster | `packages/system/linstor/templates/cluster.yaml` | | `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label stamped in `internal/crdinstall/install.go` | ### Bypass ```bash kubectl label <kind> <name> platform.cozystack.io/no-delete- kubectl delete <kind> <name> ``` ### Why VAP instead of a webhook Same outcome as #2402 but with zero runtime infrastructure: no DaemonSet, Service, TLS certificate, or image. The kube-apiserver evaluates the CEL `expression: "false"` in-process and denies because `validationActions: [Deny]`. `objectSelector` on the binding scopes evaluation to labeled objects only — every other DELETE in the cluster is unaffected. ### Release note ```release-note feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy) blocking DELETE on cozystack platform objects labeled `platform.cozystack.io/no-delete=true`. Bypass by removing the label first. Requires Kubernetes 1.30+. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Deletion-protection guardrail: objects labeled platform.cozystack.io/no-delete=true are denied DELETE requests; removal of the label allows deletion. * **Chores** * Platform migration to v43 backfills the no-delete label onto system resources and namespaces. * **Tools** * Added an "unprotect" teardown utility to remove the no-delete label for uninstall/disaster recovery. * **Documentation** * Helm NOTES now warns when the guardrail is inactive due to missing cluster API. * **Tests** * New tests and e2e checks validate deletion-protection behavior. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 72c4eb9 + a58c0d1 commit 628ebce

19 files changed

Lines changed: 520 additions & 12 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,3 +282,84 @@ EOF
282282
-o jsonpath='{range .items[*].spec.limits[*]}{.default.cpu}{" "}{.default.memory}{" "}{.defaultRequest.cpu}{" "}{.defaultRequest.memory}{"\n"}{end}' \
283283
| grep -qx '250m 128Mi 25m 128Mi'
284284
}
285+
286+
@test "Deletion-protection VAP denies delete on labeled cozystack-version ConfigMap" {
287+
# Locks down the contract delivered by packages/core/platform/templates/
288+
# deletion-protection.yaml: a DELETE on any object carrying
289+
# platform.cozystack.io/no-delete=true is rejected by the
290+
# ValidatingAdmissionPolicy with the documented message, and the bypass
291+
# path (remove the label, then delete) succeeds.
292+
#
293+
# This covers every regression the PR is meant to prevent in a single
294+
# pass: capability gate inverted, binding objectSelector mistyped,
295+
# validationActions flipped Deny→Warn, expression flipped false→true,
296+
# label-key drift between the binding and the manifests.
297+
298+
# Preflight: VAP requires Kubernetes 1.30+. Skip on older clusters so
299+
# the suite stays green where the capability gate intentionally elides
300+
# the policy. Detect by attempting to fetch the policy by name; if the
301+
# API is present, the resource will be retrievable, otherwise kubectl
302+
# exits non-zero on an unknown resource type.
303+
if ! kubectl api-resources --api-group=admissionregistration.k8s.io \
304+
2>/dev/null | grep -qw validatingadmissionpolicies; then
305+
skip "ValidatingAdmissionPolicy API not available on this cluster"
306+
fi
307+
kubectl get validatingadmissionpolicy cozystack-no-delete-guardrail
308+
309+
# The cozystack-version ConfigMap is created with the no-delete label
310+
# baked in by the chart (templates/cozystack-version.yaml) and is
311+
# backfilled by the migration on upgrades. Asserting the label is the
312+
# precondition for the deny check below: if the label is gone the deny
313+
# will not fire and the test would misreport as a pass on a regressed
314+
# binding.
315+
kubectl get configmap cozystack-version -n cozy-system \
316+
-o jsonpath='{.metadata.labels.platform\.cozystack\.io/no-delete}' \
317+
| grep -qx 'true'
318+
319+
# The actual deny check. Capture both stdout+stderr and exit code so a
320+
# network/auth failure does not silently look like a deny success.
321+
local output rc
322+
output=$(kubectl delete configmap cozystack-version -n cozy-system 2>&1) \
323+
&& rc=0 || rc=$?
324+
echo "kubectl delete exit=$rc, output=$output"
325+
# Delete MUST have failed: success means the VAP regressed.
326+
[ "$rc" -ne 0 ]
327+
# Assert the user-facing deny message is the one this PR ships — guards
328+
# against expression flipped to "true" or message reworded away from the
329+
# documented bypass. The CEL message is on one line in the api-server
330+
# response, so grep for the literal substring with the --namespace flag.
331+
echo "$output" | grep -q 'Deletion blocked: object carries platform.cozystack.io/no-delete=true'
332+
echo "$output" | grep -q -- '--namespace'
333+
334+
# And confirm the ConfigMap is still there — a partial deny that races
335+
# tombstone creation would also be a regression.
336+
kubectl get configmap cozystack-version -n cozy-system >/dev/null
337+
338+
# Bypass path: remove the label, delete must succeed. Re-stamp the label
339+
# afterward so the cluster ends the test in the same state it started.
340+
kubectl label configmap cozystack-version -n cozy-system \
341+
platform.cozystack.io/no-delete- --overwrite
342+
# Stash the data so we can reconstruct after delete.
343+
local version
344+
version=$(kubectl get configmap cozystack-version -n cozy-system \
345+
-o jsonpath='{.data.version}')
346+
kubectl delete configmap cozystack-version -n cozy-system
347+
! kubectl get configmap cozystack-version -n cozy-system 2>/dev/null
348+
# Reconstruct: declarative apply matches the chart template at
349+
# packages/core/platform/templates/cozystack-version.yaml — same label set
350+
# AND the helm.sh/resource-policy: keep annotation that pins the ConfigMap
351+
# across helm uninstall.
352+
cat <<EOF | kubectl apply -f -
353+
apiVersion: v1
354+
kind: ConfigMap
355+
metadata:
356+
name: cozystack-version
357+
namespace: cozy-system
358+
labels:
359+
platform.cozystack.io/no-delete: "true"
360+
annotations:
361+
helm.sh/resource-policy: keep
362+
data:
363+
version: "${version}"
364+
EOF
365+
}

‎internal/crdinstall/install.go‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,17 @@ func Install(ctx context.Context, k8sClient client.Client, writeEmbeddedManifest
8989
}
9090
}
9191

92+
// Stamp the deletion-protection label so the platform VAP guards every
93+
// Cozystack CRD against accidental kubectl delete.
94+
for _, obj := range objects {
95+
labels := obj.GetLabels()
96+
if labels == nil {
97+
labels = make(map[string]string)
98+
}
99+
labels["platform.cozystack.io/no-delete"] = "true"
100+
obj.SetLabels(labels)
101+
}
102+
92103
logger.Info("Applying Cozystack CRDs", "count", len(objects))
93104
for _, obj := range objects {
94105
patchOptions := &client.PatchOptions{

‎internal/crdinstall/install_test.go‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,77 @@ func TestInstall_appliesAllCRDs(t *testing.T) {
212212
if err != nil {
213213
t.Fatalf("Install() error = %v", err)
214214
}
215+
216+
for _, name := range []string{"packages.cozystack.io", "packagesources.cozystack.io"} {
217+
applied := &unstructured.Unstructured{}
218+
applied.SetGroupVersionKind(apiextensionsv1.SchemeGroupVersion.WithKind("CustomResourceDefinition"))
219+
if err := fakeClient.Get(ctx, client.ObjectKey{Name: name}, applied); err != nil {
220+
t.Fatalf("failed to read back applied CRD %s: %v", name, err)
221+
}
222+
labels := applied.GetLabels()
223+
if labels["platform.cozystack.io/no-delete"] != "true" {
224+
t.Errorf("CRD %s missing platform.cozystack.io/no-delete=true label; got labels=%v", name, labels)
225+
}
226+
}
227+
}
228+
229+
// testCRDWithLabels mirrors testCRD1 but already carries a non-cozystack label.
230+
// Exercises the merge path in Install: the existing label must survive, and the
231+
// deletion-protection label must be added alongside it.
232+
var testCRDWithLabels = `apiVersion: apiextensions.k8s.io/v1
233+
kind: CustomResourceDefinition
234+
metadata:
235+
name: packages.cozystack.io
236+
labels:
237+
app.kubernetes.io/managed-by: foo
238+
spec:
239+
group: cozystack.io
240+
names:
241+
kind: Package
242+
plural: packages
243+
scope: Namespaced
244+
versions:
245+
- name: v1alpha1
246+
served: true
247+
storage: true
248+
schema:
249+
openAPIV3Schema:
250+
type: object
251+
`
252+
253+
func TestInstall_preservesExistingLabels(t *testing.T) {
254+
log.SetLogger(zap.New(zap.UseDevMode(true)))
255+
256+
scheme := runtime.NewScheme()
257+
if err := apiextensionsv1.AddToScheme(scheme); err != nil {
258+
t.Fatalf("failed to add apiextensions to scheme: %v", err)
259+
}
260+
261+
fakeClient := fake.NewClientBuilder().
262+
WithScheme(scheme).
263+
WithInterceptorFuncs(establishedInterceptor()).
264+
Build()
265+
266+
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
267+
defer cancel()
268+
ctx = log.IntoContext(ctx, log.FromContext(context.Background()))
269+
270+
if err := Install(ctx, fakeClient, newCRDManifestWriter(testCRDWithLabels)); err != nil {
271+
t.Fatalf("Install() error = %v", err)
272+
}
273+
274+
applied := &unstructured.Unstructured{}
275+
applied.SetGroupVersionKind(apiextensionsv1.SchemeGroupVersion.WithKind("CustomResourceDefinition"))
276+
if err := fakeClient.Get(ctx, client.ObjectKey{Name: "packages.cozystack.io"}, applied); err != nil {
277+
t.Fatalf("failed to read back applied CRD: %v", err)
278+
}
279+
labels := applied.GetLabels()
280+
if labels["app.kubernetes.io/managed-by"] != "foo" {
281+
t.Errorf("existing label app.kubernetes.io/managed-by=foo was lost; got labels=%v", labels)
282+
}
283+
if labels["platform.cozystack.io/no-delete"] != "true" {
284+
t.Errorf("deletion-protection label missing after merge; got labels=%v", labels)
285+
}
215286
}
216287

217288
func TestInstall_noManifests(t *testing.T) {

‎packages/core/installer/templates/cozy-system-labels.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,4 +149,5 @@ spec:
149149
- namespace
150150
- cozy-system
151151
- cozystack.io/system=true
152+
- platform.cozystack.io/no-delete=true
152153
- pod-security.kubernetes.io/enforce=privileged

‎packages/core/installer/templates/cozy-system-namespace.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ metadata:
1919
name: cozy-system
2020
labels:
2121
cozystack.io/system: "true"
22+
platform.cozystack.io/no-delete: "true"
2223
pod-security.kubernetes.io/enforce: privileged
2324
annotations:
2425
helm.sh/resource-policy: keep

‎packages/core/platform/Makefile‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,14 @@ reconcile: apply
1717
diff:
1818
cozyhr diff --namespace $(NAMESPACE) $(NAME)
1919

20+
# Strip the platform.cozystack.io/no-delete label from every guarded object so
21+
# `helm uninstall cozy-platform` (and ad-hoc deletes of protected resources)
22+
# is no longer denied by the deletion-protection VAP. After this runs, the
23+
# cluster has no guardrail until the next `helm upgrade` re-stamps the label.
24+
# Intended for teardown / disaster-recovery, not routine use.
25+
unprotect:
26+
hack/unprotect.sh
27+
2028
image: image-migrations
2129

2230
image-migrations:
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
#!/bin/sh
2+
# Strip the platform.cozystack.io/no-delete=true label from every object that
3+
# carries it, so `helm uninstall cozy-platform` (and ad-hoc `kubectl delete`
4+
# on a protected resource) is no longer denied by the deletion-protection
5+
# ValidatingAdmissionPolicy.
6+
#
7+
# Intended for chart teardown and disaster-recovery — NOT a routine operation.
8+
# After this runs, the cluster has no guardrail until the next `helm upgrade`
9+
# re-stamps the label.
10+
#
11+
# Idempotent: a label that is already absent is a no-op.
12+
#
13+
# The set of guarded kinds is enumerated locally rather than discovered
14+
# dynamically because some of the kinds may not exist in every cluster
15+
# (e.g. linstorcluster on a non-LINSTOR install) and kubectl returns an
16+
# error for unknown resource types. The per-kind get is best-effort.
17+
18+
set -eu
19+
20+
LABEL="platform.cozystack.io/no-delete"
21+
22+
# Run `kubectl get` and tolerate only the "unknown resource type" error
23+
# (a kind that doesn't exist in this cluster — e.g. linstorcluster on a
24+
# non-LINSTOR install). Any other error (RBAC, wrong context, API
25+
# unreachable) is fatal — otherwise we'd silently report success while
26+
# leaving labels behind.
27+
#
28+
# Keep stdout (the resource list) and stderr (kubectl warnings, errors)
29+
# separate so unrelated warnings can't sneak into the loop variable.
30+
kubectl_get_or_skip_unknown_kind() {
31+
err=$(mktemp)
32+
if out=$(kubectl get "$@" 2>"$err"); then
33+
rm -f "$err"
34+
printf '%s\n' "$out"
35+
return 0
36+
fi
37+
errmsg=$(cat "$err")
38+
rm -f "$err"
39+
case "$errmsg" in
40+
*"the server doesn't have a resource type"*|\
41+
*"the server could not find the requested resource"*)
42+
return 0
43+
;;
44+
esac
45+
printf '%s\n' "$errmsg" >&2
46+
return 1
47+
}
48+
49+
# Cluster-scoped guarded kinds. CRDs and Namespaces sit here, plus the
50+
# VAP/Binding pair (which is also self-labeled) and LinstorCluster.
51+
CLUSTER_KINDS="
52+
customresourcedefinition
53+
clusterissuer
54+
namespace
55+
validatingadmissionpolicy
56+
validatingadmissionpolicybinding
57+
linstorcluster
58+
"
59+
60+
# Namespaced guarded kinds. ConfigMap, Flux source kinds (OCI/Git), HelmRelease.
61+
# Both OCIRepository and GitRepository are valid platform sourceRef.kind values
62+
# — repository.yaml templates `kind: {{ $sourceRef.kind }}` unrestricted — so
63+
# either may carry the no-delete label and must be enumerated here.
64+
# Use --all-namespaces with -o jsonpath so we get back (ns, name) pairs and
65+
# can pass each to a per-object `kubectl label`.
66+
NS_KINDS="
67+
configmap
68+
ocirepository
69+
gitrepository
70+
helmrelease
71+
"
72+
73+
echo "Removing $LABEL label from all guarded objects..."
74+
75+
for kind in $CLUSTER_KINDS; do
76+
names=$(kubectl_get_or_skip_unknown_kind "$kind" --selector="$LABEL=true" --output=name)
77+
if [ -n "$names" ]; then
78+
# One kubectl per resource — `kubectl label` takes the label op as the
79+
# last argument, so xargs's trailing-args model would put it in the
80+
# wrong position. The protected set is small (<20 objects) so the
81+
# extra round-trips are negligible.
82+
printf '%s\n' "$names" | while read -r ref; do
83+
[ -z "$ref" ] && continue
84+
kubectl label "$ref" "$LABEL-"
85+
done
86+
fi
87+
done
88+
89+
for kind in $NS_KINDS; do
90+
pairs=$(kubectl_get_or_skip_unknown_kind "$kind" --all-namespaces --selector="$LABEL=true" \
91+
--output=jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}')
92+
if [ -n "$pairs" ]; then
93+
printf '%s\n' "$pairs" | while read -r ns name; do
94+
[ -z "$name" ] && continue
95+
kubectl label --namespace "$ns" "$kind" "$name" "$LABEL-"
96+
done
97+
fi
98+
done
99+
100+
echo "Done. helm uninstall is now unblocked."
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/sh
2+
# Migration 42 --> 43
3+
# Backfill the platform.cozystack.io/no-delete=true label on the
4+
# cozystack-version ConfigMap so the deletion-protection
5+
# ValidatingAdmissionPolicy guards it on upgraded clusters.
6+
#
7+
# On first install the chart template renders the ConfigMap with the
8+
# label inline. On upgrades the template skips re-rendering (lookup-if-not
9+
# preserves migration-managed data.version), so the label needs an
10+
# explicit one-shot backfill here.
11+
#
12+
# Idempotent: kubectl apply re-runs are a no-op once the ConfigMap matches
13+
# the declared shape. Same label set the chart bakes in at install time, so
14+
# the next chart-driven apply sees no diff.
15+
#
16+
# 3-way-merge note: this apply records the no-delete label in the kubectl
17+
# last-applied-configuration annotation under the default field manager.
18+
# Any future migration that re-stamps cozystack-version via the same field
19+
# manager MUST include the label inline (matching the chart template at
20+
# templates/cozystack-version.yaml) — a label-less apply by the same field
21+
# manager would strip it.
22+
23+
set -euo pipefail
24+
25+
NAMESPACE="${NAMESPACE:-cozy-system}"
26+
DRY_RUN="${MIGRATION_DRY_RUN:-0}"
27+
28+
MANIFEST=$(cat <<EOF
29+
apiVersion: v1
30+
kind: ConfigMap
31+
metadata:
32+
name: cozystack-version
33+
namespace: ${NAMESPACE}
34+
labels:
35+
platform.cozystack.io/no-delete: "true"
36+
data:
37+
version: "43"
38+
EOF
39+
)
40+
41+
if [ "$DRY_RUN" = "1" ]; then
42+
echo "[dry-run] would apply:"
43+
printf '%s\n' "$MANIFEST"
44+
echo "Dry-run complete; version stamp not advanced"
45+
exit 0
46+
fi
47+
48+
printf '%s\n' "$MANIFEST" | kubectl apply --filename -
49+
echo "Applied configmap $NAMESPACE/cozystack-version with deletion-protection label"

‎packages/core/platform/images/migrations/run-migrations.sh‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -23,19 +23,26 @@ if [ "$CURRENT_VERSION" -ge "$TARGET_VERSION" ]; then
2323
exit 0
2424
fi
2525

26-
# Run migrations sequentially from current version to target version
26+
# Run migrations sequentially from current version to target version.
27+
# Every version below TARGET_VERSION must have a corresponding migration file
28+
# baked into the image — the release build rebuilds the migrations image from
29+
# the same source tree that carries the files and restamps the digest in the
30+
# same commit, so image and targetVersion are always consistent at release
31+
# time. A missing file therefore indicates a packaging mistake (digest
32+
# advanced without the file landing) and must fail loudly rather than stall
33+
# the cluster at a stale version.
2734
for i in $(seq $CURRENT_VERSION $((TARGET_VERSION - 1))); do
28-
if [ -f "/migrations/$i" ]; then
29-
echo "Running migration $i"
30-
chmod +x /migrations/$i
31-
/migrations/$i || {
32-
echo "Migration $i failed"
33-
exit 1
34-
}
35-
echo "Migration $i completed successfully"
36-
else
37-
echo "Migration $i not found, skipping"
35+
if [ ! -f "/migrations/$i" ]; then
36+
echo "Migration $i not found in image — refusing to advance past a missing migration" >&2
37+
exit 1
3838
fi
39+
echo "Running migration $i"
40+
chmod +x /migrations/$i
41+
/migrations/$i || {
42+
echo "Migration $i failed"
43+
exit 1
44+
}
45+
echo "Migration $i completed successfully"
3946
done
4047

4148
echo "All migrations completed successfully"

0 commit comments

Comments
 (0)