Skip to content

[Backport release-1.6] fix(cozystack-basics): refuse a direct delete of a tenant namespace - #5012

Open
europrinter (yankawai) wants to merge 1 commit into
cozystack:release-1.6from
yankawai:backport-5011-to-release-1.6
Open

europrinter (yankawai) wants to merge 1 commit into
cozystack:release-1.6from
yankawai:backport-5011-to-release-1.6

Conversation

@yankawai

Copy link
Copy Markdown
Contributor

What this PR does

Backport of #5011 to release-1.6, for #5002, which was seen on a v1.6.5 cluster. The policy, its helm-unittest suite and the tenant-root test are byte for byte the same as in #5011. On release-1.6 the tenant chart puts the same tenant.cozystack.io/tenant-root label on tenant namespaces, cozystack-no-delete-guardrail is the same, and helm-controller runs as the same flux ServiceAccount in cozy-fluxcd, so the policy applies unchanged.

Two parts are adapted. release-1.6 has no external TLS policy test, so the Go test carries its own loader and compiler helpers and builds VersionedAttributes with plain objects, as k8s.io/apiserver v0.35 expects. The e2e case goes before the container-lane test, which hack/linstor-storage-mode_test.bats requires to stay the last install test.

It follows the review of #5011 and should pick up whatever changes there.

Checked locally on this branch.

  • go test ./pkg/registry/apps/application/ -run TestTenantNamespaceDeletePolicy -count=1 -v passes both tests, the decision test with 11 cases. go test ./pkg/registry/... ./pkg/config/... ./pkg/cmd/server/... passes in all 10 packages that have tests.
  • helm unittest in packages/system/cozystack-basics with helm-unittest 1.1.2 in docker passes 10 suites and 56 tests, against 9 and 52 on release-1.6. The new tests on release-1.6 without the template fail, both Go tests and 4 helm tests in 2 suites.
  • The same 12 template mutations as in fix(cozystack-basics): refuse a direct delete of a tenant namespace #5011 each fail a test here.
  • The policy and the namespaces rendered from this branch's charts pass the same 16 checks as in fix(cozystack-basics): refuse a direct delete of a tenant namespace #5011 on kube-apiserver v1.33.0 and v1.35.0 from setup-envtest.
  • hack/linstor-storage-mode_test.bats passes 32 tests and hack/helmrelease-readiness.bats passes 8 under hack/cozytest.sh.

The e2e case was not run, for the same reason as in #5011.

Screenshots

Not a UI change.

Downstream repositories

Same change as #5011, and no entry of the trigger map covers it.

Release note

fix(cozystack-basics): refuse a direct DELETE of a tenant namespace unless it comes from helm-controller or the garbage collector, and point to deleting the Tenant instead, so a tenant namespace deleted by hand no longer gets stuck Terminating on Kamaji finalizers

A tenant namespace is meant to go away with its Tenant. The tenant
chart's pre-delete hook deletes the applications first and the tenant
modules such as etcd after them, and helm-controller deletes the
namespace at the end. A direct delete hands the namespace to the
namespace controller instead. It deletes the pods first and then
everything else together, so the tenant etcd and its etcd-ca-tls Secret
can be gone before the TenantControlPlanes that use them. Kamaji then
cannot reach the datastore to clean up, the TenantControlPlane finalizer
never clears and the namespace stays Terminating.

The policy refuses DELETE of a namespace labelled
tenant.cozystack.io/tenant-root, which the tenant chart puts on every
tenant namespace below tenant-root. tenant-root carries no such label
and stays with the no-delete guardrail. Two callers are let through.
helm-controller and its shards run as the flux ServiceAccount in
cozy-fluxcd and delete the namespace during a Tenant uninstall. The
garbage collector runs in kube-system and removes a child namespace
through its ownerReference once the parent namespace is gone. The
namespace controller only finalizes a namespace and never sends a
DELETE for it, so it is not affected. Everyone else is refused,
system:masters included, as with the no-delete guardrail. The message
names the Tenant to delete and the label to remove when the Tenant is
already gone.

release-1.6 has no external TLS policy test, so the new test carries
its own loader and compiler helpers. The e2e case goes before the
container-lane test, which has to stay the last install test here.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>
(cherry picked from commit 6bb0fda)
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 275b35ce-d964-4698-932a-5cc7367a520e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Oct 21, 2026
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Nov 13, 2026
…5011)

## What this PR does

Fixes #5002. A DELETE of a tenant namespace is refused unless it comes
from the identities that remove it on the normal path, and the denial
says to delete the Tenant instead.

The new ValidatingAdmissionPolicy in cozystack-basics matches DELETE of
core namespaces, and its binding selects namespaces labelled
`tenant.cozystack.io/tenant-root`. The tenant chart puts that label on
every tenant namespace below tenant-root (`tenant.ancestorTenantLabels`,
and migration 49 backfilled older ones). tenant-root itself has no such
label and stays with `cozystack-no-delete-guardrail`. The `tenant-*`
probe namespaces of the e2e suites carry only
`namespace.cozystack.io/host` and are not selected either.

Two groups pass. `system:serviceaccounts:cozy-fluxcd`, because
helm-controller and its shards run as `flux` there, and a Tenant
uninstall is helm-controller deleting the namespace after the pre-delete
hook. `system:serviceaccounts:kube-system`, for the garbage collector
following the ownerReference of a child namespace once its parent is
gone. The namespace controller never sends a DELETE for a namespace, it
updates `finalize`, so it is not affected
(`pkg/controller/namespace/deletion` and
`pkg/registry/core/namespace/storage` at v1.33.12 and v1.34.1).
`system:masters` is not exempt, the same as with the no-delete
guardrail.

This is the denial kube-apiserver 1.35 returns.

```
namespaces "tenant-foo" is forbidden: ValidatingAdmissionPolicy 'cozystack-tenant-namespace-delete-policy' with binding 'cozystack-tenant-namespace-delete-policy' denied request: Namespace tenant-foo belongs to the Tenant foo in tenant-root. Delete the Tenant instead with kubectl delete tenants.apps.cozystack.io foo --namespace tenant-root. Deleting the namespace directly can remove the tenant etcd before the Kubernetes clusters that use it and leave the namespace stuck in Terminating. If the Tenant is already gone, remove the label first with kubectl label namespace tenant-foo tenant.cozystack.io/tenant-root-
```

There are a few limits.

- The bypass is removing the label, like with the guardrail. That also
takes the namespace out of the tenant-root egress policy and the
dashboard tree, which only matters for a namespace that is about to go.
- Migration 49 labelled every `tenant-*` namespace that existed when it
ran, so a non-tenant namespace with such a name is refused too and needs
the same label removal.
- Because the garbage collector is exempt, a parent namespace whose
label was removed and that is deleted with foreground propagation still
cascades into its children directly.
- A DELETE of a namespace that is already Terminating is answered before
admission, so the policy does not see it. It changes nothing either.
- A cluster that made helm-controller impersonate another account
(`--default-service-account`, or `spec.serviceAccountName` on Tenant
HelmReleases) would need that account allowed. Cozystack sets neither.
- It does not change how Kamaji cleans up once the order is broken,
which is the second idea in #5002.

The helper rename in `external_tls_policy_test.go` (`loadPolicy(t,
path)` and `compilePolicy`) is byte for byte the one in #2622, so the
two merge in either order.

Checked locally.

- `go test ./pkg/registry/apps/application/ -run
'TestTenantNamespaceDeletePolicy|TestDatabaseExternalTLSPolicy' -count=1
-v` passes 4 tests, the new `TestTenantNamespaceDeletePolicy_Decisions`
with 11 cases. `go test ./pkg/registry/... ./pkg/config/...
./pkg/cmd/server/...` passes in all 11 packages that have tests.
- `helm unittest` in `packages/system/cozystack-basics` with
helm-unittest 1.1.2 in docker passes 12 suites and 70 tests, against 11
and 66 on main. The new tests on main without the template fail, both Go
tests and 4 helm tests in 2 suites.
- 12 mutations of the template each fail a test. They exempt
`system:masters`, drop either exempt group, select on the no-delete
label, add UPDATE, always name tenant-root as the parent, drop the
capability gate, label tenant-root, switch to Warn, switch failurePolicy
to Ignore, drop the bypass from the message and drop the objectSelector.
- The rendered policy, the guardrail and namespaces rendered from the
tenant chart were applied to kube-apiserver v1.33.0, v1.35.0 and v1.36.2
from setup-envtest, 16 checks on each. Real and dry-run deletes by a
cluster admin, by an OIDC user bound to cluster-admin and by a
cozy-system and a tenant ServiceAccount are refused with 403. The flux
and garbage collector identities get their delete accepted. tenant-root
gets the guardrail's denial, and with the guardrail binding removed a
dry-run delete of it goes through, so this policy does not select it. A
`tenant-*` namespace without the label and a tenant namespace after its
label is removed are accepted for deletion. With this binding removed, a
dry-run delete by the cluster admin goes through. That harness is not
part of the PR.
- `hack/bats-no-negated-assert.bats`, `hack/bats-no-exit-trap.bats`,
`hack/bats-runner-coverage.bats`, `hack/helmrelease-readiness.bats` and
`hack/route-hostname-policy-version-coverage.bats` pass under
`hack/cozytest.sh`.

The new e2e case in `hack/e2e-install-cozystack.bats` creates an empty
Tenant, checks that a server dry-run delete of its namespace is refused
with the message, deletes the Tenant and waits for the namespace to go.
It was not run here, there is no Cozystack cluster at hand. Its kubectl
steps were run against the envtest apiserver 1.35, the selector and
field selector read, the refused dry-run with the exact strings it greps
for, and `kubectl wait --for=delete` on a namespace that is gone. The
Chainsaw suites that delete Tenants, the gateway inheritance ones among
them, go through the uninstall path as well.

The release-1.6 backport is #5012.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The trigger map was walked against the diff. It adds a policy and tests
to cozystack-basics and an e2e case, and no entry covers that. The
website docs do not tell users to delete a tenant namespace directly,
and terraform-provider-cozystack deletes Tenants through the apps API.

### Release note

```release-note
fix(cozystack-basics): refuse a direct DELETE of a tenant namespace unless it comes from helm-controller or the garbage collector, and point to deleting the Tenant instead, so a tenant namespace deleted by hand no longer gets stuck Terminating on Kamaji finalizers
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant