Repository navigation
Commit 959a9b1
authored
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (cozystack#3062) (cozystack#3078)
## What this PR does
After deleting a `Kubernetes` app CR / tenant, the secret
`<release>-datastore-config` kept the finalizer
`finalizer.kamaji.clastix.io/datastore-secret`, which was never removed
—
**blocking deletion of the entire namespace** (observed stuck in
`Terminating`
for 115 minutes). Kamaji adds this finalizer to the datastore connection
Secret
and normally strips it as part of TenantControlPlane (TCP) deletion; if
the TCP
is force-deleted or Kamaji does not reconcile in time, the finalizer
hangs and
the Secret can never be garbage-collected, wedging the namespace.
This extends the existing `pre-delete` cleanup hook
(`packages/apps/kubernetes/templates/delete.yaml`):
- **Step 4** now deletes the TenantControlPlane with `--wait=true
--timeout=120s`
so Kamaji's own finalizer-cleanup chain has a chance to run before we
fall
through.
- **Step 4b (backstop)** patches `metadata.finalizers: null` on
`<release>-datastore-config` only if it still exists. By this point the
TCP is
gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC,
so
clearing the finalizer merely lets GC reap it — it cannot strip a
finalizer
while a live TCP still needs the datastore. The Secret holds only
connection
config, not tenant data (which lives in the external etcd datastore), so
no
data is lost.
- **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly
`<release>-datastore-config`, verbs `get`/`patch` only.
The Job is hardened: non-root (uid 65534), read-only root fs, all caps
dropped,
`seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache.
Failures at
both steps are logged (the Step 4b warning explicitly mentions the
namespace may
stay Terminating) but the hook still completes so it cannot itself block
teardown.
This is the most impactful of the create→delete leak sweep findings:
without it,
tenant teardown hangs and the create→delete cycle requires manual
intervention.
Fixes cozystack#3062.
### Release note
```release-note
fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Extended the pre-delete cleanup workflow to more reliably remove
`TenantControlPlane` and handle slow deletions with a bounded wait.
* **Security**
* Hardened the cleanup job by running as non-root with restricted
security settings, dropped capabilities, and a safe writable temp
location.
* **Bug Fixes**
* Added a conditional backstop to remove leftover `finalizers` from the
relevant datastore Secret so namespace termination isn’t blocked.
* **Tests**
* Added a YAML test suite to validate the wait-and-patch behavior and
ensure the patch runs only after the `TenantControlPlane` is gone.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 151 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
24 | 30 | | |
25 | 31 | | |
26 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
27 | 48 | | |
28 | 49 | | |
29 | 50 | | |
| |||
48 | 69 | | |
49 | 70 | | |
50 | 71 | | |
51 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
52 | 80 | | |
53 | | - | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
54 | 115 | | |
55 | 116 | | |
56 | 117 | | |
| |||
64 | 125 | | |
65 | 126 | | |
66 | 127 | | |
67 | | - | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
68 | 134 | | |
69 | 135 | | |
70 | 136 | | |
| |||
130 | 196 | | |
131 | 197 | | |
132 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
133 | 211 | | |
134 | 212 | | |
135 | 213 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
0 commit comments