Repository navigation
Commit bf26ce9
authored
fix(apps): let the cleanup hooks watch the objects they delete (#4135)
## What this PR does
Fixes #4085.
`kubectl delete` waits for the object to disappear unless `--wait=false`
is given, and that wait is a watch on the object. The post-delete hook
Roles of `harbor`, `clickhouse`, `mariadb` and `qdrant` grant `get`,
`list` and `delete` only, so every uninstall logs
```
Failed to watch *unstructured.Unstructured: persistentvolumeclaims "..." is forbidden: User "system:serviceaccount:<ns>:<release>-cleanup" cannot watch resource "persistentvolumeclaims"
```
the wait is skipped, and the hook prints `PVC cleanup complete` while
the claims are still terminating.
Every collection-scoped rule behind a delete in these four hooks now
also carries `watch`: `persistentvolumeclaims` in all four;
`certificates`, `challenges` and the solver `ingresses` in harbor. The
four named Secrets the mariadb hook removes keep `get` and `delete`
only: they carry no finalizer, so the delete never reaches the watch,
and a `resourceNames` rule does not authorize a collection watch anyway.
A real wait needs a bound. kubectl's default is a week, and a claim held
by a pod that will not stop, or a stuck detach, would hold the hook
until helm-controller's 300s uninstall budget ran out and leave the
HelmRelease stuck uninstalling. The PVC deletes now run with
`--wait=true --timeout=120s`, so the expiry is reported by the script
with the claims still terminating named, and each Job carries
`activeDeadlineSeconds: 270` with `terminationGracePeriodSeconds: 10`,
the numbers `packages/apps/tenant/templates/cleanup-job.yaml` documents
against that budget, as the backstop for a kubectl that never returns.
`--wait=false` was not an option: it is exactly the
report-complete-while-terminating behaviour this PR removes.
The chart unit tests are updated and pass: harbor 21/21, clickhouse
24/24, mariadb 111/111, qdrant 104/104. The behavioural suite
`hack/app-cleanup-hooks_test.bats` passes 11/11 against the changed
scripts.
The second observation in #4085, the harbor hook pod occasionally
starting after its ServiceAccount is gone, is a hook-weight question and
is not touched here.
### Downstream repositories
Hook templates and their tests only: no values, schema, README or
defaults change.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(apps): let the harbor, clickhouse, mariadb and qdrant cleanup hooks wait for the deleted objects, with the wait bounded so a stuck claim cannot hang the uninstall
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Enhancements**
- Cleanup operations for ClickHouse, Harbor, MariaDB, and Qdrant now
wait for persistent volume claim deletion to complete, with a 120-second
timeout.
- Cleanup jobs are limited to 270 seconds and allow 10 seconds for
graceful termination.
- Cleanup permissions now support monitoring relevant resources,
including persistent volume claims and, where applicable, certificates,
challenges, and ingresses.
- Named Secret access is limited to retrieval and deletion.
- **Tests**
- Updated chart validation to verify cleanup timing, deletion behavior,
permissions, and restrictions on unrelated resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->8 files changed
Lines changed: 105 additions & 22 deletions
File tree
- packages/apps
- clickhouse
- templates/hooks
- tests
- harbor
- templates/hooks
- tests
- mariadb
- templates/hooks
- tests
- qdrant
- templates/hooks
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| |||
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
54 | | - | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
87 | 89 | | |
88 | 90 | | |
89 | 91 | | |
90 | | - | |
| 92 | + | |
91 | 93 | | |
92 | 94 | | |
93 | 95 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
71 | 90 | | |
72 | 91 | | |
73 | 92 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
104 | 106 | | |
105 | 107 | | |
106 | 108 | | |
107 | | - | |
| 109 | + | |
108 | 110 | | |
109 | 111 | | |
110 | 112 | | |
| |||
201 | 203 | | |
202 | 204 | | |
203 | 205 | | |
204 | | - | |
| 206 | + | |
205 | 207 | | |
206 | 208 | | |
207 | | - | |
| 209 | + | |
208 | 210 | | |
209 | 211 | | |
210 | 212 | | |
211 | 213 | | |
212 | 214 | | |
213 | | - | |
| 215 | + | |
214 | 216 | | |
215 | 217 | | |
216 | 218 | | |
217 | 219 | | |
218 | | - | |
| 220 | + | |
219 | 221 | | |
220 | 222 | | |
221 | 223 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
106 | 124 | | |
107 | 125 | | |
108 | 126 | | |
| |||
211 | 229 | | |
212 | 230 | | |
213 | 231 | | |
214 | | - | |
| 232 | + | |
215 | 233 | | |
216 | 234 | | |
217 | 235 | | |
218 | 236 | | |
219 | 237 | | |
220 | | - | |
| 238 | + | |
221 | 239 | | |
222 | 240 | | |
223 | 241 | | |
| |||
226 | 244 | | |
227 | 245 | | |
228 | 246 | | |
229 | | - | |
| 247 | + | |
230 | 248 | | |
231 | 249 | | |
232 | 250 | | |
233 | 251 | | |
234 | 252 | | |
235 | | - | |
| 253 | + | |
236 | 254 | | |
237 | 255 | | |
238 | 256 | | |
239 | 257 | | |
240 | 258 | | |
241 | 259 | | |
242 | 260 | | |
243 | | - | |
| 261 | + | |
244 | 262 | | |
245 | 263 | | |
246 | 264 | | |
247 | 265 | | |
248 | 266 | | |
249 | 267 | | |
250 | 268 | | |
251 | | - | |
| 269 | + | |
252 | 270 | | |
253 | 271 | | |
254 | 272 | | |
255 | 273 | | |
256 | 274 | | |
257 | | - | |
| 275 | + | |
258 | 276 | | |
259 | 277 | | |
260 | 278 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
| 22 | + | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
| |||
58 | 60 | | |
59 | 61 | | |
60 | 62 | | |
61 | | - | |
| 63 | + | |
62 | 64 | | |
63 | 65 | | |
64 | 66 | | |
| |||
113 | 115 | | |
114 | 116 | | |
115 | 117 | | |
116 | | - | |
| 118 | + | |
117 | 119 | | |
118 | 120 | | |
119 | 121 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
63 | 81 | | |
64 | 82 | | |
65 | 83 | | |
| |||
69 | 87 | | |
70 | 88 | | |
71 | 89 | | |
72 | | - | |
| 90 | + | |
73 | 91 | | |
74 | 92 | | |
75 | 93 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| |||
104 | 106 | | |
105 | 107 | | |
106 | 108 | | |
107 | | - | |
| 109 | + | |
108 | 110 | | |
109 | 111 | | |
110 | 112 | | |
| |||
147 | 149 | | |
148 | 150 | | |
149 | 151 | | |
150 | | - | |
| 152 | + | |
151 | 153 | | |
152 | 154 | | |
153 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
68 | 88 | | |
69 | 89 | | |
70 | 90 | | |
| |||
80 | 100 | | |
81 | 101 | | |
82 | 102 | | |
83 | | - | |
| 103 | + | |
0 commit comments