Repository navigation
Commit a7240ba
authored
fix(backups): delete a MariaDB dump from S3 when its Backup is deleted (#5233)
## What this PR does
Deleting a MariaDB `Backup` removed only the Cozystack object: the dump
stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the
namespace. Nothing else prunes such a dump on the platform bucket.
mariadb-operator's `maxRetention` runs only inside the next backup run
of the same prefix, and a `Plan` never deletes a `Backup`.
Deleting a MariaDB `Backup` now deletes its dump, then the operator
`Backup` that wrote it, and only then lets the `Backup` go. Every run of
an application writes under the same prefix, so the only key ever
deleted is one proven to belong to this `Backup`:
- a `Backup` taken with #5227 records its dump in `driverMetadata`
(`k8s.mariadb.com/dump-file`), and that key is deleted unless another
run of the same prefix, finished or not, could have written it. The size
probe that records the name lets only runs finished by then contest it,
so a run that uploaded its dump and pruned this one through
`maxRetention` before completing can have its dump recorded here; the
delete contests the recorded dump again against every run as it stands,
keeps it when any could own it, and releases the `Backup` with a
`Warning`;
- a `Backup` taken before that is matched with the same ownership-proven
attribution #5223 introduced for the size probe and #5227 uses for
legacy restores: one dump in the operator `Backup`'s lifetime on both
clocks (the timestamp in the name and `LastModified`), and in the
lifetime of no other run of the same prefix, finished or not: a run that
has not completed yet may already have uploaded its dump and pruned this
one through `maxRetention`, so its window is treated as still open. The
match is recorded on the `Backup` before the delete, so a retry deletes
the same key by name;
- when no dump is proven, the `Backup` is released with a `Warning`
event and nothing is deleted, the operator `Backup` included, since it
is what any later attribution starts from.
The delete runs inside the controller through the minio client #5223
added, the same library mariadb-operator writes with, so TLS is verified
the same way. Bucket, prefix and dump name come from the operator
`Backup`. When the operator `Backup` writes to the platform bucket,
which it names by the bucket and by the projected `cozy-backups-creds`
Secret, the endpoint and keys are the platform's current ones (read from
the Secret `cozy-backups-creds` is projected from, TLS from the scheme
of the platform endpoint, as the default strategy decides it, still
trusting a CA the strategy pinned), so a `Backup` taken before the
endpoint moved is still deleted. Any other bucket is reached as the
operator `Backup` recorded it.
Other cases:
- a restore from the `Backup` holds the deletion while it runs: a
`RestoreJob` of the `Backup` that has not finished, or an operator
`Restore` of its operator `Backup` that mariadb-operator has not marked
`Complete` (it does so once the Restore's Job has completed or failed).
The second check matters because a `RestoreJob` can fail on its deadline
while its operator `Restore` keeps reading the dump. Both are listed
from the API server rather than the informer cache, so a restore that
started just before the delete is not missed;
- a restore does not start from a `Backup` that is being deleted or is
gone: the MariaDB restore reads the `Backup` from the API server before
creating the operator `Restore` and fails the `RestoreJob` otherwise,
since mariadb-operator restores the nearest other dump once the
requested one is deleted. A failed lookup of an existing operator
`Restore` is retried rather than failing the `RestoreJob`;
- a failed dump delete keeps the `Backup` in `Terminating` and retries
every 30 seconds with a `Warning` naming the
`backups.cozystack.io/skip-artifact-cleanup` annotation, which releases
the `Backup` and deletes nothing; a failed delete of the operator
`Backup` is retried with the controller's backoff;
- a missing or malformed platform credentials Secret falls back to the
credentials the operator `Backup` names, so it never releases a `Backup`
on its own: a delete that then fails holds it as above. A failed read of
that Secret (an API error) is retried instead, because the recorded
endpoint may be the one the platform moved off;
- a recorded dump that is not in the listing is not deleted again: the
operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent`
event names the key and the endpoint that was listed, so a dump left on
storage the platform moved off can still be found;
- in a terminating namespace a failed delete releases the `Backup` with
a `Warning`, so it does not hold the namespace;
- an operator `Backup` that is gone, that stores on a PVC or volume, or
whose Job failed (`Complete=True` with reason `JobFailed`, which wrote
no dump) releases the `Backup` with a `Warning` and nothing is deleted,
as does a cluster without the mariadb-operator CRDs.
The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No
Job, image or flag is added.
Known limitation: when a namespace is deleted with its `Backup`s, the
namespace controller may delete the operator `Backup` first, and the
`Backup` is then released with a `Warning` and its dump left behind,
because the bucket and prefix are read from the operator `Backup`.
Known limitation: an operator `Backup` that never completes, for example
one whose Job pod never starts, contests every later dump its lifetime
covers on that prefix, so those `Backup`s are released with a `Warning`
and their dumps left behind. That leaks storage rather than deleting a
dump another `Backup` may restore from. The contest reads the runs from
the operator `Backup`s that still exist, so a run whose operator
`Backup` was deleted by hand no longer contests a dump. Likewise, an
operator `Restore` that never completes holds its `Backup`'s deletion
until the skip annotation releases it. These and the namespace-teardown
limitation above are tracked in #5291.
Stacked on #5227 and targets its branch until that merges; only the
commits after #5227's head belong here.
Tests: controller tests on the fake client cover each path above,
including the window bound (`takenAt`, not the operator's `Complete`
transition) of the legacy match, ordering (the operator `Backup`
survives a failed dump delete, and a legacy `Backup` retried after a
failed operator `Backup` delete finishes by its recorded name) and the
finalizer at `Reconcile` level. The minio adapter is tested against an
HTTP server for the escaped key and an error response. A helm-unittest
pins the RBAC verbs. An unfinished neighbour run is covered both ways:
one that started inside the `Backup`'s lifetime keeps its candidate
dump, one that started after it does not contest an earlier dump. The
coordination with restores is covered from both sides: a `RestoreJob` or
operator `Restore` that has not finished, including one only the API
server has seen yet, holds the deletion, and a restore from a `Backup`
being deleted or gone is refused without creating an operator `Restore`.
A recorded dump is contested the same way, down to the full sequence of
the size probe recording a dump an unfinished run uploaded and the
`Backup` then being deleted, and a recorded dump already gone is not
contested. Every condition was mutated, each mutation turns a test red,
including the timeout that bounds an unreachable endpoint. Checked on a
test cluster against SeaweedFS with the controller function and its real
minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with
its wildcard certificate a recorded dump and a legacy dump were deleted
with their operator `Backup`s, a neighbouring dump was kept, and a
recorded dump already gone completed; with a dead recorded endpoint the
dump on the platform bucket was deleted through the platform's current
endpoint; on a foreign bucket behind a dead endpoint the `Backup` was
held with everything kept. The contest of a recorded dump and the
coordination with restores were added after that run and are covered by
the controller tests only.
### Screenshots
Not a UI change.
### Downstream repositories
Walked against the diff. It touches no CRD or API type, no app
`values.schema.json`, no `packages/core/platform/values.yaml`, no
`hack/` tooling the trigger map names, and no ApplicationDefinition.
Only `cozystack/website` is affected:
`content/en/docs/next/operations/services/backup-classes.md` describes
deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka
and the MongoDB system bucket but not for MariaDB.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: needed, not opened yet: document MariaDB dump deletion on
`Backup` removal and the skip annotation
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused.
```10 files changed
Lines changed: 2002 additions & 41 deletions
File tree
- internal/backupcontroller
- packages/system/backupstrategy-controller
- templates
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
41 | 48 | | |
42 | 49 | | |
43 | 50 | | |
| |||
63 | 70 | | |
64 | 71 | | |
65 | 72 | | |
66 | | - | |
67 | | - | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
68 | 77 | | |
69 | 78 | | |
70 | 79 | | |
| |||
74 | 83 | | |
75 | 84 | | |
76 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
77 | 97 | | |
78 | 98 | | |
79 | 99 | | |
| |||
94 | 114 | | |
95 | 115 | | |
96 | 116 | | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
97 | 121 | | |
98 | 122 | | |
99 | 123 | | |
| |||
103 | 127 | | |
104 | 128 | | |
105 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
106 | 138 | | |
107 | 139 | | |
108 | 140 | | |
| |||
152 | 184 | | |
153 | 185 | | |
154 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
155 | 191 | | |
156 | 192 | | |
157 | 193 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
43 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
44 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
45 | 58 | | |
46 | 59 | | |
47 | 60 | | |
| |||
126 | 139 | | |
127 | 140 | | |
128 | 141 | | |
129 | | - | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
140 | 151 | | |
141 | 152 | | |
142 | 153 | | |
143 | 154 | | |
144 | | - | |
| 155 | + | |
145 | 156 | | |
146 | 157 | | |
147 | 158 | | |
| |||
154 | 165 | | |
155 | 166 | | |
156 | 167 | | |
157 | | - | |
| 168 | + | |
158 | 169 | | |
159 | 170 | | |
160 | 171 | | |
| |||
291 | 302 | | |
292 | 303 | | |
293 | 304 | | |
| 305 | + | |
294 | 306 | | |
295 | 307 | | |
296 | 308 | | |
297 | 309 | | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
| 179 | + | |
| 180 | + | |
184 | 181 | | |
185 | 182 | | |
186 | 183 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
68 | 80 | | |
69 | 81 | | |
70 | 82 | | |
| |||
125 | 137 | | |
126 | 138 | | |
127 | 139 | | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
128 | 144 | | |
129 | 145 | | |
130 | 146 | | |
| |||
150 | 166 | | |
151 | 167 | | |
152 | 168 | | |
153 | | - | |
154 | | - | |
| 169 | + | |
| 170 | + | |
155 | 171 | | |
156 | 172 | | |
157 | 173 | | |
| |||
183 | 199 | | |
184 | 200 | | |
185 | 201 | | |
186 | | - | |
| 202 | + | |
187 | 203 | | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
194 | 213 | | |
195 | 214 | | |
196 | 215 | | |
| |||
207 | 226 | | |
208 | 227 | | |
209 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
210 | 232 | | |
211 | 233 | | |
212 | 234 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
184 | 185 | | |
185 | 186 | | |
186 | 187 | | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
187 | 299 | | |
188 | 300 | | |
189 | 301 | | |
| |||
0 commit comments