Repository navigation
Commit 7212e9b
authored
fix(backups/postgres): purge stale recovery cluster on repeat in-place restore (#3318)
## What this PR does
Fixes a silent data-integrity failure in the CNPG backup driver: a
repeat in-place PostgreSQL `RestoreJob` into a target that was already
restored once reported `Succeeded` but did nothing. The PVC was
untouched (same uid), no `*-full-recovery` pods appeared, and the
database kept its pre-restore contents.
Root cause: the restore patch sets `spec.bootstrap.enabled=true` on the
Postgres app and never resets it, so the chart keeps rendering
`spec.bootstrap.recovery` on the live cnpg.io Cluster. On a second
`RestoreJob` the re-purge guard (`cnpgPurgeNeeded`) saw that leftover
`bootstrap.recovery` and treated it as "an earlier reconcile of this
restore already purged", skipping the destructive Cluster + PVC purge.
That guard was meant only to avoid re-purging a freshly-recovered
Cluster when the post-purge status write races, but it could not tell
that case apart from a Cluster left over by a previous completed
restore.
The fix distinguishes the two by the live Cluster's `creationTimestamp`
relative to the `RestoreJob`'s `StartedAt`:
- created after `StartedAt`: freshly recovered by this restore's own
purge, so it is skipped (unchanged behaviour for the status-write-race
path);
- created at or before `StartedAt`: leftover from an earlier completed
restore, so it is purged and the restore re-bootstraps from the backup.
Design note: an identity-based alternative (record the purged Cluster's
UID on the `TargetPurged` condition and compare UIDs on later
reconciles) was considered and rejected. It is immune to clock skew but
leans on the same status write that the status-write-race path assumes
can fail, so it cannot cover the exact case the skip exists for. The
timestamp comparison needs no extra persisted state, and the
fresh-vs-stale gap (a full purge plus chart re-render cycle) always
dwarfs any plausible control-plane clock skew. The comparison is strict,
so an exact-second tie resolves to purge (the conservative direction).
### Tests
- New `cnpgClusterFreshlyRecovered` unit test: fresh / stale / boundary
/ missing-timestamp cases.
- `TestCNPGPurgeNeeded` extended with the repeat-restore (stale recovery
cluster) case that reproduced the bug.
- `recoveryBootstrapClusterState` test updated to assert the returned
`creationTimestamp`.
- New reconcile-level regression test driving `reconcileCNPGRestore` end
to end: a stale leftover recovery Cluster is purged, a freshly-recovered
Cluster is not re-purged.
Bug report: #3311
### Screenshots
Not applicable (no UI changes).
### Downstream repositories
The diff touches only `internal/backupcontroller` (operator reconcile
logic); no API types, CRDs, chart values, or docs change, so no
downstream repository is affected.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(postgres): a repeat in-place restore no longer silently succeeds without restoring; a recovery cluster left over from an earlier completed restore is now purged so the restore re-bootstraps from the backup
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved CNPG restore handling for repeat in-place restores.
- Stale recovery clusters are now correctly purged instead of being
mistaken for newly recovered clusters.
- Newly recovered clusters are protected from unnecessary destructive
cleanup.
- Restore behavior is more reliable when recovery clusters are left over
from earlier attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 338 additions & 46 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
318 | | - | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
319 | 320 | | |
320 | 321 | | |
321 | 322 | | |
322 | | - | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
323 | 331 | | |
324 | 332 | | |
325 | 333 | | |
326 | | - | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
327 | 373 | | |
328 | 374 | | |
329 | 375 | | |
| |||
614 | 660 | | |
615 | 661 | | |
616 | 662 | | |
617 | | - | |
| 663 | + | |
618 | 664 | | |
619 | 665 | | |
620 | 666 | | |
621 | 667 | | |
622 | | - | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
623 | 675 | | |
624 | 676 | | |
625 | 677 | | |
| |||
1126 | 1178 | | |
1127 | 1179 | | |
1128 | 1180 | | |
1129 | | - | |
1130 | | - | |
1131 | | - | |
1132 | | - | |
1133 | | - | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
1134 | 1190 | | |
1135 | 1191 | | |
1136 | 1192 | | |
| |||
1143 | 1199 | | |
1144 | 1200 | | |
1145 | 1201 | | |
1146 | | - | |
| 1202 | + | |
1147 | 1203 | | |
1148 | 1204 | | |
1149 | 1205 | | |
1150 | | - | |
| 1206 | + | |
1151 | 1207 | | |
1152 | | - | |
| 1208 | + | |
1153 | 1209 | | |
1154 | 1210 | | |
1155 | | - | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
1156 | 1215 | | |
1157 | | - | |
| 1216 | + | |
| 1217 | + | |
1158 | 1218 | | |
1159 | 1219 | | |
1160 | 1220 | | |
| |||
0 commit comments