Skip to content

Commit 8aeedde

Browse files
authored
feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported, tested capability for the new `backups.cozystack.io` API. Fixes #2774. Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the `recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime` plumbing; #3313 has since merged and this branch is rebased on `main`. This PR adds the reliability + verification layer #2774 asked for. ### Changes - **Precise failure reason for an unreachable target.** The restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the sole authority for "this restore is stuck" — the client waits the full window either way. What changes is the *diagnosis*: once the deadline has elapsed, if `recoveryTime` was set and the recovery Cluster never went healthy, the CNPG driver reads the `full-recovery` pod logs and — when several distinct attempts exited with PostgreSQL's `FATAL: recovery ended before configured recovery target was reached` — fails the RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason `RecoveryTargetUnreachable`) and an actionable message, instead of a bare generic timeout. Timing is unchanged; only the reason/message differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.) Design points: - **We deliberately do NOT trip early off the FATAL.** A valid near-now target emits that same FATAL transiently — repeatedly, for many attempts, on a slow archiver — before it converges, so any earlier trip would wrongly reject a recoverable restore (observed twice in CI). The FATAL is used only to *explain* a failure the deadline has already declared. - **Health wins unconditionally** — the deadline/classification check precedes the healthy check, but a recovery that converges within the window reaches success and is never classified as failed (a live dev7 restore did exactly this). - **Counting distinct FATAL-ing attempts** (not pod failures) tolerates a one-off archive-lag miss on a near-now target and ignores transient recovery-pod crashes (a node blip / brief API-server unreachability never emits that FATAL). - Adds the `pods/log` RBAC grant the log read needs (distinct subresource from `pods`); a `Forbidden` is logged loudly and surfaced in the failure message rather than silently disabling the classification. - **e2e** (`examples/backups/postgres/run-all.sh`, wired into the Chainsaw `postgres-2-backup-roundtrip` suite): - Step 45 — write `before`, capture server timestamp (µs precision), write `after`, restore to that timestamp, assert `before` present and `after` absent. - Step 46 — a `recoveryTime` an hour ahead must fail with reason `RecoveryTargetUnreachable` once the deadline elapses (asserts the *reason*, so it proves the classification fired, not just that a timeout occurred). - Chainsaw budget raised 25m → 40m to cover three sequential restores + the post-deadline classification. - **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a `docs/operations/backup-classes.md` PITR section (recoverable window, discovering earliest/latest restorable time from the backup catalog since `firstRecoverabilityPoint` is unreliable under the plugin, archive-lag tolerance, idempotency under GitOps). ### Verification - `go build` / `go vet` clean; unit tests pass (`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`, `recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level success-path test asserting `Succeeded` + `RecoveryConverged=True`, and — added per review — `TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`, a reconcile-level test driving the deadline branch through all three sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL absent → generic timeout, `Forbidden` log read → generic timeout + missing-`pods/log`-RBAC note. It fails if the call-site reason is reverted to the generic one). - Reviewed via adversarial branch-review; fixes include a critical missing-RBAC dead-on-arrival, a classification false-positive on transient recovery-pod crashes, and a sub-second e2e flake. - **dev7**: backup + WAL archiving + recovery convergence verified live on the #3313 build. The load-bearing assumption of the classification — CNPG **accumulates** failed `full-recovery` pods rather than reusing one — was confirmed (observed 7 simultaneous failed recovery pods). A fully-green live step-46 run was blocked by intermittent COSI bucket-provisioning flakiness on dev7; the negative path is covered by the CI Chainsaw e2e. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only user-facing surface is the new `docs/operations/backup-classes.md` PITR section; the website keeps a hand-mirrored copy per docs version. - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: PITR docs mirror warranted (not yet opened; same pending mirror as #3313's backup-classes changes) - [x] No other downstream repository is affected by this change ```release-note feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added PostgreSQL point-in-time recovery (PITR) support via RFC3339 `recoveryTime`. * Introduced a PITR RestoreJob example, plus helper variables and an unreachable-target scenario. * **Bug Fixes** * Improved restore deadline handling with deterministic, log-based fail-fast classification for unreachable recovery targets, including clearer conditions. * **Documentation** * Added detailed PITR operational documentation (recovery windows, failure behavior, and GitOps/idempotent behavior). * Expanded README and example comments for PITR verification and failure expectations. * **Tests** * Extended unit tests and end-to-end chainsaw workflow to cover PITR success and unreachable failures. * **Chores** * Updated RBAC to allow pod log reads; refined cleanup and run scripts for additional RestoreJob artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 18e459a + ad3f8b2 commit 8aeedde

12 files changed

Lines changed: 1180 additions & 49 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,3 +202,60 @@ spec:
202202
kind: Postgres
203203
name: orders-db
204204
```
205+
206+
## Point-in-time recovery (PostgreSQL)
207+
208+
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.
209+
210+
```yaml
211+
apiVersion: backups.cozystack.io/v1alpha1
212+
kind: RestoreJob
213+
metadata:
214+
name: orders-db-pitr
215+
namespace: tenant-acme
216+
spec:
217+
backupRef:
218+
name: orders-db-adhoc # a completed Backup of the source
219+
targetApplicationRef: # omit for a destructive in-place restore
220+
apiGroup: apps.cozystack.io
221+
kind: Postgres
222+
name: orders-db-copy # a pre-deployed, empty Postgres app
223+
options:
224+
recoveryTime: "2026-07-21T09:30:00Z"
225+
```
226+
227+
A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) is in [`examples/backups/postgres/`](../../examples/backups/postgres/) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.
228+
229+
### The recoverable window
230+
231+
`recoveryTime` must fall inside the window the archive can reconstruct:
232+
233+
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
234+
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.
235+
236+
A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.
237+
238+
Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).
239+
240+
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.
241+
242+
### Discovering the earliest / latest restorable time
243+
244+
CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is unreliable under the barman-cloud plugin — it was observed empty on every plugin-backed cluster on the dev7 test cluster (CNPG 1.28.1) — so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:
245+
246+
```bash
247+
# Completed base backups, oldest first: STOP is the earliest instant that
248+
# backup alone can restore to; the oldest STOP is the window's lower bound.
249+
kubectl -n <ns> get backups.postgresql.cnpg.io \
250+
--sort-by=.status.stoppedAt \
251+
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal
252+
253+
# The cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
254+
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
255+
```
256+
257+
The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.
258+
259+
### Idempotency under GitOps
260+
261+
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.

‎examples/backups/postgres/00-helpers.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,13 @@ export PG_TARGET_CLUSTER="postgres-${PG_TARGET_NAME}"
2424
export STRATEGY_NAME="${STRATEGY_NAME:-cnpg-strategy-default}"
2525
export BACKUPCLASS_NAME="${BACKUPCLASS_NAME:-postgres-cnpg}"
2626
export BACKUPJOB_NAME="${BACKUPJOB_NAME:-pg-src-adhoc}"
27+
# A second ad-hoc BackupJob taken after the PITR marker writes; its completion
28+
# is the deterministic "WAL past the recovery target is archived" gate for the
29+
# point-in-time restore (see run-all.sh step 45).
30+
export BACKUPJOB_POSTMARKER_NAME="${BACKUPJOB_POSTMARKER_NAME:-pg-src-postmarker}"
2731
export RESTOREJOB_TOCOPY_NAME="${RESTOREJOB_TOCOPY_NAME:-pg-src-to-pg-target}"
32+
export RESTOREJOB_PITR_NAME="${RESTOREJOB_PITR_NAME:-pg-src-to-pg-target-pitr}"
33+
export RESTOREJOB_UNREACHABLE_NAME="${RESTOREJOB_UNREACHABLE_NAME:-pg-src-to-pg-target-unreachable}"
2834
export PLAN_NAME="${PLAN_NAME:-pg-src-daily}"
2935
# App user password baked into 05-postgres-src.yaml (REPLACE_WITH_PASSWORD).
3036
export PG_PASSWORD="${PG_PASSWORD:-Xai7Wepo0aeThie8}"

‎examples/backups/postgres/35-restorejob-in-place.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,12 @@
22
# back into pg-src. WARNING: deletes the live cnpg.io Cluster and its PVCs
33
# before re-bootstrapping from S3 - any data not yet streamed to WAL is lost.
44
#
5-
# Optional: spec.options.recoveryTime (RFC3339) for point-in-time recovery.
5+
# Optional: spec.options.recoveryTime (RFC3339) turns this into a point-in-time
6+
# recovery, same as the to-copy variant in 45-restorejob-pitr.yaml (which the
7+
# e2e drives with a real, in-window timestamp). Uncomment and set a timestamp
8+
# inside your recoverable window - see 45-restorejob-pitr.yaml and
9+
# docs/operations/backup-classes.md ("Point-in-time recovery") for how to find
10+
# the earliest/latest restorable time.
611
---
712
apiVersion: backups.cozystack.io/v1alpha1
813
kind: RestoreJob
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Point-in-time recovery (PITR) to a copy.
2+
#
3+
# Restores the source's Barman archive into pg-target as of a specific instant
4+
# via spec.options.recoveryTime (RFC3339). The chart maps it onto CNPG's
5+
# bootstrap.recovery.recoveryTarget.targetTime: PostgreSQL restores the newest
6+
# base backup taken at/before that instant and replays archived WAL up to it,
7+
# so the copy reflects the database exactly as of recoveryTime - later writes
8+
# are absent.
9+
#
10+
# recoveryTime MUST fall inside the recoverable window: at/after the base
11+
# backup's completion and at/before the latest WAL shipped to object storage.
12+
# A target past the latest archived WAL cannot converge - the RestoreJob fails
13+
# with condition Ready=False/RecoveryTargetUnreachable instead of hanging. See
14+
# docs/operations/backup-classes.md ("Point-in-time recovery") for how to
15+
# discover the earliest/latest restorable time.
16+
#
17+
# run-all.sh substitutes REPLACE_WITH_RECOVERY_TIME with a timestamp it captures
18+
# between two marker writes, then asserts the pre-marker row is present and the
19+
# post-marker row is absent. Editing by hand, replace it with an RFC3339
20+
# timestamp inside your window, e.g. "2026-05-01T12:00:00Z".
21+
---
22+
apiVersion: backups.cozystack.io/v1alpha1
23+
kind: RestoreJob
24+
metadata:
25+
name: pg-src-to-pg-target-pitr
26+
spec:
27+
backupRef:
28+
name: pg-src-adhoc
29+
targetApplicationRef:
30+
apiGroup: apps.cozystack.io
31+
kind: Postgres
32+
name: pg-target
33+
options:
34+
recoveryTime: "REPLACE_WITH_RECOVERY_TIME"

‎examples/backups/postgres/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ The numbered manifests are the documented flow; a human can read and apply them
1414
- `25-backupjob-adhoc.yaml` — an ad-hoc `BackupJob`.
1515
- `30-postgres-target.yaml` + `40-restorejob-to-copy.yaml` — restore into a fresh copy, leaving the source running.
1616
- `35-restorejob-in-place.yaml` — the destructive in-place restore variant.
17+
- `45-restorejob-pitr.yaml` — point-in-time recovery: restore into the copy as of a specific `spec.options.recoveryTime` (RFC3339).
1718

1819
## Placeholders and derived Secrets
1920

@@ -30,7 +31,7 @@ NAMESPACE=tenant-root examples/backups/postgres/run-all.sh
3031
NAMESPACE=tenant-root examples/backups/postgres/cleanup.sh
3132
```
3233

33-
`run-all.sh` writes a sentinel row into the source, waits for the `BackupJob` to reach `Succeeded`, restores to a copy, and asserts the sentinel round-tripped through S3 into the restored copy. Set `SKIP_RESTORE=1` to stop after a successful backup.
34+
`run-all.sh` writes a sentinel row into the source, waits for the `BackupJob` to reach `Succeeded`, restores to a copy, and asserts the sentinel round-tripped through S3 into the restored copy. It then runs a **point-in-time recovery**: it writes a `before` marker, captures the server timestamp, writes an `after` marker, and restores the copy to that timestamp (`45-restorejob-pitr.yaml`), asserting the `before` row survived and the `after` row did not. Finally it submits a RestoreJob with a `recoveryTime` an hour in the future and asserts it **fails fast** with `status.phase: Failed` and reason `RecoveryTargetUnreachable` (rather than hanging to the restore deadline). Set `SKIP_RESTORE=1` to stop after a successful backup, or `SKIP_PITR=1` to stop after the latest-point restore.
3435

3536
## Automated e2e
3637

‎examples/backups/postgres/cleanup.sh‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,14 @@ for app in "$PG_SRC_NAME" "$PG_TARGET_NAME"; do
2020
fi
2121
done
2222

23-
log_substep "Deleting RestoreJob..."
24-
kubectl -n "$NAMESPACE" delete restorejob.backups.cozystack.io "$RESTOREJOB_TOCOPY_NAME" --ignore-not-found
23+
log_substep "Deleting RestoreJobs..."
24+
kubectl -n "$NAMESPACE" delete restorejob.backups.cozystack.io \
25+
"$RESTOREJOB_TOCOPY_NAME" "$RESTOREJOB_PITR_NAME" "$RESTOREJOB_UNREACHABLE_NAME" --ignore-not-found
2526

2627
log_substep "Deleting Plan + BackupJob + derived Backup artefacts..."
2728
kubectl -n "$NAMESPACE" delete plan.backups.cozystack.io "$PLAN_NAME" --ignore-not-found
28-
kubectl -n "$NAMESPACE" delete backupjob.backups.cozystack.io "$BACKUPJOB_NAME" --ignore-not-found
29+
kubectl -n "$NAMESPACE" delete backupjob.backups.cozystack.io \
30+
"$BACKUPJOB_NAME" "$BACKUPJOB_POSTMARKER_NAME" --ignore-not-found
2931
# Only the Backups of THIS demo's apps — never `--all`: on a shared cluster the
3032
# namespace can hold Backup artefacts of other applications and flows.
3133
for b in $(kubectl -n "$NAMESPACE" get backups.backups.cozystack.io \

0 commit comments

Comments
 (0)