Skip to content

Commit 45183b9

Browse files
docs(backups): bound Postgres PITR by the last archived transaction
The recoverable window was described as ending at the newest archived WAL segment, and the safe fallback as "at/before the most recent completed backup". A time-targeted recovery ends just before the first transaction after the target, and a segment switch or a backup is not one, so on a database at rest every instant after its last write fails, including the end of a later backup. Aiming at a backup's stoppedAt also misses it: stoppedAt is truncated to the second and the plugin compares barman's end time to the microsecond. Describe the bound the recovery actually has, how to read it from the archive, the one-second margin on stoppedAt, what pinning the base backup changes, and why archives with several timelines need it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent f1bb5b5 commit 45183b9

1 file changed

Lines changed: 23 additions & 11 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -320,7 +320,9 @@ spec:
320320

321321
## Point-in-time recovery (PostgreSQL)
322322

323-
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.
323+
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). The CNPG barman-cloud plugin restores a base backup and replays archived WAL from it: up to `recoveryTime` when one is set, so the restored cluster reflects the database as of that instant and later writes are absent, and to the end of the archive otherwise.
324+
325+
The base backup a restore starts from is the one the `Backup` refers to: the driver passes its barman backup ID to CNPG as `bootstrap.recovery.recoveryTarget.backupID`. The one exception is a `recoveryTime` earlier than one second after that backup ended, which the backup cannot serve (recovery would have to stop before the backup is consistent); for it the choice is left to the plugin, which takes the newest base backup that ended at or before `recoveryTime`. Left to itself for every restore, the plugin would start from the newest base backup in the archive whenever `recoveryTime` is empty, and from the newest one ending by `recoveryTime` otherwise, whichever `Backup` the `RestoreJob` names and on any timeline — see [Archives with several timelines](#archives-with-several-timelines) for why that matters.
324326

325327
```yaml
326328
apiVersion: backups.cozystack.io/v1alpha1
@@ -345,31 +347,41 @@ A full, scripted example (write a marker, capture the timestamp, restore to it,
345347

346348
`recoveryTime` must fall inside the window the archive can reconstruct:
347349

348-
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
349-
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.
350+
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them. Aim at least one second after a backup's `status.stoppedAt`: `stoppedAt` is truncated to the second, while the plugin compares `recoveryTime` with the end time barman records in `backup.info`, to the microsecond, and keeps only a backup that ended at or before it. A `recoveryTime` equal to `stoppedAt` therefore falls before that backup's actual end: an older backup is used instead, and when there is none the restore fails with `no target backup found`.
351+
- **Latest** — the last transaction, commit or abort, contained in the WAL shipped to object storage. It is not the newest WAL segment: a time-targeted recovery ends just before the first transaction recorded after `recoveryTime`, and a segment switch, a checkpoint or the end of a base backup is not one. A `recoveryTime` at or after the last archived transaction leaves PostgreSQL nothing to stop on. On a busy database the last transaction trails "now" by the archive lag; on a database at rest it is its last write, however recent the WAL archive and however recent the latest base backup. Once the source is deleted, the window is frozen at the last transaction that made it to S3 before it went away.
350352

351-
A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.
353+
A `recoveryTime` **at or after the last archived transaction cannot converge**: PostgreSQL replays every available WAL, never finds a transaction after the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise pick a time before the last archived transaction. The state as of the last write needs no `recoveryTime` at all: omitting it replays the whole archive, and that is the state it recovers.
352354

353-
Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).
355+
Restoring to a **very recent** instant is safe as long as the database keeps writing and WAL archiving is current: the segment covering the target, or the transaction after it, may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed: on a database that has stopped writing, that is every target after its last transaction. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is followed by an archived transaction (see the discovery section below).
354356

355-
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.
357+
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at least one second after the oldest base backup's `stoppedAt` (below) avoids it.
356358

357359
### Discovering the earliest / latest restorable time
358360

359361
CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is unreliable under the barman-cloud plugin — it was observed empty on every plugin-backed cluster on the dev7 test cluster (CNPG 1.28.1) — so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:
360362

361363
```bash
362-
# Completed base backups, oldest first: STOP is the earliest instant that
363-
# backup alone can restore to; the oldest STOP is the window's lower bound.
364+
# Completed base backups, oldest first: STOP plus one second is the earliest
365+
# instant that backup alone can restore to; the oldest one bounds the window.
364366
kubectl -n <ns> get backups.postgresql.cnpg.io \
365367
--sort-by=.status.stoppedAt \
366-
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal
368+
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ID:.status.backupId,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal
367369
368-
# The cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
370+
# The cozystack Backup points at its underlying cnpg.io/Backup, its barman
371+
# backup ID and its S3 prefix.
369372
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
370373
```
371374

372-
The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.
375+
The upper bound is the timestamp of the last `COMMIT` or `ABORT` record in the archived WAL, which nothing in the cluster status reports. `pg_waldump` prints it from the newest segments under `<destinationPath>/<serverName>/wals/`, working back from the newest segment until one holds such a record; on a database at rest that can be several segments back, because every segment switch and every base backup adds one with no transaction in it. Any `recoveryTime` before that timestamp is reachable; for the state as of that last transaction, omit `recoveryTime`.
376+
377+
### Archives with several timelines
378+
379+
A WAL archive can hold several PostgreSQL timelines under one `serverName`. Every failover starts one, and so did every in-place restore made before restored clusters archived to a fresh prefix (`bootstrap.newServerName`): those wrote the new timeline back into the prefix they had recovered from, often without its `.history` file, so the archive ends up with branches that are siblings rather than a single line of descent.
380+
381+
Recovery follows `recovery_target_timeline: latest`, which CNPG leaves as PostgreSQL's default: starting from the base backup's timeline, PostgreSQL looks for the history file of the next timeline, then the next, and follows the last one it finds, replaying only the segments on that line of descent. Without any history file that is the base backup's own timeline alone. Two consequences:
382+
383+
- the last archived transaction that bounds a restore is the last one on the timeline that restore follows, not the newest one in the archive;
384+
- a restore must start from a base backup on the branch it means to bring back. The plugin's own choice ignores timelines — CNPG passes no `targetTLI` — so the newest backup, or the newest one ending by `recoveryTime`, can lie on another branch and restore that branch's data instead. The driver pins the `Backup`'s own base backup with `recoveryTarget.backupID` for that reason; a `recoveryTime` earlier than that backup's end still goes through the plugin's time-based choice, and on such an archive can land on another branch.
373385

374386
### Idempotency under GitOps
375387

0 commit comments

Comments
 (0)