You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 454d0eb
Browse filesBrowse the repository at this point in the historyBrowse files
Mirror the new "Point-in-time recovery (PostgreSQL)" section from the monorepo
docs/operations/backup-classes.md into the website copies (next + v1.6): how to
set spec.options.recoveryTime on a RestoreJob, the recoverable window, how to
discover the earliest/latest restorable time from the backup catalog, the
RecoveryTargetUnreachable failure behavior, and idempotency under GitOps.
Follows up cozystack/cozystack#3383 (PITR support, issue #2774), which added
the section to the monorepo docs the website mirrors.
Signed-off-by: Andrey Kolkov <[email protected]>
Copy file name to clipboardExpand all lines: content/en/docs/next/operations/services/backup-classes.md
+57Lines changed: 57 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -201,6 +201,63 @@ spec:
201
201
name: orders-db
202
202
```
203
203
204
+
## Point-in-time recovery (PostgreSQL)
205
+
206
+
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.
207
+
208
+
```yaml
209
+
apiVersion: backups.cozystack.io/v1alpha1
210
+
kind: RestoreJob
211
+
metadata:
212
+
name: orders-db-pitr
213
+
namespace: tenant-acme
214
+
spec:
215
+
backupRef:
216
+
name: orders-db-adhoc # a completed Backup of the source
217
+
targetApplicationRef: # omit for a destructive in-place restore
218
+
apiGroup: apps.cozystack.io
219
+
kind: Postgres
220
+
name: orders-db-copy # a pre-deployed, empty Postgres app
221
+
options:
222
+
recoveryTime: "2026-07-21T09:30:00Z"
223
+
```
224
+
225
+
A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) lives in the monorepo at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.
226
+
227
+
### The recoverable window
228
+
229
+
`recoveryTime` must fall inside the window the archive can reconstruct:
230
+
231
+
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
232
+
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.
233
+
234
+
A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.
235
+
236
+
Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).
237
+
238
+
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.
239
+
240
+
### Discovering the earliest / latest restorable time
241
+
242
+
CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is not reliably populated under the barman-cloud plugin, so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:
243
+
244
+
```bash
245
+
# Completed base backups, oldest first: STOP is the earliest instant that
246
+
# backup alone can restore to; the oldest STOP is the window's lower bound.
# The Cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
252
+
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
253
+
```
254
+
255
+
The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.
256
+
257
+
### Idempotency under GitOps
258
+
259
+
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.
260
+
204
261
## See also
205
262
206
263
- [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) — the tenant guide for database backups (BackupJob, Plan, RestoreJob).
Copy file name to clipboardExpand all lines: content/en/docs/v1.6/operations/services/backup-classes.md
+57Lines changed: 57 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -201,6 +201,63 @@ spec:
201
201
name: orders-db
202
202
```
203
203
204
+
## Point-in-time recovery (PostgreSQL)
205
+
206
+
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). Under the hood the CNPG barman-cloud plugin restores the newest base backup taken at/before that instant and replays archived WAL up to it, so the restored cluster reflects the database exactly as of `recoveryTime`; later writes are absent.
207
+
208
+
```yaml
209
+
apiVersion: backups.cozystack.io/v1alpha1
210
+
kind: RestoreJob
211
+
metadata:
212
+
name: orders-db-pitr
213
+
namespace: tenant-acme
214
+
spec:
215
+
backupRef:
216
+
name: orders-db-adhoc # a completed Backup of the source
217
+
targetApplicationRef: # omit for a destructive in-place restore
218
+
apiGroup: apps.cozystack.io
219
+
kind: Postgres
220
+
name: orders-db-copy # a pre-deployed, empty Postgres app
221
+
options:
222
+
recoveryTime: "2026-07-21T09:30:00Z"
223
+
```
224
+
225
+
A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) lives in the monorepo at [`examples/backups/postgres/`](https://github.com/cozystack/cozystack/tree/main/examples/backups/postgres) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`.
226
+
227
+
### The recoverable window
228
+
229
+
`recoveryTime` must fall inside the window the archive can reconstruct:
230
+
231
+
- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them.
232
+
- **Latest** — the timestamp of the most recent WAL segment shipped to object storage. While the source runs with `backup.enabled: true` it archives continuously, so the latest restorable time trails "now" only by the archive lag (seconds for a healthy cluster). Once the source is deleted, the latest restorable time is frozen at whatever WAL made it to S3 before it went away.
233
+
234
+
A `recoveryTime` **past the latest archived WAL cannot converge**: PostgreSQL replays every available WAL, never reaches the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise widen the window (take a fresh backup, or restore closer to now) or pick a time inside it.
235
+
236
+
Restoring to a **very recent** instant is safe as long as WAL archiving is current: the segment covering the target may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is archived (e.g. at/before the most recent completed backup, per the discovery query below).
237
+
238
+
A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at/after the oldest base backup's completion (below) avoids it.
239
+
240
+
### Discovering the earliest / latest restorable time
241
+
242
+
CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is not reliably populated under the barman-cloud plugin, so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`:
243
+
244
+
```bash
245
+
# Completed base backups, oldest first: STOP is the earliest instant that
246
+
# backup alone can restore to; the oldest STOP is the window's lower bound.
# The Cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix.
252
+
kubectl -n <ns> get backup.backups.cozystack.io <name> -o jsonpath='{.spec.driverMetadata}'
253
+
```
254
+
255
+
The upper bound — the latest archived WAL — is whatever the source has shipped so far; with archiving healthy, any time up to a few seconds ago is safe. To confirm the archive is current, check that the source cluster's WAL is flowing to S3 (the `barman-cloud` sidecar logs an upload per segment) before restoring to a near-now target.
256
+
257
+
### Idempotency under GitOps
258
+
259
+
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.
260
+
204
261
## See also
205
262
206
263
- [Application Backup and Recovery]({{% ref "/docs/v1.6/applications/backup-and-recovery" %}}) — the tenant guide for database backups (BackupJob, Plan, RestoreJob).
0 commit comments