Repository navigation
postgres backup: BackupJob stays Running for 30 minutes after the cnpg Backup failed, and without a bound while it is pending #5201
Description
Describe the bug
A Postgres BackupJob does not fail when its cnpg.io/Backup has failed. It stays Running with Ready=False and reason CNPGBackupTransientFailure for 30 minutes after startedAt and only then becomes Failed. A BackupJob whose cnpg.io/Backup is in any other phase than completed or failed (pending, started, running, finalizing, walArchivingFailing) has no deadline at all.
Environment
Cozystack v1.7.0 (backupstrategy-controller v1.7.0, CloudNativePG 1.30.0, plugin-barman-cloud v0.13.0, Postgres 18). The controller code is the same on main (4087218).
To reproduce
Failed backup. Copy the strategy cozy-default-cnpg, change the bucket in barmanObjectStore.destinationPath to a bucket that does not exist, bind Postgres to it from a separate BackupClass and run a BackupJob for a Postgres app with backup.useSystemBucket: true that already took one healthy backup.
Pending backup. Annotate the Cluster with cnpg.io/fencedInstances='["*"]' (any state where the target pod is not ready does the same) and create a BackupJob with cozy-default.
Actual behaviour
Failed backup: cnpg.io/Backup goes to failed within a second and stays there. BackupJob is Running / Ready=False / CNPGBackupTransientFailure for the next 30 minutes, then Failed at startedAt + 30m02s with the message rpc error: code = Unknown desc = exit status 1. Same in three runs. The reason says transient and the message does not say what went wrong, the real cause (AccessDenied from the bucket, or Connect timeout on endpoint URL) is only in the log of the plugin-barman-cloud sidecar. CloudNativePG treats a failed Backup as terminal and in my runs it never left failed, so the wait does not help.
Pending backup: cnpg.io/Backup stays pending ("Backup target pod not ready"), BackupJob stays Running with no condition and no message. It was still Running after 43 minutes and finished a few seconds after I removed the annotation. The default: branch of reconcileCNPG in internal/backupcontroller/cnpgstrategy_controller.go requeues without looking at cnpgDefaultBackupDeadline (the cap is applied only in the failed branch, line 455 and after).
Expected behaviour
BackupJob moves to Failed as soon as cnpg.io/Backup is failed, with the failure text. A Backup that never completes is bounded by the same deadline whatever its phase, with a condition that says what it waits for (for example the not-ready target pod).
Related, same family: #4599, where a comment describes the CNPG variant (20 minutes in CNPGBackupTransientFailure).
Activity
dosubot commentedon Nov 4, 2026
One detail that reinforces this: the 30-minute wait does not initiate a retry.
ensureCNPGBackupreturns the existing Backup unchanged, and the failed-phase branch only updates the condition and polls. Recovery during that interval therefore depends entirely on CNPG changing that same Backup’s status. [1] [2]The error-reporting part is separate from the deadline:
cnpgBackupPhasereads onlystatus.error, and the terminal failure already forwards that string. Failing immediately would surface the RPC error sooner, but would not recover the sidecar-onlyAccessDeniedor timeout detail you observed. That needs an additional diagnostic source, not just a change to the phase handling. [3] [4] [5]To reply, just mention @dosu.
Docs are dead. Just use Dosu.
fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211
fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211
scooby87 commentedon Nov 12, 2026
Fixed by #5211: a BackupJob now fails as soon as its cnpg Backup fails. The pending case, a cnpg Backup that never leaves pending and a Plan that does not wait for its previous run, is tracked in #5209.