Repository navigation
Description
Describe the bug
A Postgres BackupJob whose cnpg.io/Backup never leaves pending stays Running with no bound. The fix for #5201 makes such a BackupJob report a Ready=False / WaitingForCNPGBackup condition with the Backup name and phase, and fails it at once when the Backup is failed, but it deliberately leaves non-terminal phases uncapped, because a fixed deadline cannot tell a stuck Backup from a healthy one in CloudNativePG.
In CloudNativePG 1.30.0 a Backup is pending in three different situations:
- The target instance is not ready (
startBackupManagedByInstance, eventBackupPending/FindingPod). This one can last forever, for example while the Cluster is fenced. - Another backup of the same Cluster is in progress (
waitIfOtherBackupsRunningwithCanExecuteBackup). The Backup is healthy and runs once the other one ends. - The backup is restarted on another instance after a container restart, an inactive pod or a different elected pod (
isValidBackupRunning). The Backup goes back topendingand then runs.
Case 2 is made common by the Plan controller: PlanReconciler creates a BackupJob on every cron tick and does not check whether the previous BackupJob of the same Plan has finished, so a backup that takes longer than the schedule interval queues the next one behind it.
Environment
- Cozystack: main (also v1.7.0)
- CloudNativePG 1.30.0, plugin-barman-cloud
To Reproduce
- Annotate a Postgres Cluster with
cnpg.io/fencedInstances='["*"]'. - Create a BackupJob for that app with
cozy-default.
Expected behaviour
- A BackupJob whose Backup is stuck in
pendingfor reasons of case 1 endsFailedafter a bound, with the cause in its condition, and the abandonedcnpg.io/Backupis removed so it does not run later without a CozystackBackuprecording it. - A Backup that is queued (case 2) or restarted (case 3) is not failed by that bound.
- A Plan does not start a new run while its previous BackupJob is still running, or has an explicit concurrency policy.
- The condition carries the reason CloudNativePG gives, which today exists only as an event on the Backup (
Backup target pod not ready: ...). The controller's ClusterRole has onlycreateandpatchon events, so reading them needs RBAC and should not watch events cluster-wide.
Actual behaviour
The BackupJob stays Running until the Backup leaves pending. Its condition names the Backup and its phase.
Additional context
Activity
dosubot commentedon Nov 4, 2026
The concurrency part also intersects #5202: that report reproduced two BackupJobs for the same Postgres app overwriting each other’s destinations. The driver patches an ObjectStore named after the Cluster, then patches the Cluster’s plugin configuration, so those settings are shared rather than isolated per BackupJob. A same-Plan guard would reduce scheduled overlap, but would not cover separate Plans or ad-hoc jobs targeting that app. [1] [2]
#4559 tracks the broader coordination requirement, including backup-versus-restore overlap. Its September 29 report proposes an atomic per-application claim shared by both controllers, because checking for another job alone leaves a race. That is useful context if the concurrency work here extends beyond a scheduling policy for one Plan. [3]
To reply, just mention @dosu.
Docs are dead. Just use Dosu.
fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211
fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211