Skip to content

postgres backup: a BackupJob whose cnpg Backup stays pending has no bound, and a Plan does not wait for its previous run #5209

Open

Description

@scooby87

Describe the bug

A Postgres BackupJob whose cnpg.io/Backup never leaves pending stays Running with no bound. The fix for #5201 makes such a BackupJob report a Ready=False / WaitingForCNPGBackup condition with the Backup name and phase, and fails it at once when the Backup is failed, but it deliberately leaves non-terminal phases uncapped, because a fixed deadline cannot tell a stuck Backup from a healthy one in CloudNativePG.

In CloudNativePG 1.30.0 a Backup is pending in three different situations:

  1. The target instance is not ready (startBackupManagedByInstance, event BackupPending / FindingPod). This one can last forever, for example while the Cluster is fenced.
  2. Another backup of the same Cluster is in progress (waitIfOtherBackupsRunning with CanExecuteBackup). The Backup is healthy and runs once the other one ends.
  3. The backup is restarted on another instance after a container restart, an inactive pod or a different elected pod (isValidBackupRunning). The Backup goes back to pending and then runs.

Case 2 is made common by the Plan controller: PlanReconciler creates a BackupJob on every cron tick and does not check whether the previous BackupJob of the same Plan has finished, so a backup that takes longer than the schedule interval queues the next one behind it.

Environment

  • Cozystack: main (also v1.7.0)
  • CloudNativePG 1.30.0, plugin-barman-cloud

To Reproduce

  1. Annotate a Postgres Cluster with cnpg.io/fencedInstances='["*"]'.
  2. Create a BackupJob for that app with cozy-default.

Expected behaviour

  • A BackupJob whose Backup is stuck in pending for reasons of case 1 ends Failed after a bound, with the cause in its condition, and the abandoned cnpg.io/Backup is removed so it does not run later without a Cozystack Backup recording it.
  • A Backup that is queued (case 2) or restarted (case 3) is not failed by that bound.
  • A Plan does not start a new run while its previous BackupJob is still running, or has an explicit concurrency policy.
  • The condition carries the reason CloudNativePG gives, which today exists only as an event on the Backup (Backup target pod not ready: ...). The controller's ClusterRole has only create and patch on events, so reading them needs RBAC and should not watch events cluster-wide.

Actual behaviour

The BackupJob stays Running until the Backup leaves pending. Its condition names the Backup and its phase.

Additional context

Split out of #5201. Same family as #4599.

Activity

  1. self-assigned this
    on Nov 4, 2026
  2. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    on Nov 4, 2026
  3. dosubot commented on Nov 4, 2026

    @dosubot

    The concurrency part also intersects #5202: that report reproduced two BackupJobs for the same Postgres app overwriting each other’s destinations. The driver patches an ObjectStore named after the Cluster, then patches the Cluster’s plugin configuration, so those settings are shared rather than isolated per BackupJob. A same-Plan guard would reduce scheduled overlap, but would not cover separate Plans or ad-hoc jobs targeting that app. [1] [2]

    #4559 tracks the broader coordination requirement, including backup-versus-restore overlap. Its September 29 report proposes an atomic per-application claim shared by both controllers, because checking for another job alone leaves a race. That is useful context if the concurrency work here extends beyond a scheduling policy for one Plan. [3]

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about cozystack Share Dosu with your team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugtriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions