Repository navigation
[Bug]: Backup election logic can preempt an already-started backup #10622
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
Version
1.29 (latest patch)
What version of Kubernetes are you using?
1.35
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
When multiple backups are concurrently in progress for the same cluster, the election logic in CanExecuteBackup() selects which one is allowed to proceed by sorting them alphabetically by name and picking the first one. Because this election runs on every reconciliation loop, it can preempt an already-started backup in favor of another backup whose name happens to sort earlier.
This breaks the assumption that at most a single backup runs at a given time, and can cause serious side effects. In particular, when an online VolumeSnapshot backup is preempted after its prepare phase has already opened a Postgres session and created a temporary replication slot on the primary, neither the session nor the slot are cleaned up. Since temporary replication slots are only dropped when their owning connection closes, the slot remains active indefinitely. This can ultimately lead to WAL disk saturation.
Steps to reproduce:
- Create a default cnpg test environment (./hack/setup-cluster.sh)
- Configure a cluster with both VolumeSnapshot and Plugin backup
- Scale to 0 the
snapshot-controllerdeployment in thekube-systemnamespace - Create a volume-snapshot backup -- its phase will get stuck on
started - Create a plugin backup whose name takes precedence over the volume-snapshot one in
alphabetical order
Cluster resource
Relevant log output
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
dosubot commentedon May 5, 2026
Thanks for the detailed report. I can confirm the issue you've identified in the election logic.
Looking at
CanExecuteBackup()[1], the problem is clear: when two backups are bothIsInProgress()(i.e., inpending,started, orrunningphase) [2], the function iterates the alphabetically-sorted list and only allows the first running backup it encounters to proceed (line 175-176). If a second backup that sorted earlier transitions into an in-progress state, it will preempt the one that was already running — because the check at line 180-181 returnsfalsefor the previously-running backup that now sorts later.This means the function doesn't distinguish between "just became in-progress" vs. "has been running and already acquired resources (like a temporary replication slot)." The side effect you describe — an orphaned temporary replication slot leading to WAL saturation — is a direct consequence, since the temporary slot created during the VolumeSnapshot prepare phase is only cleaned up when the owning Postgres session closes [3], and preempting the backup reconciliation doesn't close that session.
The
cleanupStaleCollections()routine [4] monitors backup connections every minute and will close stale ones if the backup is marked as done or deleted — but a preempted backup that's stuck instartedphase without making progress won't be caught by this cleanup path, since it's still technically "in progress."A potential fix could involve giving priority to backups that are already in
startedorrunningphase over those still inpending, rather than relying solely on alphabetical order. This would preserve the "at most one running backup" invariant while preventing preemption of backups that have already acquired resources.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
test(backups): cover newer-name preemption regression
fix(backups): avoid resource leaks when concurrent backups race (#10747)
fix(backups): avoid resource leaks when concurrent backups race (#10747)
fix(backups): avoid resource leaks when concurrent backups race (#10747)
fix(backups): avoid resource leaks when concurrent backups race (#10747)