Repository navigation
[Bug]: Plugin backup phase regression from completed back to pending #11010
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
Version
1.30.0-rc1
What version of Kubernetes are you using?
1.36
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
A Backup using spec.method: plugin can have its status.phase silently reverted from completed to pending, leaving the backup stuck in pending forever even though the backup actually succeeded.
Because plugin backups are managed by the instance manager and never transition through a running phase, the only terminal status update arrives asynchronously from the instance manager. That completion write races against the operator's reconcile loop, and the operator overwrites the completed phase back to pending.
The resulting object is diagnostic: status.phase: pending, but it still carries other status fields that are only ever written on success: backupName, beginWal, endWal and so on.
Cluster resource
{
"kind": "Backup",
"apiVersion": "postgresql.cnpg.io/v1",
"metadata": {
"name": "test-backup",
"namespace": "test",
"uid": "f94f599f-a725-44b6-a591-ce6284697a30",
"resourceVersion": "8127",
"generation": 1,
"creationTimestamp": "2026-06-22T13:54:55Z",
},
"spec": {
"cluster": {
"name": "pg-source"
},
"target": "prefer-standby",
"method": "plugin",
"pluginConfiguration": {
"name": "..."
}
},
"status": {
"majorVersion": 18,
"backupId": "backup-20260622135456",
"backupName": "backup-20260622135456",
"phase": "pending",
"startedAt": "2026-06-22T13:54:56Z",
"stoppedAt": "2026-06-22T13:54:58Z",
"reconciliationStartedAt": "2026-06-22T13:54:55Z",
"reconciliationTerminatedAt": "2026-06-22T13:55:02Z",
"beginWal": "000000010000000000000003",
"endWal": "000000010000000000000003",
"beginLSN": "0/3000028",
"endLSN": "0/3000120",
"backupLabelFile": "U1RBUlQgV0FMIExPQ0FUSU9OOiAwLzMwMDAwMjggKGZpbGUgMDAwMDAwMDEwMDAwMDAwMDAwMDAwMDAzKQpDSEVDS1BPSU5UIExPQ0FUSU9OOiAwLzMwMDAwODAKQkFDS1VQIE1FVEhPRDogc3RyZWFtZWQKQkFDS1VQIEZST006IHByaW1hcnkKU1RBUlQgVElNRTogMjAyNi0wNi0yMiAxMzo1NDo1NiBVVEMKTEFCRUw6IGJhY2t1cC0yMDI2MDYyMjEzNTQ1NgpTVEFSVCBUSU1FTElORTogMQo=",
"instanceID": {
"podName": "pg-source-1",
"ContainerID": "containerd://a33b34ad1809533644982906fee9f1b758c86234df50296d12b5096e47a86017",
"sessionID": "e7c82352-94a2-427a-b632-bccdaa2cba44"
},
"snapshotBackupStatus": {},
"method": "plugin",
"online": true
}
},Relevant log output
Instance manager log line:
{"level":"info","ts":"2026-06-22T13:55:02.243287064Z","logger":"events","msg":"Backup completed","logger":"instance-manager","logging_pod":"pg-source-1","type":"Normal","object":{"kind":"Backup","namespace":"test","name":"test-backup","uid":"f94f599f-a725-44b6-a591-ce6284697a30","apiVersion":"postgresql.cnpg.io/v1","resourceVersion":"7579"},"reason":"Completed"}
Operator log line:
{"level":"info","ts":"2026-06-22T13:55:02.286317669Z","msg":"A backup is already in progress or waiting to be started, retrying","controller":"backup","controllerGroup":"postgresql.cnpg.io","controllerKind":"Backup","Backup":{"name":"test-backup","namespace":"test"},"namespace":"test","name":"test-backup","reconcileID":"8b572853-782e-4165-b04e-e2c6b91b006c","targetBackup":"test-backup"}The two timestamps helps debugging the race conditions between the operator reconciler and the instance manager
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
dosubot commentedon Jun 23, 2026
I can confirm this race condition exists in the backup reconciler. The root cause is that several code paths in
pkg/controller/backup_controller.gocallSetAsPending()without first checking if the backup is already in a terminal phase [1].Specifically, the reconciler resets to pending when:
None of these paths guard against overwriting a completed status. Since plugin backups are managed asynchronously by the instance manager [6], the completion write and the operator's reconcile can race — and the operator wins, reverting
completedback topending.The codebase already has an
IsDone()method that returnstruefor terminal phases (completed/failed). A fix would add a guard like:at each of those call sites.
There have been several recent PRs addressing related backup phase issues (orphaned "started" backups [7], concurrent backup election [8]), but none specifically prevent completed backups from being reverted to pending.
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
Utkarsh-sharma47 commentedon Jun 24, 2026
Hi @GabriFedi97,
I'd like to investigate this issue and work on a fix if nobody is currently working on it.
I'll start by tracing the backup reconciliation flow and reproducing the reported race condition before proposing a fix.
Thanks!
fix(backup): don't regress backup phase back to pending (#11056)
fix(backup): don't regress backup phase back to pending (#11056)