[Bug]: VolumeSnapshot backup fails permanently on a transient instance-manager connection during finalize #11068

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

trunk (main)

What version of Kubernetes are you using?

1.35

What is your Kubernetes environment?

Cloud: Google GKE

How did you install the operator?

YAML manifest

What happened?

A volumeSnapshot backup targeting a standby was marked failed even though both VolumeSnapshots (PGDATA and WAL) had already been created by the CSI driver and reported ready to use. It did not fail in the snapshot step, it failed afterwards in the finalize step, where the operator reads the backup-mode status back from the target instance manager over HTTP.

The standby pod was healthy throughout (Running, ready, restartCount: 0, container up well before and after the failure). The instance manager was never down: the TCP connection to its status port simply got no reply for a moment, a transient pod-network disruption. The backup status ended with:

can't execute snapshot backup: while getting status while finalizing: while executing http request: Get "https://10.84.1.156:8000/pg/mode/backup": dial tcp 10.84.1.156:8000: i/o timeout

The volume snapshot reconciler already has a retry classifier for this class of failure: isNetworkErrorRetryable in pkg/reconciler/backup/volumesnapshot/errors.go, added in #7010 (fixing #7000) and documented as covering "errors that occur during communication between the operator and the Kubernetes API server, as well as between the operator and the instance manager". The top-level Reconcile requeues whenever it returns true. The gap is that it recognizes context.DeadlineExceeded, the API-server timeout/conflict/internal errors, and the 503 certificate-propagation rejection, but not a transport-level net.Error such as a dial timeout. A connection that never completes is exactly an operator-to-instance-manager communication failure, yet it escapes the classifier and is flagged as failed.

There is a second cost: because the failed finalize never runs pg_backup_stop, the standby is left in an open backup session, so a subsequent backup against the same instance fails in prepare until the instance is restarted.

This is a sibling of #10718, which reports the same retry-classification gap on the other classifier (isCSIErrorMessageRetriable, for CSI driver errors).

I expect a transient connection failure to the instance manager to be retried, not to discard a backup whose snapshots are already provisioned. It reproduces deterministically by interrupting operator-to-instance connectivity on the status port during the finalize step.

Originally seen on the nightly e2e (Replica Mode > can bootstrap a replica cluster from a backup > using a Volume Snapshot, tests/e2e/replica_mode_cluster_test.go) on GKE.

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: snap-demo
spec:
  instances: 2
  storage:
    size: 1Gi
    storageClass: csi-hostpath-sc
  walStorage:
    size: 1Gi
    storageClass: csi-hostpath-sc
  backup:
    volumeSnapshot:
      className: csi-hostpath-snapclass
      online: true

Relevant log output

info  Waiting for VolumeSnapshot to be provisioned    cluster-replica-src-snapshot
...   Snapshot cluster-replica-src-snapshot-wal is ready to use
...   Snapshot cluster-replica-src-snapshot is ready to use
error while executing snapshot backup | while getting status while finalizing: while executing http request: Get "https://10.84.1.156:8000/pg/mode/backup": dial tcp 10.84.1.156:8000: i/o timeout

Code of Conduct

• I agree to follow this project's Code of Conduct