Repository navigation
[Bug]: Reconciliation dead-lock during switchover with barman plugin enabled #10981
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
No response
Version
1.29 (latest patch)
What version of Kubernetes are you using?
1.34
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
Helm
What happened?
The operator seem to get stuck with cluster switchover operation, when Barman Cloud Plugin is enabled on existing cluster.
Reproduce steps:
Step 1. Create new cluster with 2 instances without barman cloud plugin
When cluster is created, the pods are created without sidecar container, as barman cloud plugin is not enabled.
NAME READY STATUS RESTARTS AGE INSTANCEROLE
barman-test-1 1/1 Running 0 2m41s primary
barman-test-2 1/1 Running 0 109s replica
Step 2. Enable barman cloud plugin
When barman cloud is enabled, the replica pod is re-deployed first (as expected):
NAME READY STATUS RESTARTS AGE INSTANCEROLE
barman-test-1 1/1 Running 0 6m16s primary
barman-test-2 1/2 Running 0 85s replica
To upgrade primary pod, operator issues switchover, and here the problem starts:
NAME READY STATUS RESTARTS AGE INSTANCEROLE
barman-test-1 1/1 Running 0 6m46s unhealthy
barman-test-2 2/2 Running 0 115s replica
As soon as barman cloud plugin is enabled the instance-manager complains that it cannot load plugins:
while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io
This is expected, as barman cloud plugin requires as sidecar container, which can only be created when pod is re-deployed.
To do that as soon as replica is up and running, CNPG performs switchover, but this in effects triggers archiving command on primary instance, which cannot be executed as plugin is not yet enabled:
{
"level": "error",
"ts": "2026-06-22T06:52:45.784562329Z",
"logger": "wal-archive",
"msg": "failed to run wal-archive command",
"logging_pod": "barman-test-1",
"error": "wal archive plugin is not available: barman-cloud.cloudnative-pg.io",
"stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/[email protected]/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.3/x64/src/runtime/proc.go:290"
}This in effect blocks the switchover operation, with operator continuously logs:
There is a switchover or a failover in progress, waiting for the operation to complete
Even worse, the cnpg.io/instanceRole label is updated to unhealthy on primary pod, even though the primary pod is still accepting connections. This blocks primary pod from being selected in barman-test-rw SVC making primary instance unreachable for the application.
If I understand correctly, the switchover operation results in pg_switch_wal() function call by postres, which then triggers archiving to be executed, blocking the process. My expectation was that archiving command is configured only after plugin is successfully installed on primary pod, not immediately.
Workaround
The current workaround is to either scale down the cluster to single instance, or configure primaryUpdateMethod to restart when barman cloud plugin is enabled for the first time. This allows the upgrade to be applied but the problem is that it results in a temporary unavailability of the cluster, which is not ideal, especially for production.
I also tried installing the plugin first with enabled: false , but this doesn't install sidecar container.
Cluster resource
Before barman cloud plugin is enabled:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
labels:
app.kubernetes.io/instance: barman-test
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cloudnative-postgresql-database
app.kubernetes.io/part-of: barman-test
cnpg.io/cluster: barman-test
cnpg.io/reload: "true"
name: barman-test
namespace: barman-test
spec:
affinity:
podAntiAffinityType: preferred
topologyKey: topology.kubernetes.io/zone
bootstrap:
initdb:
database: foo
encoding: UTF8
localeCType: C
localeCollate: C
owner: foo_owner_user
enablePDB: true
enableSuperuserAccess: true
failoverDelay: 0
imageCatalogRef:
apiGroup: postgresql.cnpg.io
kind: ClusterImageCatalog
major: 15
name: postgresql
imagePullPolicy: Always
instances: 2
logLevel: info
maxSyncReplicas: 0
minSyncReplicas: 0
postgresGID: 26
postgresUID: 26
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
full_page_writes: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_parallel_workers: "32"
max_replication_slots: "32"
max_worker_processes: "32"
password_encryption: scram-sha-256
shared_memory_type: mmap
shared_preload_libraries: ""
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 512MB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
pg_hba:
- host all all all scram-sha-256
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: switchover
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
memory: 512Mi
requests:
cpu: 50m
memory: 512Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 5Gi
switchoverDelay: 3600
walStorage:
resizeInUseVolumes: true
size: 3Gi
After barman cloud plugin is enabled:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
labels:
app.kubernetes.io/instance: barman-test
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cloudnative-postgresql-database
app.kubernetes.io/part-of: barman-test
cnpg.io/cluster: barman-test
cnpg.io/reload: "true"
name: barman-test
namespace: barman-test
spec:
affinity:
podAntiAffinityType: preferred
topologyKey: topology.kubernetes.io/zone
backup:
target: prefer-standby
bootstrap:
initdb:
database: foo
encoding: UTF8
localeCType: C
localeCollate: C
owner: foo_owner_user
enablePDB: true
enableSuperuserAccess: true
failoverDelay: 0
imageCatalogRef:
apiGroup: postgresql.cnpg.io
kind: ClusterImageCatalog
major: 15
name: postgresql
imagePullPolicy: Always
instances: 2
logLevel: info
maxSyncReplicas: 0
minSyncReplicas: 0
plugins:
- enabled: true
isWALArchiver: true
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: barman-test-object-store
postgresGID: 26
postgresUID: 26
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
full_page_writes: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_parallel_workers: "32"
max_replication_slots: "32"
max_worker_processes: "32"
password_encryption: scram-sha-256
shared_memory_type: mmap
shared_preload_libraries: ""
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 512MB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
pg_hba:
- host all all all scram-sha-256
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: switchover
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
memory: 512Mi
requests:
cpu: 50m
memory: 512Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 5Gi
switchoverDelay: 3600
walStorage:
resizeInUseVolumes: true
size: 3GiRelevant log output
{
"level": "error",
"ts": "2026-06-22T06:52:45.784562329Z",
"logger": "wal-archive",
"msg": "failed to run wal-archive command",
"logging_pod": "barman-test-1",
"error": "wal archive plugin is not available: barman-cloud.cloudnative-pg.io",
"stacktrace": "github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/[email protected]/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/walarchive.NewCmd.func1\n\tinternal/cmd/manager/walarchive/cmd.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/[email protected]/command.go:1071\nmain.main\n\tcmd/manager/main.go:71\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.3/x64/src/runtime/proc.go:290"
}Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
dosubot commentedon Jun 22, 2026
Hi @jakubmalek! I'm Dosu and I'm helping the CloudNativePG team.
This is a confirmed issue that matches a bug reported in plugin-barman-cloud#901 — "Primary instance won't restart when switching during migration from in-tree to plugin." The exact same deadlock scenario (plugin added to existing cluster → switchover triggered → WAL archiving fails because sidecar not yet present on primary) was reproduced there with identical error messages.
The root cause is that the
archive_commandis always set to/controller/manager wal-archive, which delegates to the plugin at runtime [1]. When the cluster spec is updated with the plugin configuration, the instance manager immediately tries to use the plugin for archiving, but the sidecar container (required for the plugin to function) can only be injected when the pod is recreated. This creates a circular dependency: switchover needs archiving to complete, but archiving needs the sidecar, which needs the pod to be recreated, which is what the switchover is trying to accomplish.This was fixed in PR #8236 [2]. The fix introduces:
MissingWALArchivePlugin = 5) so the instance manager can signal that the configured archiver plugin is unavailable [3]requireWALArchivingPluginOrDelete) in the cluster controller that detects pods terminated due to missing plugins and automatically deletes/recreates them with the sidecar included [3]Since you're on v1.29 (latest patch), you should have this fix. However, given that you're still hitting the deadlock, it's possible the fix didn't fully cover your scenario (fresh plugin enablement vs. the in-tree-to-plugin migration path that #901 originally reported). Your reproduction is slightly different in that you're adding the barman-cloud plugin to a cluster that previously had no WAL archiver configured via plugin at all, whereas #901 was about migrating from the in-tree barman-cloud to the plugin version.
Workaround (confirming yours): Setting
primaryUpdateMethod: restartwhen initially enabling the plugin avoids the deadlock by skipping the switchover entirely. Once all pods have been restarted with the sidecar, you can switch back toswitchover.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
2 remaining items
fix(switchover): recreate primary when WAL-archiver sidecar is missing (
fix(switchover): recreate primary when WAL-archiver sidecar is missing (
fix(switchover): recreate primary when WAL-archiver sidecar is missing (
fix(switchover): recreate primary when WAL-archiver sidecar is missing (
fix(switchover): recreate primary when WAL-archiver sidecar is missing (
fix(switchover): detect missing WAL-archiver native sidecars
fix(switchover): detect missing WAL-archiver native sidecars
fix(switchover): detect missing WAL-archiver native sidecars (#11059)
fix(switchover): detect missing WAL-archiver native sidecars (#11059)
fix(switchover): detect missing WAL-archiver native sidecars (#11059)
fix(switchover): detect missing WAL-archiver native sidecars (#11059)