[Bug]: Race condition in parallel WAL restore causes "invalid checkpoint record" errors when maxParallel > 1 #10092

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

1.28 (latest patch)

What version of Kubernetes are you using?

1.35

What is your Kubernetes environment?

Cloud: Google GKE

How did you install the operator?

Helm

What happened?

Description

When restoring a PostgreSQL cluster from backup with maxParallel set to a value greater than 1, a race condition can cause PostgreSQL to read partially-written WAL files from the spool directory,
resulting in recovery failures with errors like:

"message":"invalid checkpoint record"
"message":"could not locate required checkpoint record at 54/B8011F98"

Please note that I used Claude to help track down this issue. I'm not familiar with this codebase at all, but the race condition it describes seems feasible.

Environment

• Plugin version: 0.5.0 (likely affects all versions)
• barman-cloud library version: 0.3.1+
• maxParallel setting: 8 (reproduces more reliably with higher values)

Root Cause Analysis

The parallel WAL restore mechanism has a race condition between concurrent download goroutines and the spool read operations.

The Flow

1. PostgreSQL requests WAL file N
2. gatherWALFilesToRestore() creates a list of N, N+1, N+2, ... up to maxParallel files
3. RestoreList() spawns concurrent goroutines to download all files simultaneously (https://github.com/cloudnative-pg/barman-cloud/blob/main/pkg/restorer/restorer.go#L159-L211)
4. File N goes to pg_wal/, files N+1 onwards go to the spool directory
5. When PostgreSQL requests file N+1, RestoreFromSpool() → MoveOut() copies it from spool

The Problem

1. Non-atomic writes: barman-cloud-wal-restore writes directly to the final destination path. There's no atomic write pattern (write to .tmp then rename).
   (https://github.com/cloudnative-pg/barman-cloud/blob/main/pkg/restorer/restorer.go#L237-L242)
2. No file completeness validation: spool.Contains() only checks if a file exists, not if it's fully written or fsynced.
   (https://github.com/cloudnative-pg/barman-cloud/blob/main/pkg/spool/spool.go#L54-L58)
3. No synchronization: MoveOut() can copy a file while another goroutine is still writing to it. (https://github.com/cloudnative-pg/barman-cloud/blob/main/pkg/spool/spool.go#L87-L98)

Race Condition Timeline

T0: RestoreList spawns goroutines for WAL files A, B, C (maxParallel=3)
T1: Goroutine-A: barman-cloud-wal-restore writes WAL-A to pg_wal/ (completes)
T2: Goroutine-B: barman-cloud-wal-restore starts writing WAL-B to spool/
T3: PostgreSQL finishes replaying WAL-A, requests WAL-B
T4: RestoreFromSpool sees spool/WAL-B exists (but only partially written!)
T5: MoveOut copies partial file to pg_wal/WAL-B
T6: PostgreSQL reads corrupt WAL → "invalid checkpoint record"

The race window is larger with higher maxParallel values because:

• More concurrent I/O contention slows individual downloads
• PostgreSQL processes the first WAL quickly while subsequent downloads are still in progress
• Higher chance of buffer cache delays before fsync

Steps to Reproduce

1. Create a PostgreSQL cluster with backups configured
2. Take a backup and generate significant WAL activity
3. Create a restore cluster with maxParallel: 16 (or another high value)
4. Observe recovery failure with checkpoint errors

Expected Behavior

Parallel WAL restore should ensure files are completely written before they can be read from the spool.

Actual Behavior

Partially-written WAL files can be moved from the spool and read by PostgreSQL, causing recovery failures.

Workaround

Set maxParallel: 1 (or omit it) to disable parallel prefetching:

wal:
maxParallel: 1

Cluster resource

Relevant section of ObjectStore

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: foo
spec:
  configuration:
    wal:
      maxParallel: 8

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct