Repository navigation
linstor: restoring a PVC from a VolumeSnapshot leaves two of three replicas empty (silent data loss) #4862
Description
Describe the bug
Restoring a PVC from a LINSTOR VolumeSnapshot gives one replica with the data and two empty replicas, and all three are marked UpToDate. A pod that lands on an empty replica mounts a filesystem without the snapshot's files. This is silent data loss on restore: nothing fails up front, and a reader can see an empty volume before anything looks wrong.
Environment
- Cozystack main at
1d813c944, plus the same main with thepiraeus-serverimage rebuilt on Debian trixie (an open dependency roll-up). Both behave the same, so the base image is not the cause. - Test cluster: three Talos nodes, kernel 6.18, DRBD 9.3.2, thick ZFS storage pool
data. - linstor-csi v1.11.2, LINSTOR 1.33.3, drbd-utils 9.34.3 (bookworm image) and 9.35.0 (trixie image).
- StorageClass
replicatedwithplacementCount: "3",storagePool: data,allowRemoteVolumeAccess: "true".
To Reproduce
- Create a 1Gi RWO PVC on
replicated, write a file from a pod, record its sha256, delete the pod. - Take a
VolumeSnapshotwith classlinstor-snapshotsand wait forreadyToUse. - Create a PVC with
dataSourceset to the snapshot and a pod that reads the file. - Run
zfs get origin data/<pv>_00000on each node andlinstor resource list --resources <pv>.
Expected behaviour
All three replicas hold the snapshot's data, as clones or after a full resync from the restored replica, and the pod reads the same sha256.
Actual behaviour
Five restores out of five ended the same way, on both satellite images.
Only one node's zvol is a clone of the snapshot. On the other two nodes origin is -, so these are new empty zvols. The satellite on those nodes logs DRBD meta data created for <pv>/0 and then DRBD skipping initial sync for <pv>/0. They come up UpToDate with a fresh UUID and nothing is resynced.
When the peers connect, the kernel logs uuid_compare()=unrelated-data by rule=history-both and Unrelated data, aborting!, and the connections go StandAlone. After that piraeus-ha-controller logs resource '<pv>' on node '<node>' has failed, evicting every minute and keeps deleting the VolumeAttachment. The node also gets the drbd.linbit.com/lost-quorum:NoSchedule taint, so nothing without a toleration is scheduled there until the volume is deleted.
In one run the reader pod got an empty replica:
sha256sum: can't open '/d/blob': No such file or directory
The snapshot itself is fine. I cloned the ZFS snapshot by hand on every node and read it with drbdmeta ... internal dump-md: the metadata is valid and current-uuid is the same on all three.
Additional context
linstor-csi restores on one node on purpose and then autoplaces the rest. See reconcileSnapshotResources in pkg/client/linstor.go at v1.11.2, which calls RestoreSnapshot with Nodes: []string{selectedNode}, followed by reconcileResourcePlacement.
LINSTOR creates the autoplaced replicas as brand new volumes. In the satellite's DrbdLayer.createMetaData (v1.33.3) a new replica either wins the initial UpToDate race and gets a random current UUID, or asks DrbdLayerUtils.skipInitSync. That one returns true for any ZFS or ZFS_THIN volume unless DrbdOptions/ForceInitialSyncPermanent is set. For a resource definition that already holds data this shortcut is wrong, the new replica has to sync from the restored one.
A possible mitigation, not tested yet: set DrbdOptions/ForceInitialSync=true on the resource group. The controller copies it into ForceInitialSyncPermanent when it creates a resource (CtrlRscCrtApiHelper.copyForceInitialSyncProp), and then skipInitSync returns false. The price is a full initial sync for every new ZFS volume, so it needs measuring before it goes into platform defaults. Restoring on every node that has the snapshot would avoid the empty replicas too.
I'm preparing an upstream report for LINSTOR and will link it here.
Activity
lexfrei commentedon Oct 11, 2026
I went through the LINSTOR v1.33.3 and linstor-csi v1.11.2 sources. The empty replicas come from two separate problems, and the resource-group workaround from the description would not fix either of them for snapshots that already exist.
The first problem is a stale winner of the initial UpToDate race. linstor-csi creates the volume definition with
RestoreVolumeDefinitionSnapshot, which copies onlyDrbdCurrentGi, so the new volume definition has noDRBD_INITIALIZEDflag. The restore then copies the snapshot's volume definition properties into it, andLinstor/Drbd/InitialUpToDateOnstill names the node that won the race for the source volume (seerestoreOnNode). The restored replica already has metadata, so it never takes part in the race. When autoplace puts a replica on that old winner node,DrbdLayer.createMetaDatagives the empty zvol a random current UUID and marks it UpToDate. This matches the run whereInitialUpToDateOnwasNODE0and the restore ran onnode1. Since the empty volume has no filesystem, the winner path also runs mkfs on it.The second problem is the
skipInitSyncshortcut for the other autoplaced replicas, as described in the report.DrbdOptions/ForceInitialSyncon the resource group or the controller does not help here.copyForceInitialSyncPropruns increateResourceonly, not increateResourceFromSnapshot. So the restored resource definition getsForceInitialSyncPermanentwhen autoplace runs, aftercheckDrbdInitializedStatehas already found nothing to do, and the stale winner stays. It would only cover snapshots of volumes created after the property was set, and every new thick ZFS volume would pay a full initial sync for that.LINSTOR already does the right thing for backup shipping.
CtrlBackupCreateApiCallHandlersetsForceInitialSyncPermanenton the resource definition props of the snapshot. On restore this recreates metadata on the restored replica, makes it the race winner and clearsDRBD_INITIALIZED, so every other replica does a full sync from it. The existing mkfs step checks for a filesystem first, so restored data is not touched. My plan is to set the same property on the target resource definition at the start of a restore that comes through the API. Rollback goes through the same function but restores every node from its own snapshot, so it stays as is. The cost lands only on restored volumes: one full sync per extra replica, which a one-node restore can't avoid anyway. New volumes keep skipping the initial sync.The stale winner path does not depend on the storage provider, so LVM thick is affected too, and thin pools hit
skipInitSyncthe same way as ZFS.DrbdOptions/auto-diskfulis enabled in Cozystack and adds diskful replicas later throughcreateResource, so a restored volume can get an empty replica long after the restore.In Cozystack this hits every PVC created from a VolumeSnapshot. That includes Velero backups with
snapshotMoveData: the data mover reads a PVC restored from the snapshot, so the backup itself can capture an empty volume. Tenant Kubernetes PVCs restored through kubevirt-csi go the same way, via a CDI DataVolume with a snapshot source. Golden image clones usecsi-clone, and LINSTOR clones on every source node and copies the volume definition flags, so that path looks fine. I'll check it on a test cluster together with the fix.Next: a piraeus-server patch for the controller, a check on a three-node cluster (checksum from every replica,
drbdadm verifywith zero out-of-sync, a snapshot taken before the change), resync time, and the same change sent upstream.lexfrei commentedon Oct 11, 2026
A correction to my previous comment first. Only the stale-winner problem needs a fix, and LINBIT already fixed it upstream in LINBIT/linstor-server@9d8cdf5, released in v1.34.0. I backported that commit to our v1.33.3 build and ran it on a three-node cluster. The replicas that skip the initial sync are fine in practice. DRBD resyncs them from the restored replica, and the backing zvols end up byte-identical. A carried patch that forces a full initial sync is not needed, so I dropped that plan.
Results with the backport: 5 restores out of 5 came up with every replica connected and UpToDate, the same checksum from a pod on each node, and identical zvols on all three nodes. In 4 of the 5, the source volume's winner was on a different node than the restore, which is the case that broke before. A snapshot taken before the change restores fine too. The resync after a restore costs about the written data: around 40 s for 1Gi and 134 s for 7.5 GiB written on a 10Gi volume.
One measurement trap, in case someone repeats this: compare replicas through their backing devices only with
dd iflag=direct. A buffered read of/dev/zvol/...can return stale page cache. On the same replica I got zeros from a buffered read and the real data withO_DIRECT.Known limitation, which upstream v1.34 shares since that code did not change: the autoplaced replicas rely on the source volume's day0 UUID being in the restored replica's history. If it ever isn't, DRBD goes StandAlone with "unrelated data" instead of syncing. That failure is visible, not silent, and I could not reproduce it.
How to check and repair volumes restored before the fix
The fix does not touch volumes that were already restored. In every broken restore I saw, exactly one replica was wrong: the stale winner. It holds an empty filesystem created by LINSTOR, while the restored replica and the others are intact. Every replica still shows UpToDate. The only sign in
linstor resource listisStandAlonein the connections column, and nothing marks the volume as degraded.An affected volume has all of these:
Linstor/Drbd/InitialUptoDateOninlinstor volume-definition list-properties <pv> 0names a node whose backing zvol has no origin (zfs get -H -o value origin <pool>/<pv>_00000prints-), while another node's zvol has a snapshot as its origin.StandAlonetowards every peer inlinstor resource list --resources <pv>.uuid_compare()=unrelated-dataandUnrelated data, aborting!for the resource.On a healthy restored volume,
InitialUptoDateOnnames the node whose zvol is a clone of the snapshot, and every connection is Ok.To repair it, recreate the bad replica so it resyncs from the others:
drbdadm connect --discard-my-datadoes not help here, because DRBD treats unrelated data differently from a split brain. I tested the delete and recreate on a broken volume: the replica resynced and its zvol matched the other two byte for byte.Before deleting, check where the volume was attached. If an application already ran on the stale replica, it wrote to a different filesystem than the one on the other replicas. Deleting the replica throws those writes away, so decide which copy to keep first.
fix(linstor): mark volumes restored from a snapshot DRBD-initialized
fix(linstor): mark volumes restored from a snapshot DRBD-initialized
fix(linstor): mark volumes restored from a snapshot DRBD-initialized
fix(linstor): mark volumes restored from a snapshot DRBD-initialized
fix(linstor): mark volumes restored from a snapshot DRBD-initialized (c…
[Backport release-1.6] fix(linstor): mark volumes restored from a sna…