Skip to content

Commit a347039

Browse files
authored
fix(linstor): mark volumes restored from a snapshot DRBD-initialized (cozystack#4866)
## What this PR does Restoring a PVC from a LINSTOR VolumeSnapshot can leave one replica with an empty volume that DRBD marks UpToDate. A pod that lands on that replica sees an empty filesystem. linstor-csi restores the snapshot on one node and autoplaces the other replicas. LINSTOR v1.33.3 copies `Linstor/Drbd/InitialUptoDateOn` from the source volume into the restored volume definition and leaves the definition without `DRBD_INITIALIZED`. When autoplace puts a replica on the node that name points to, that empty new volume wins the initial UpToDate race, gets a random current UUID and a fresh filesystem, and goes StandAlone against the restored replica. LINBIT fixed this in v1.34.0. This PR backports that commit, LINBIT/linstor-server@9d8cdf5, to our v1.33.3 build as a patch. The README entry says to drop the patch with the move to v1.34.0 or later. Tested on a three-node cluster with a thick ZFS pool and `placementCount: 3`. Before the change the reproduction failed every time. With the patch I ran 5 restores, 4 of them with the source volume's winner on a different node than the restore. I also restored a snapshot taken before the change, and a 10Gi volume. Every time all replicas came up connected and UpToDate, a pod on each node read the same checksum, and the backing zvols were byte-identical when read with `O_DIRECT`. The resync after a restore moves about the written data, around 40 s for 1Gi and 134 s for 7.5 GiB on 10Gi. CSI clone PVC-to-PVC was checked too and is not affected. There is no e2e test. Every e2e lane runs without DRBD (`COZY_LINSTOR_DRBD_ENABLED=false`, StorageClass `local`), so a suite for replicated restores would never run in CI. Volumes restored before this fix are not repaired by it. How to find and fix them is in [this comment](cozystack#4862 (comment)). Known limitation, shared by upstream v1.34 since that code did not change: the autoplaced replicas skip the initial sync and rely on the source volume's day0 UUID still being in the restored replica's history. If that ever fails, DRBD goes StandAlone with "unrelated data" instead of syncing, which is visible, not silent. I could not reproduce it. Closes cozystack#4862 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): a PVC restored from a VolumeSnapshot no longer gets a replica with an empty volume marked UpToDate. Volumes restored before this release may still have such a replica; cozystack#4862 (comment) explains how to check and repair them. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Restored snapshots with diskful DRBD resources are now recognized as initialized by default, helping them resume without unnecessary metadata recreation. * When force-initial-sync is enabled, restored volumes continue through the existing metadata-recreation path. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents d060c1d + e5689e2 commit a347039

2 files changed

Lines changed: 73 additions & 0 deletions

File tree

‎packages/system/linstor/images/piraeus-server/patches/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,9 @@ Custom patches for piraeus-server (linstor-server) v1.33.3.
2828

2929
- **native-protoc.diff** — Download the protoc build for the build host's architecture instead of always `linux-x86_64`. The builder stage runs on `$BUILDPLATFORM`, so on a native arm64 builder the x86_64 protoc cannot execute and `./gradlew generateJava` fails with `Syntax error: ")" unexpected`.
3030
- Not proposed upstream yet.
31+
32+
- **snapshot-restore-drbd-initialized.diff** — Mark the volume definitions of a restored snapshot as DRBD-initialized, unless the restored resource definition carries `DrbdOptions/ForceInitialSyncPermanent`, which keeps the existing path that recreates metadata, as for a backup. Without the flag the initial UpToDate race is still open on a restored volume, and `Linstor/Drbd/InitialUptoDateOn`, copied from the source volume through the snapshot, names whichever node won it for the source. When linstor-csi autoplaces a replica on that node, the empty new volume wins the race, gets mkfs and goes StandAlone against the restored replica, so a pod attached there sees an empty filesystem. With the flag set, nothing wins the race, and an autoplaced replica resyncs from the restored one like any replica added to an existing volume.
33+
- Backported from upstream commit: [`9d8cdf511`](https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6), first released in v1.34.0; the `CHANGELOG.md` hunk is left out
34+
- Drop when LINSTOR is upgraded to v1.34.0 or later
35+
- On ZFS and thin pools the autoplaced replicas still skip the initial sync and rely on the source volume's day0 UUID being in the restored replica's history, upstream included; if it is not, DRBD reports unrelated data and goes StandAlone rather than syncing
36+
- Fixes [cozystack/cozystack#4862](https://github.com/cozystack/cozystack/issues/4862)
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
2+
index a639c27cc..4dabc1a83 100644
3+
--- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
4+
+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java
5+
@@ -434,26 +434,39 @@ public class CtrlSnapshotRestoreApiCallHandler
6+
}
7+
}
8+
9+
- /*
10+
- * When a snapshot was received as a backup, it is marked to re-create its local metadata. Before the
11+
- * snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
12+
+ /**
13+
+ * <p>Sets the {@value InternalApiConsts#KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON} property to the nodeName
14+
+ * of the first Snapshot (which has to have a disk) of the given SnapshotDefinition.</p>
15+
+ *
16+
+ * <p>If this SnapshotDefinition is a backup, we also unset the {@link VolumeDefinition.Flags#DRBD_INITIALIZED}
17+
+ * flag. Otherwise we ensure the same flag is set.</p>
18+
+ *
19+
+ * <p>Backup path: When a snapshot was received as a backup, it is marked to re-create its local metadata.
20+
+ * Before the snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
21+
* cleared. If DRBD_INITIALZED would be set, no peer would even read the property (on purpose).
22+
* Without the property, no peer would think of itself as a winner, thus the DRBD resource would never
23+
* become UpToDate.
24+
* To avoid this, this method sets the InitialUpToDateOn property to let the first resource win the
25+
- * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state.
26+
+ * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state. </p>
27+
*
28+
- * If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
29+
+ * <p>If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
30+
* In that case we do not have to do anything. If however all snapshots are descendants from backups
31+
* we must pick one and decide the UpToDate race upfront so that the winner can set its local DRBD
32+
- * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.
33+
+ * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.</p>
34+
+ *
35+
+ * <p>Non-Backup path: When a regular snapshot (i.e. not from a backup) is restored, we can or even should
36+
+ * assume that the backing snapshot LVs have correct metadata which do not need be be initialized. When
37+
+ * a SnapshotVolumeDefinition is created it unfortunately does not properly store the VolumeDefinition's flags,
38+
+ * therefore the restored VolumeDefinition will also not have the DRBD_INITIALIZED flag set, although it
39+
+ * refers to a device with healthy and UpToDate metadata. To fix this issue we simply set the flag in this case.</p>
40+
*/
41+
private void checkDrbdInitializedState(ResourceDefinition rscDfnRef) throws AccessDeniedException
42+
{
43+
AccessContext accCtx = peerAccCtx.get();
44+
- if (DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef) && rscDfnRef.getDiskfulCount(accCtx) > 0)
45+
+ if (rscDfnRef.getDiskfulCount(accCtx) > 0)
46+
{
47+
String winnerNodeName = rscDfnRef.getDiskfulResources(accCtx).get(0).getNode().getName().value;
48+
+ boolean uninitializeDrbd = DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef);
49+
Iterator<VolumeDefinition> vlmDfnIt = rscDfnRef.iterateVolumeDfn(accCtx);
50+
while (vlmDfnIt.hasNext())
51+
{
52+
@@ -462,7 +475,14 @@ public class CtrlSnapshotRestoreApiCallHandler
53+
{
54+
vlmDfn.getProps(accCtx)
55+
.setProp(InternalApiConsts.KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON, winnerNodeName);
56+
- vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
57+
+ if (uninitializeDrbd)
58+
+ {
59+
+ vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
60+
+ }
61+
+ else
62+
+ {
63+
+ vlmDfn.getFlags().enableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
64+
+ }
65+
}
66+
catch (InvalidKeyException | InvalidValueException exc)
67+
{

0 commit comments

Comments
 (0)