Skip to content

Commit 9d8cdf5

Browse files
ghernadighernadi
committed
Snap,restore: Make sure DRBD_INITIALIZED is handled correctly
As the update JavaDoc explains, the code so far only handled snapshot restore cases that came from backup restore, but not regular "s r rst ...". Since SnapVlmDfns do not properly store VlmDfn Flags, the VlmDfn flag DRBD_INITIALIZED was never set on the restored VlmDfn, which caused the restored to run the "I won the race" logic. The symptoms depend on the situation: If the restored resource is placed as diskless resource on the same node where the original resource won the race, the diskless resource most likely tries to "drbdadm primary --force" itself before trying to check/create FS. That "primary" most likely fails simply because it is too fast and it is attempted before DRBD had a chance to connect to a diskful node. If the restored resource on the previously winning node is a diskful one, the restored "winner" will create new metadata (and run mkfs) - this could lead to split-brain scenarios. This patch fixes this issue two-fold: * Make sure that normal "s r rst .." have the DRBD_INITIALIZED flag set * Always update the winner-property to a current diskful node. The latter point would also fix without having the first point in place since a winning restored diskful resource would find its existing metadata (no create-md), it _would_ become primary briefly only to see that a FS already exists (no mkfs), get secondary again and notify the controller that it has initialized the VlmDfn which also causes the flag to get set.
1 parent a7d2984 commit 9d8cdf5

2 files changed

Lines changed: 29 additions & 8 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1616
- Fixed implementation error ("attempt to replace an active transMgr") when creating or modifying a resource group concurrently with a backup restore/shipment that moves a resource definition into its target resource group
1717
- Fixed BalanceResources counting an unhealthy diskful and possibly deleting therefore a healthy diskful
1818
- Fixed implementation error during DRBD .res file regeneration when the existing on-disk file was empty or truncated, which aborted the regeneration instead of rewriting the file
19+
- Fixed regular snapshot restore not setting DRBD_INITIALIZED on the restored volume definition, which could cause spurious metadata re-creation and split-brain
1920

2021
## [1.34.0-rc.1] - 2026-05-28
2122

‎controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -434,26 +434,39 @@ private Flux<ApiCallRc> restoreResourceInTransaction(
434434
}
435435
}
436436

437-
/*
438-
* When a snapshot was received as a backup, it is marked to re-create its local metadata. Before the
439-
* snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
437+
/**
438+
* <p>Sets the {@value InternalApiConsts#KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON} property to the nodeName
439+
* of the first Snapshot (which has to have a disk) of the given SnapshotDefinition.</p>
440+
*
441+
* <p>If this SnapshotDefinition is a backup, we also unset the {@link VolumeDefinition.Flags#DRBD_INITIALIZED}
442+
* flag. Otherwise we ensure the same flag is set.</p>
443+
*
444+
* <p>Backup path: When a snapshot was received as a backup, it is marked to re-create its local metadata.
445+
* Before the snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were
440446
* cleared. If DRBD_INITIALZED would be set, no peer would even read the property (on purpose).
441447
* Without the property, no peer would think of itself as a winner, thus the DRBD resource would never
442448
* become UpToDate.
443449
* To avoid this, this method sets the InitialUpToDateOn property to let the first resource win the
444-
* race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state.
450+
* race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state. </p>
445451
*
446-
* If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
452+
* <p>If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata.
447453
* In that case we do not have to do anything. If however all snapshots are descendants from backups
448454
* we must pick one and decide the UpToDate race upfront so that the winner can set its local DRBD
449-
* as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.
455+
* as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.</p>
456+
*
457+
* <p>Non-Backup path: When a regular snapshot (i.e. not from a backup) is restored, we can or even should
458+
* assume that the backing snapshot LVs have correct metadata which do not need be be initialized. When
459+
* a SnapshotVolumeDefinition is created it unfortunately does not properly store the VolumeDefinition's flags,
460+
* therefore the restored VolumeDefinition will also not have the DRBD_INITIALIZED flag set, although it
461+
* refers to a device with healthy and UpToDate metadata. To fix this issue we simply set the flag in this case.</p>
450462
*/
451463
private void checkDrbdInitializedState(ResourceDefinition rscDfnRef) throws AccessDeniedException
452464
{
453465
AccessContext accCtx = peerAccCtx.get();
454-
if (DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef) && rscDfnRef.getDiskfulCount(accCtx) > 0)
466+
if (rscDfnRef.getDiskfulCount(accCtx) > 0)
455467
{
456468
String winnerNodeName = rscDfnRef.getDiskfulResources(accCtx).get(0).getNode().getName().value;
469+
boolean uninitializeDrbd = DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef);
457470
Iterator<VolumeDefinition> vlmDfnIt = rscDfnRef.iterateVolumeDfn(accCtx);
458471
while (vlmDfnIt.hasNext())
459472
{
@@ -462,7 +475,14 @@ private void checkDrbdInitializedState(ResourceDefinition rscDfnRef) throws Acce
462475
{
463476
vlmDfn.getProps(accCtx)
464477
.setProp(InternalApiConsts.KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON, winnerNodeName);
465-
vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
478+
if (uninitializeDrbd)
479+
{
480+
vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
481+
}
482+
else
483+
{
484+
vlmDfn.getFlags().enableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED);
485+
}
466486
}
467487
catch (InvalidKeyException | InvalidValueException exc)
468488
{

0 commit comments

Comments
 (0)