Repository navigation
Commit 6c2dba9
authored
fix(linstor): retry drbdadm secondary after mkfs on transient EBUSY (#2538)
## What this PR does
Adds a fifth piraeus-server patch that retries `drbdadm secondary` after
`mkfs` when DRBD reports the device as held open by an external process.
### Problem
On Talos (and any system with active block-device probing — udev,
multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device
causes an external probe to `open()` the device for superblock
inspection. If LINSTOR satellite issues `drbdadm secondary` while the
probe still holds the fd, DRBD returns `(-12) Device is held open by
someone` and aborts initialization with `StorageException: Failed to
become secondary again after creating filesystem`.
The satellite never reports the final `UpToDate` event to the
controller, so the controller view stays in `SyncTarget(N%) /
Inconsistent / DELETING` permanently — even though the DRBD kernel state
on the satellite is `Established/UpToDate`. This blocks cleanup of
orphan PVs in `Released` phase and accumulates with every PVC creation.
Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in
24 hours across 3 nodes for different PVCs. Manual recovery requires
`drbdadm down/up` on the Primary to force the satellite to re-report
state.
### Fix
New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`:
- 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2
s, capped)
- Retry only triggers when stderr contains `Device is held open by
someone`; any other failure fails fast as before
- Existing `secondary()` contract is unchanged — only
`DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the
new wrapper
The retry window covers ~10 s in the worst case, which comfortably
exceeds typical probe durations (probes release the fd within
milliseconds).
### Related upstream
-
[LINBIT/linstor-server#268](LINBIT/linstor-server#268)
— open since 2022, identical symptom, no upstream fix
- [LINBIT/drbd#74](LINBIT/drbd#74) — same
EBUSY pattern with multipathd holding the fd
### Validation
- `git apply *.diff` on a fresh `v1.33.2` checkout applies all five
patches cleanly in alphabetical order
- Java brace/paren/bracket balance verified on the patched file
- `helm template packages/system/linstor` renders without errors
### Release note
```release-note
fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented automatic retry logic for device secondary state
transitions after filesystem creation. System now gracefully handles
scenarios where devices are temporarily held open by other processes.
* **Documentation**
* Added documentation for the new device state transition patch.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 107 additions & 0 deletions
File tree
- packages/system/linstor/images/piraeus-server/patches
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
0 commit comments