Skip to content

Commit 44f9e49

Browse files
kvapsclaude
authored andcommitted
fix(linstor): plunger demotes idle-Primary DRBD resources
DRBD auto-promotes a resource to Primary during the provisioning path (e.g. mkfs.ext4 against a freshly created replica), but auto-demote sometimes fails to fire on close. The resource then stays Primary on a node that has no consumer, and CSI on the legitimate target node fails NodePublishVolume with "failed to set source device readwrite". The only recovery is an explicit drbdadm secondary on the stuck node. Add a per-satellite plunger handler that detects local Primaries where every device has open == false and demotes them. Both the detection and the action are local to the satellite, so no peer coordination is needed. open:no makes drbdadm secondary safe; if a real consumer is about to arrive, auto-promote will redo it. Upstream tracker: piraeusdatastore/piraeus-operator#942 Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]> (cherry picked from commit a6ac048)
1 parent 313a1ac commit 44f9e49

1 file changed

Lines changed: 31 additions & 0 deletions

File tree

‎packages/system/linstor/hack/plunger/plunger-satellite.sh‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,34 @@ drbd_status_json() {
2020
drbdsetup status --json 2>/dev/null || true
2121
}
2222

23+
# Detect DRBD resources stuck in Primary on this node with no consumer.
24+
# Auto-promote can promote a resource to Primary (e.g. during mkfs.ext4
25+
# at provisioning time), but auto-demote sometimes fails to fire on
26+
# close. The resource then stays Primary on a node that has no
27+
# consumer and blocks NodePublishVolume on the actual target node with
28+
# "failed to set source device readwrite". Upstream tracker:
29+
# https://github.com/piraeusdatastore/piraeus-operator/issues/942
30+
#
31+
# `open == false` on every device of the resource guarantees nobody is
32+
# using it, so `drbdadm secondary` is safe — at worst it's a no-op
33+
# that auto-promote will redo when a real consumer arrives.
34+
drbd_fix_idle_primary() {
35+
local json
36+
json="$(drbd_status_json)"
37+
[ -n "$json" ] || return 0
38+
39+
printf '%s' "$json" | jq -r '
40+
.[]?
41+
| select(.role == "Primary")
42+
| select(all(.devices[]?; .open == false))
43+
| .name
44+
' | while IFS= read -r res; do
45+
[ -n "$res" ] || continue
46+
log "Idle Primary detected: res=$res open:no -> drbdadm secondary"
47+
drbdadm secondary "$res" || log "WARN: secondary failed for $res"
48+
done
49+
}
50+
2351
# Detect DRBD resources where resync is stuck:
2452
# - at least one local device is Inconsistent
2553
# - there is an active SyncTarget peer
@@ -154,6 +182,9 @@ while true; do
154182
drbdadm up "${secondary}" || echo "Command failed"
155183
); done
156184

185+
# Detect and demote idle Primary resources stuck after auto-promote
186+
drbd_fix_idle_primary || true
187+
157188
# Detect and fix stalled DRBD resync by switching SyncTarget peer
158189
drbd_fix_stalled_sync || true
159190

0 commit comments

Comments
 (0)