Repository navigation
talos: a node with LINSTOR on ZFS hangs on reboot while DRBD holds its zvols #4597
Description
A Talos node that runs LINSTOR on ZFS never finishes a reboot, upgrade or reset while DRBD resources are up on its zvols. The node sits in the rebooting stage forever, and only a hard power reset of the machine gets it back. I hit it twice on a three-node Talos 1.13.6 arm64 test cluster: on the reboot after a talosctl upgrade, and on a talosctl reset.
The hang is in the ZFS extension service. On shutdown ext-zfs-service runs zpool export, and the export never returns. From /proc on the stuck node:
zpool (in the ext-zfs-service PID namespace), thread state D:
taskq_wait [spl]
spa_export_common [zfs]
zfs_ioc_pool_export [zfs]
zfsdev_ioctl
zfs-service (PID 1 of that namespace), state Z, remaining thread:
do_wait
kernel_wait4
zap_pid_ns_processes
do_exit
The export waits for ZFS work that cannot finish, because the zvols are still open. Six of the seven zd* devices had a DRBD device as their holder (/sys/block/zd112/holders/drbd1000 and so on). By that point Talos has already stopped the kubelet and the CRI, so linstor-satellite is gone and nothing brings those DRBD resources down. With zpool stuck in D state, the container's PID 1 cannot exit, and Talos waits for the service without a timeout, so the node never reaches the reboot.
A control run confirms the cause. On the same node, with the same pool, I stopped linstor-satellite with SIGSTOP, ran drbdsetup down all until no zvol had a holder, and then ran the same talosctl reset. This time ext-zfs-service finished 0.3 seconds after SIGTERM, and the reset went on to wipe EPHEMERAL and reboot.
A drain does not help. It unmounts the volumes and demotes DRBD to Secondary, but the replicas stay configured on the zvols.
I think the fix belongs in the node reboot path we document and script. DRBD on the node has to go down after the drain and before the reboot. That can be done with drbdadm down all from the satellite, or by evacuating the node in LINSTOR. Piraeus ships a drbd-shutdown-guard for this, but it is a systemd unit, so it does not run on Talos.
Upstream Talos has a related gap: it waits for services to stop with no deadline. siderolabs/talos#14405 proposed a 30-second bound and was closed without merging, pending details in siderolabs/talos#14404. Until one of these lands, --reboot-mode force avoids the hang, at the cost of an unclean ZFS export and a DRBD resync.
Activity
lexfrei commentedon Sep 29, 2026
The extension already has a switch for this. The shutdown export came in with siderolabs/extensions#1028 and was bounded by #1104 (in the zfs extension since Talos 1.13.4), but the bound does not help here.
zpool export -aruns underexec.CommandContextwith an 8s timeout, so it gets SIGKILL, yet a thread blocked in the export ioctl cannot die. The zombiezpoolkeeps the container's PID namespace alive, which is thezap_pid_ns_processeswait in the stack above.ZFS_EXPORT_TIMEOUT=0in the extension's environment skips the export entirely, andzpool import -falpicks the pools up again on the next boot. The export exists for LUKS under ZFS, where the pool has to let go of the device beforecryptsetup close. We put raw disks under the pool and DRBD on top of it, so we gain nothing from it. Setting it in our Talos machine config templates is probably the whole fix on our side. I have not tested it yet.Upstream prior report: siderolabs/extensions#1085.
lexfrei commentedon Sep 29, 2026
Reported upstream as siderolabs/extensions#1270, with the stacks and the control run.
lexfrei commentedon Sep 30, 2026
ZFS_EXPORT_TIMEOUT=0fixes it. I tested on one node of the same cluster with this document added to its machine config:With 6 zvols held by DRBD and nothing brought down by hand,
talosctl rebootfinished in about 90 seconds. The pool imported on boot, and all DRBD resources on the node came back UpToDate.One catch for existing clusters: applying the document does not restart the running
ext-zfs-service, so the process keeps its old environment. The first reboot after the change still runs the old shutdown and can wedge. On the test node I brought DRBD down by hand for that one reboot, and the setting took effect from the next boot on. New installs that ship the document in their config are not affected.The fix on our side is to add the document to the Talos config templates and describe the one-time reboot for clusters that are already running.
docs(talm): document the ZFS export-on-shutdown knob (#721)
myasnikovdaniil commentedon Nov 6, 2026
cozystack/talm#251 makes the cozystack preset skip ZFS pool export on shutdown by default (Talos 1.13.4+), released in talm v0.36.0. cozystack/website#721 documents first reboot DRBD preparation and the layouts that need export enabled.