v1.14: kernel no longer loads extension modules on demand (drbd_transport_tcp) #14501

Bug Report

What happened? What did you expect?

After upgrading a node from v1.13.6 to v1.14.2, DRBD on that node couldn't connect to any peer. drbd was loaded, but drbd_transport_tcp from the drbd extension was not, and drbdsetup new-peer failed:

Failure: (172) Failed to create transport (drbd_transport_xxx module missing?)

Every DRBD resource on the node stayed in Connecting. On v1.13.6 and v1.12.12 nodes in the same cluster, with the same machine config, drbd_transport_tcp shows up in /proc/modules without being listed anywhere: DRBD loads it through request_module() on the first new-peer.

I expected extension modules the kernel asks for to keep loading on v1.14, or a note in the upgrade notes that they have to be listed explicitly now.

What I saw on the nodes:

# v1.14.2 node
/proc/sys/kernel/modprobe:   (empty)
# v1.13.6 node, same cluster
/proc/sys/kernel/modprobe:   /sbin/modprobe

kernel.modules_disabled is 0 on both. The v1.14.2 kernel config in pkgs has CONFIG_MODPROBE_PATH="" and CONFIG_STATIC_USERMODEHELPER=y, which came in with siderolabs/pkgs#1565 and reached Talos in e317d4b ("fix: drop modprobe path and enforce usermode helper"). The PR description says: "I don't think we have any usecase for it (all stuff which is reachable via the kernel is already =y in the config). If this breaks something, we can revert." That holds for =y code, but not for modules from system extensions or for in-tree =m modules like dm-thin-pool and dm-multipath. The kernel can't load any of those on demand anymore.

Adding drbd_transport_tcp to machine.kernel.modules fixed it without a reboot.

Steps to reproduce

1. A node on v1.13.x with the siderolabs/drbd extension, drbd listed in machine.kernel.modules, and drbd_transport_tcp not listed.
2. Upgrade the node to v1.14.2 (drbd 9.3.4-v1.14.2).
3. Bring up any DRBD resource with a TCP peer. drbdsetup new-peer fails as above.

It failed on every DeviceManager cycle of the LINSTOR satellite until the module was listed.

Logs and diagnostics

I don't have a support bundle from the failure window, since the node was fixed right away. I can collect one from a v1.14.2 node with drbd_transport_tcp removed from the config if that helps.

Environment

• Talos version: v1.14.2 (upgraded from v1.13.6)
• Platform: bare metal
• System extensions: siderolabs/drbd 9.3.4-v1.14.2, siderolabs/zfs 2.4.4-v1.14.2
• Last known working version: v1.13.6