Skip to content

Commit a32591c

Browse files
fix(linstor): constrain CSI provisioning to the scheduler's chosen node
Without --strict-topology, external-provisioner passes every topology segment as `requisite` and only *prefers* the node the scheduler picked, so when that node is out of space linstor-csi may legally provision the volume somewhere else. For a node-pinned volume -- allowRemoteVolumeAccess false, which is what the cluster-default `local` class uses -- the result is a PV whose node affinity contradicts the pod's other volumes, and the pod is then unschedulable forever: pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms It bites hardest on CDI imports, because an importer pod mounts two volumes: the disk, and a ~21 GiB scratch volume CDI creates only AFTER the pod is scheduled. The scheduler therefore sizes the node against the disk alone, can place two importers where both disks plus both scratch volumes do not fit, and the second scratch is placed elsewhere. The failure shape is what makes it worth fixing rather than documenting: nothing errors, the cluster looks healthy, and the DataVolume sits at ImportScheduled indefinitely. Reproduced and verified by measurement 2026-08-25 on a three-node cluster. Forcing both importers onto one node with 45 GiB free -- enough for two 20 GiB disks but not their two 21.2 GiB scratch volumes -- put the scratch PV on a different node than its disk and hung the import with no error anywhere. With the flag set, the same conditions leave the scratch PVC Pending and the pod reporting the actionable "1 node(s) did not have enough free storage"; freeing space let it bind and the import completed on its own. A silent permanent deadlock becomes a legible, self-correcting wait. Note the scheduler was not working from stale data: CSIStorageCapacity reported srv1 at 4.82 GiB against LINSTOR's 4.60 GiB and ZFS's 4.60G. It had the right numbers and still could not prevent this, because the scratch volume does not exist when the placement decision is made. Verified for node-pinned volumes. For `replicated` the PV is remotely accessible so the constraint is satisfiable wherever the pod lands; that case is reasoned rather than measured, hence the switch. The patch restates the whole argument list because `args` has no patchMergeKey and a strategic merge replaces it, so a piraeus-operator bump could silently drop a newly added upstream flag. A test pins the argument count as a tripwire that forces a re-check. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 467968b commit a32591c

4 files changed

Lines changed: 106 additions & 1 deletion

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -204,4 +204,6 @@ The nodes run as **siblings on the runner's docker daemon**, not nested inside t
204204

205205
What the lane gives up is **DRBD, and with it more than the `replicated` StorageClass**. DRBD's resource registry is global per kernel, so three containers sharing one kernel cannot represent three DRBD nodes — a second `new-resource` for the same name returns success while changing nothing, which is worse than an error because LINSTOR then records a replica it does not have. It applies to kind equally; it is a property of one kernel, not of Talos. `linstor` must therefore run with `drbd.enabled: false`, without which every satellite CrashLoops and no PVC binds at all. Because linstor-csi couples RWX to DRBD, tenant worker disks become RWO and **live migration is unavailable on this lane** — a larger loss than the missing StorageClass and the reason the QEMU lane must not be retired from nightly coverage. Everything else survives: kube-ovn runs the real OVS dataplane, and LINSTOR keeps real ZFS pools and the production `local` StorageClass.
206206

207-
Two sharp edges are worth knowing before running it. A container node reports the **host's** memory as `allocatable`, not its compose `mem_limit`, so three nodes advertise roughly three times the box's RAM and the scheduler will overcommit against it. And with local-only placement the CDI importer's data and scratch PVCs can be provisioned on **different** nodes, which deadlocks scheduling permanently on a cluster that otherwise looks healthy; it is a race rather than a certainty, so it presents as a rare hang. Pools are sized 120 GB per node for the same reason the QEMU lane gives each guest a 200 GB disk: LINSTOR can co-locate both workers, and two worker disks plus two import scratch volumes plus etcd does not fit in 60 GB.
207+
Two sharp edges are worth knowing before running it. A container node reports the **host's** memory as `allocatable`, not its compose `mem_limit`, so three nodes advertise roughly three times the box's RAM and the scheduler will overcommit against it. And pools are sized 120 GB per node for the same reason the QEMU lane gives each guest a 200 GB disk: LINSTOR can co-locate both workers, and two worker disks plus two ~21 GiB CDI import scratch volumes plus etcd does not fit in 60 GB.
208+
209+
That second point used to be much worse than a sizing note. An importer pod mounts two node-pinned volumes -- the disk, and a scratch volume CDI creates only *after* the pod is scheduled -- so the scheduler sizes the node against the disk alone and can place two importers where both disks plus both scratch volumes do not fit. The scratch volume was then provisioned on a **different** node and the pod became unschedulable forever against `pv ... node affinity doesn't match node`, with no error surfaced anywhere and the DataVolume reporting `ImportScheduled` indefinitely. That is fixed by `linstorCSI.strictTopology` (on by default), which constrains provisioning to the node the scheduler actually chose; the failure becomes a legible `1 node(s) did not have enough free storage` that resolves on its own once space frees up. Both the deadlock and the fix were reproduced by measurement on 2026-08-25.

‎packages/system/linstor/templates/cluster.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,30 @@ spec:
8383
containers:
8484
- name: linstor-csi
8585
image: {{ .Values.linstorCSI.image.repository }}:{{ .Values.linstorCSI.image.tag }}
86+
{{- if .Values.linstorCSI.strictTopology }}
87+
# See values.yaml linstorCSI.strictTopology for what this prevents.
88+
#
89+
# The full argument list is restated because `args` has no patchMergeKey,
90+
# so a strategic merge REPLACES it rather than appending. That makes this
91+
# list a drift hazard: re-check it against the operator's defaults on
92+
# every piraeus-operator bump, or a newly added upstream flag is silently
93+
# dropped. The $(VAR) references are resolved from the container's env by
94+
# Kubernetes and are stable across versions.
95+
- name: csi-provisioner
96+
args:
97+
- --v=$(VERBOSE)
98+
- --csi-address=$(ADDRESS)
99+
- --timeout=$(TIMEOUT)
100+
- --leader-election=$(LEADER_ELECTION)
101+
- --leader-election-namespace=$(NAMESPACE)
102+
- --worker-threads=$(WORKER_THREADS)
103+
- --http-endpoint=$(HTTP_ENDPOINT)
104+
- --default-fstype=$(DEFAULT_FS_TYPE)
105+
- --enable-capacity=$(ENABLE_CAPACITY)
106+
- --extra-create-metadata=$(EXTRA_CREATE_METADATA)
107+
- --capacity-ownerref-level=$(CAPACITY_OWNERREF_LEVEL)
108+
- --strict-topology
109+
{{- end }}
86110
csiNode:
87111
podTemplate:
88112
spec:
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
suite: CSI provisioner strict topology
2+
3+
# Without --strict-topology, external-provisioner passes every topology segment
4+
# as `requisite` and only *prefers* the node the scheduler picked, so linstor-csi
5+
# may legally place a node-pinned volume elsewhere when that node is full. The
6+
# pod is then unschedulable forever against a PV whose affinity contradicts its
7+
# other volumes -- a silent, permanent deadlock on a healthy-looking cluster.
8+
#
9+
# Reproduced and fixed by measurement 2026-08-25; see values.yaml for the full
10+
# mechanism and what was measured versus reasoned.
11+
12+
templates:
13+
- templates/cluster.yaml
14+
15+
tests:
16+
- it: the provisioner is constrained to the scheduler's chosen node by default
17+
template: templates/cluster.yaml
18+
asserts:
19+
- contains:
20+
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
21+
content: --strict-topology
22+
23+
# The whole argument list is restated by the patch because `args` has no
24+
# patchMergeKey and a strategic merge replaces it. If a piraeus-operator bump
25+
# adds an upstream flag, this list silently drops it -- so pin the count as a
26+
# tripwire that forces a re-check rather than letting the loss go unnoticed.
27+
- it: the restated argument list is complete
28+
template: templates/cluster.yaml
29+
asserts:
30+
- lengthEqual:
31+
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
32+
count: 12
33+
- contains:
34+
path: spec.csiController.podTemplate.spec.containers[?(@.name=="csi-provisioner")].args
35+
content: --enable-capacity=$(ENABLE_CAPACITY)
36+
37+
# Asserted by container count, not a negated `contains`: a negation against a
38+
# path that no longer resolves passes vacuously.
39+
- it: the override can be switched off
40+
template: templates/cluster.yaml
41+
set:
42+
linstorCSI.strictTopology: false
43+
asserts:
44+
- lengthEqual:
45+
path: spec.csiController.podTemplate.spec.containers
46+
count: 1
47+
- equal:
48+
path: spec.csiController.podTemplate.spec.containers[0].name
49+
value: linstor-csi

‎packages/system/linstor/values.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,36 @@ linstor:
3333
minutes: 30
3434
allowCleanup: true
3535
linstorCSI:
36+
## Constrain provisioning to the node the scheduler actually picked.
37+
##
38+
## Without it, external-provisioner passes EVERY topology segment as
39+
## `requisite` and merely *prefers* the selected node, so when that node is out
40+
## of space linstor-csi may legally provision the volume somewhere else. For a
41+
## node-pinned volume (`allowRemoteVolumeAccess: false`, the `local` class that
42+
## is the cluster default) the result is a PV whose node affinity contradicts
43+
## the pod's other volumes, and the pod is then unschedulable FOREVER:
44+
##
45+
## pv "pvc-..." node affinity doesn't match node "srv1": no matching NodeSelectorTerms
46+
##
47+
## It bites hardest on CDI imports, because an importer pod mounts two volumes
48+
## -- the disk and a ~21 GiB scratch volume that CDI only creates AFTER the pod
49+
## is scheduled. The scheduler therefore sizes the node against the disk alone,
50+
## can place two importers on a node that cannot hold both disks plus both
51+
## scratch volumes, and the second scratch is then placed elsewhere.
52+
##
53+
## Reproduced and fixed by measurement 2026-08-25 on a three-node cluster: with
54+
## the flag absent the scratch PV landed on a different node than its disk and
55+
## the import hung indefinitely at ImportScheduled with no error anywhere; with
56+
## it set the scratch PVC stays Pending and the pod reports the actionable
57+
## `1 node(s) did not have enough free storage`, then binds and completes on
58+
## its own once space frees up. A silent permanent deadlock becomes a legible,
59+
## self-correcting wait.
60+
##
61+
## Verified for node-pinned (`local`) volumes, which is what the cluster
62+
## default uses. For `replicated` volumes the PV is remotely accessible, so the
63+
## constraint is satisfiable wherever the pod lands; that case is reasoned
64+
## rather than measured, hence the switch.
65+
strictTopology: true
3666
image:
3767
repository: ghcr.io/cozystack/cozystack/linstor-csi
3868
tag: v1.6.0@sha256:ae854423de62e94fb89b59d92c20da8939196e643e83b2d60d9f6915d5b0e55c

0 commit comments

Comments
 (0)