Skip to content

Commit f7c04b7

Browse files
feat(kubernetes): expose kernel modules for Talos worker pools
A Talos system extension installs a kernel module but does not load it; loading is machine.kernel.modules' job. The worker machine config had no such block and no values key could add one, so a GPU pool on Talos workers was unusable as shipped: the extension supplied the NVIDIA module, nothing loaded it, and the failure was silent end to end. The VM held the PCI device, the node advertised no GPU, and no component logged an error. The field is three-state, and the states are distinguishable because it carries no default: absent lets the chart choose (a pool holding at least one nvidia.com/* GPU gets nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset, in that order, since Talos loads the list in sequence and the last three depend on the first; any other pool gets nothing), a non-empty list replaces that choice outright, and an explicit [] opts out even on a GPU pool. A default of [] would collapse absent into opted-out for every pool and make the automatic set unreachable. The automatic nvidia entry carries NVreg_NvLinkDisable=1. Individual GPUs are passed into worker VMs without their NVSwitches, and on non-Talos workers the gpu-operator driver container applies that parameter; on Talos that container has to be off, so nothing else would. Names and parameters reach the reconcile Job's unquoted heredoc, so they are validated at render and the emitted list is rebuilt from those two fields alone: the schema does not prune unknown keys, and any other key passed through toYaml would reach the shell unchecked. A pool that resolves to no modules renders the machine config it rendered before, byte for byte, so its content-hash Job name does not rotate. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 0a2054e commit f7c04b7

10 files changed

Lines changed: 511 additions & 29 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 27 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎hack/talos-reconcile-heredoc_test.bats‎

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@
3030
# Assert the refusal. This covers talos.schematicID and
3131
# talos.version, which also land in the worker DataVolume name and
3232
# the image URL, where an unquoted YAML scalar makes escaping the
33-
# wrong tool. templates/nodegroup.yaml pattern-checks them.
33+
# wrong tool. templates/nodegroup.yaml pattern-checks them. It also
34+
# covers kernelModules, whose names and parameters
35+
# templates/_helpers.tpl checks before rebuilding the list.
3436
#
3537
# A field that is in neither arm is the regression this file exists to catch.
3638
#
@@ -251,3 +253,79 @@ VALS
251253
printf '%s' "$out" | grep -qF -e '- "2001:4860:4860::8888"' || { echo "IPv6 nameserver missing from the emitted config" >&2; rm -rf "$work"; exit 1; }
252254
rm -rf "$work"
253255
}
256+
257+
# kernelModules is in the validated arm: its names and parameters are refused at
258+
# render, and the list is rebuilt from those two fields so no other key reaches
259+
# the heredoc. The refusal and the one character class it deliberately lets
260+
# through are pinned separately, because a guard that grows to reject `;` breaks
261+
# NVIDIA's documented multi-value parameter while every refusal still passes.
262+
@test "kubernetes-nodes refuses a hostile kernelModules name or parameter at render" {
263+
work=$(mktemp -d)
264+
cat > "$work/vals.yaml" <<'VALS'
265+
cluster: myk8s
266+
_cluster:
267+
cluster-domain: cozy.local
268+
version: "v1.35"
269+
minReplicas: 0
270+
maxReplicas: 3
271+
instanceType: ""
272+
diskSize: 20Gi
273+
storageClass: replicated
274+
roles: [ingress-nginx]
275+
resources: {cpu: "2", memory: 4Gi}
276+
VALS
277+
cat > "$work/name.yaml" <<'VALS'
278+
kernelModules:
279+
- name: nvidia$(id)`q`z
280+
VALS
281+
cat > "$work/param.yaml" <<'VALS'
282+
kernelModules:
283+
- name: nvidia
284+
parameters: ["opt=$(id)`q`z"]
285+
VALS
286+
for f in name param; do
287+
if helm template kubernetes-nodes-myk8s-md0 packages/apps/kubernetes-nodes -n tenant-test -f "$work/vals.yaml" -f "$work/$f.yaml" \
288+
--show-only templates/talos-reconcile-job.yaml >/dev/null 2>"$work/err"; then
289+
echo "render accepted a hostile kernelModules $f, so the value reaches the heredoc" >&2
290+
rm -rf "$work"; exit 1
291+
fi
292+
grep -qF "invalid kernelModules $f" "$work/err" \
293+
|| { echo "render failed on kernelModules $f for some other reason:" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
294+
done
295+
rm -rf "$work"
296+
}
297+
298+
@test "kubernetes-nodes worker TalosConfigTemplate heredoc keeps a semicolon kernelModules parameter literal" {
299+
work=$(mktemp -d)
300+
cat > "$work/vals.yaml" <<'VALS'
301+
cluster: myk8s
302+
_cluster:
303+
cluster-domain: cozy.local
304+
version: "v1.35"
305+
minReplicas: 0
306+
maxReplicas: 3
307+
instanceType: ""
308+
diskSize: 20Gi
309+
storageClass: replicated
310+
roles: [ingress-nginx]
311+
resources: {cpu: "2", memory: 4Gi}
312+
kernelModules:
313+
- name: nvidia
314+
parameters: ["NVreg_RegistryDwords=PowerMizerEnable=0x1;PerfLevelSrc=0x2222"]
315+
VALS
316+
helm template kubernetes-nodes-myk8s-md0 packages/apps/kubernetes-nodes -n tenant-test -f "$work/vals.yaml" \
317+
--show-only templates/talos-reconcile-job.yaml \
318+
| yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' \
319+
> "$work/cmd.sh"
320+
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
321+
awk '
322+
/^cat <<EOF \| kubectl apply/ { print "cat <<EOF"; inblock=1; next }
323+
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
324+
inblock { print }
325+
' "$work/cmd.sh" > "$work/heredoc.sh"
326+
grep -q '^cat <<EOF$' "$work/heredoc.sh" || { echo "could not extract the kubernetes-nodes heredoc" >&2; rm -rf "$work"; exit 1; }
327+
out=$(sh "$work/heredoc.sh" 2>"$work/err") || { echo "kubernetes-nodes heredoc shell exited non-zero" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
328+
printf '%s' "$out" | grep -qF -e '- NVreg_RegistryDwords=PowerMizerEnable=0x1;PerfLevelSrc=0x2222' \
329+
|| { echo "kernelModules parameter was not preserved literally" >&2; printf '%s\n' "$out" | grep -A6 'kernel:' >&2; rm -rf "$work"; exit 1; }
330+
rm -rf "$work"
331+
}

0 commit comments

Comments
 (0)