Repository navigation
Kubernetes/KubernetesNodes: model the new kernelModules and schematicID fields (upstream cozystack#3563) #20
Description
Context
cozystack/cozystack PR (linked below) adds a values surface for Talos worker kernel modules, to both apps that render a worker pool:
| App | Field | Type |
|---|---|---|
Kubernetes |
nodeGroups.<name>.kernelModules |
[]KernelModule |
KubernetesNodes |
kernelModules (root level, like gpus) |
[]KernelModule |
KernelModule is a new struct with two attributes:
| Attribute | Type | Required |
|---|---|---|
name |
string |
yes |
parameters |
[]string |
no |
It emits Talos machine.kernel.modules into the worker machine config. A Talos system extension installs a kernel module but does not load it, so a GPU node group whose schematic carries the NVIDIA extension previously came up with no working driver and no way to fix it through values (upstream cozystack/cozystack#3563).
Why this repository is affected
Per the upstream trigger map (docs/agents/contributing.md, "Downstream Repositories"), this provider is hand-written: every Kind carries a hand-maintained Terraform schema with its own expand/flatten pair, with no codegen from values.schema.json. The entry that applies is "Add, remove or rename a field in an app's values.schema.json → the schema, the model, and the expand/flatten pair." Both Kubernetes and KubernetesNodes are affected, and KubernetesNodes takes the field at the root of its schema rather than nested under a node group.
What is needed
Add kernel_modules (or the naming this provider uses for list-of-object attributes) as a nested block or list of objects, with name required and parameters an optional list of strings, to the Kubernetes resource's node-group schema and to the KubernetesNodes resource schema, plus the expand/flatten pair for each.
One point that needs care: the field is deliberately undefaulted
This is the part I would get wrong without knowing this provider, and it is the reason I am filing rather than sending a PR. The upstream field distinguishes three states, and the distinction is load-bearing:
- absent — the chart decides. A node group holding at least one
nvidia.com/*GPU getsnvidia,nvidia_uvm,nvidia_drm,nvidia_modeset; any other group gets nokernelblock at all. - non-empty list — taken verbatim, replacing what the chart would have picked.
- empty list (
[]) — explicit opt-out: emit no modules even on a GPU node group.
So there is no default to restate, and there must not be one: upstream deliberately carries no entry for this key in values.yaml and no default in values.schema.json, because a default of [] would collapse "absent" into "opted out" for every node group and make the automatic NVIDIA set unreachable. The upstream trigger map warns that this provider sends its own defaults rather than letting the API apply them, so a provider that materialises kernel_modules = [] when the user did not set it would silently disable the automatic NVIDIA modules for every GPU node group managed through Terraform — the exact production failure that motivated the upstream change, reintroduced through the provider.
Concretely: the attribute wants to stay Optional without Computed defaulting to an empty collection, and the flatten side needs to keep "unset" and "empty list" distinct rather than normalising a missing value to [].
Not done here
Reporting rather than sending a PR: I do not know this provider's conventions for list-of-object attributes, and the unset-versus-empty distinction above is precisely where a well-meaning Computed default would break users silently. Happy to follow up with a PR if a maintainer confirms the intended shape.
Filed as the downstream follow-up required by the upstream PR checklist.
Activity
mattia-eleuteri commentedon Aug 6, 2026
Upstream PR: cozystack/cozystack#3571
mattia-eleuteri commentedon Aug 6, 2026
The upstream PR grew a second node-group-scoped field, so this issue covers two rather than one.
schematicID— a plain optional string on both apps (nodeGroups.<name>.schematicIDonKubernetes,schematicIDat the root onKubernetesNodes), overriding the cluster-widetalos.schematicIDfor a single node group. It applies to the node group's Talos boot disk image and its installer image.Simpler to model than
kernelModules— no nested object, no list — but it carries the same defaulting hazard, and here it is sharper. An empty string means "inherit the cluster-wide value". A provider that materialisesschematic_id = ""when the user did not set it is harmless, since the chart treats empty as unset; but a provider that helpfully substitutes the cluster-widetalos.schematicIDas a concrete per-group default would pin every node group to the value read at apply time, and a later change totalos.schematicIDwould then silently stop propagating to the groups Terraform manages.Why the field exists, since it explains why getting the default wrong is expensive: a Talos schematic is a fixed set of system extensions,
ext-nvidia-persistencedandext-nvidia-cdi-genrequire an NVIDIA card, and Talos refuses to finish booting when an extension service cannot start. A non-GPU node booting an NVIDIA schematic therefore reboots roughly every 70 minutes, indefinitely, while still reportingReady. This field is what lets the NVIDIA schematic be scoped to the node group that has the cards.Also worth knowing for whoever picks this up: changing
schematicIDon a node group replaces its boot image and rolls that group's workers. That is inherent to the change, not a provider concern, but it means the attribute should not be marked as anything that would make Terraform reorder or batch it silently.[-]Kubernetes/KubernetesNodes: model the new kernelModules field (upstream cozystack#3563)[/-][+]Kubernetes/KubernetesNodes: model the new kernelModules and schematicID fields (upstream cozystack#3563)[/+]mattia-eleuteri commentedon Aug 7, 2026
One more constraint for whoever models
kernelModules, raised by Aleksei Sviridkin (@lexfrei) on the upstream PR and worth having here because it is the sharpest part of the three-state problem.The generated Go type is
KernelModules []KernelModulewithomitempty. An empty slice serialises away underomitempty, so that type cannot represent the[]opt-out — it collapses into "unset", which is the state that means "let the chart pick the NVIDIA modules". A typed Go client that builds the object would therefore silently re-enable the automatic set for an operator who explicitly opted out.The in-cluster path is unaffected: the
KubernetesCR spec travels as raw JSON and never passes through those structs, so applying YAML behaves exactly as documented. The exposure is specific to code that constructs the object through the generated types, which is what a Terraform provider does.So the flatten side needs to distinguish three states deliberately rather than relying on the Go zero values:
kernelModules: []If the provider's serialisation goes through the upstream
api/apps/v1alpha1types rather than building the values map directly, the empty-list case will need something other thanomitemptyon that field, or an explicit marshalling path that keeps it.Upstream reference for the three states and why they exist: cozystack/cozystack#3571
lexfrei commentedon Aug 7, 2026
Confirming the shape so a PR has something fixed to build against.
kernel_modulesis aListNestedAttributeon the node group, withnamea required string andparametersan optional list of strings;schematic_idis a plain optional string alongside it. Both areOptionaland nothing else: noComputed, no default, no plan modifier. AComputedattribute has no null state for an operator to write, and null is the state that means "let the chart pick".The
omitemptyhazard does not reach this provider. It builds the spec as a plainmap[string]anyand hands that to the dynamic client, so the generated Go types are never on the wire path. The map is where the three states live: a null attribute leaves the key out, an empty list writeskernelModules: [], a populated list goes through verbatim. The read side has to match, a stored[]must come back as an empty list rather than null, or every plan reports drift on a node group that deliberately opted out. The presence-preserving helpers for both directions are already in the 1.6 branch, ahead of the fields themselves.One asymmetry to know before writing it: on
Kubernetesneither field carries a schema default, so both stay plainOptional, but onKubernetesNodesthe rootschematicIDdefaults to"", so the server materialises the key whether or not the provider sends it, and that one needsOptionalplusComputedwith an empty-string default. TheKubernetesNodesresource itself is landing separately in #24, so by the time the fields exist that half is two attributes on an existing kind.schematic_idwill not force replacement; the rolling behaviour belongs in the attribute description, not in a lifecycle rule.None of this can land until 3571 merges and a tagged
api/apps/v1alpha1release carries the fields; v1.6.1 is the newest andNodeGrouphas neither. A PR against this shape is welcome once that tag exists, and I'm happy to review it.feat(kubernetes): let a node group set the CPU limit of its worker VMs (
mattia-eleuteri commentedon Sep 3, 2026
Scope update: the upstream PR has been rebased and reduced, and this issue narrows with it.
cozystack/cozystack#3571 originally added
kernelModulesandschematicIDto both charts that rendered a worker pool. Upstream has since mergedrefactor(kubernetes): remove worker node pools from the chart, sonodeGroupsno longer exists on theKubernetesapp and the parent-chart half of that PR was dropped.Both fields therefore land on
KubernetesNodesonly, at the root of its spec, alongsidegpusandroles. There is noKubernetes.nodeGroups[*].kernel_modulesand noKubernetes.nodeGroups[*].schematic_idto add — one resource instead of two.Everything else in this issue stands unchanged, including the part that matters most:
kernel_modulescarries no default on purpose, and a provider that materialiseskernel_modules = []when the user did not set it would disable the automatic NVIDIA module set for every GPU pool managed through Terraform.mattia-eleuteri commentedon Sep 29, 2026
Scope update from cozystack/cozystack#3571: the
schematicIDfield has been dropped from that PR, since main now scopes the schematic per pool throughosImage(cozystack/cozystack#4171), so this issue coverskernelModulesonKubernetesNodesonly. The trap described above still applies unchanged: the field has no default, and sendingkernel_modules = []when the user did not set it disables the automatic NVIDIA modules on every GPU pool.feat(kubernetes): per-pool kernel modules for Talos worker nodes (#3571)