You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4653a05
Browse filesBrowse the repository at this point in the historyBrowse files
fix(kubernetes,kubernetes-nodes): close three substrate holes found in review
Three separate faults, all of them invisible in a passing test run.
1. proxmox.dnsServers reached the reconcile Job's shell unescaped.
The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so
every byte of its body is subject to parameter expansion and command
substitution at Job runtime. The INVARIANT comment above that block says each
tenant-controlled value must be shell-escaped or pattern-validated; the new
nameserver list was neither. `quote` there is YAML quoting.
Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way
hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed
both and put their output in the worker's resolver list. The Job's pod holds the
parent cluster's Talos token, cluster id, cluster secret and bootstrap token in
its environment, and its Role can read the Talos CA and patch the parent
KamajiControlPlane.
Each entry is now validated as an IPv4 or IPv6 address at render time, in both
charts. Validation rather than escaping because Talos takes addresses here: an
address cannot carry a metacharacter, so the property holds by construction, and
the ordinary mistake of writing a hostname is caught in the same move. IPv6 is
matched on its character set rather than its full grammar — what matters is which
bytes may reach the shell, and the apiserver rejects a malformed address anyway.
2. Nothing stopped a live cluster or pool from being switched between substrates.
`substrate` had no immutability of any kind, and the `## @immutable` marker this
adds is dashboard-enforced only, so both charts now read the live object.
Switching a cluster removes sixteen objects from a default render: the
KubevirtCluster, the cloud-controller-manager, the kubevirt-csi controller and
the in-tenant -csi HelmRelease with all their RBAC. Flux then uninstalls the CSI
driver behind every PVC the tenant has. CAPI does not prevent it — its Cluster
webhook forbids removing spec.infrastructureRef but admits a change of kind.
Switching a pool rolls every worker to the other hypervisor and drops the
machine template the draining MachineSet still references, because the retention
loop looks up only the kind the new substrate uses. A pool whose substrate
differs from its parent's was not rejected either, though the README says the
two must match.
Both guards are inert offline, like assertNoForeignPool and assertParentVersion
beside them: with no apiserver the lookup is empty and a first install renders.
3. The worker WorkloadMonitor reported a proxmox pool as broken.
The controller lists Pods matching the monitor's selector and sets
Operational=false while fewer than minReplicas are ready. On kubevirt those Pods
are the virt-launcher Pods. A proxmox worker is a VM on another machine with no
Pod here, so the count is always zero and any threshold pinned the pool — and
through the lineage labels its whole application — to WorkloadsReady=False while
every node was healthy. The monitor still renders for the dashboard and the
lineage; what it no longer carries on proxmox is a threshold counted in Pods.
Two smaller things. The wait-loop log line is now written per substrate, which
restores the property the PR description claims: a default kubevirt render of
kubernetes-nodes is byte-identical to main, so no existing pool gets a new
reconcile Job on upgrade. And the comment claiming a rename of
$kubevirtmachinetemplate would roll every live worker VM is replaced: the hash is
taken over the rendered template, not the variable name — renaming it leaves the
render byte-identical, as checked.
Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
{{- fail (printf"proxmox.dnsServers[%d] is %q: entries must be IPv4 or IPv6 addresses. Talos takes addresses here, and this list is written into a shell heredoc, so anything else is both invalid for Talos and unsafe to interpolate."$i$s) -}}
211
+
{{- end -}}
212
+
{{- end -}}
213
+
{{- end -}}
214
+
215
+
{{- /*
216
+
Two ways a pool can end up on the wrong substrate, both silent, both guarded
217
+
here against the live cluster rather than against the `## @immutable` marker,
218
+
which is dashboard-only.
219
+
220
+
Switching an existing pool rolls every worker to the other hypervisor and, on
221
+
the way, drops the machine template the draining MachineSet still references:
222
+
the retention loop in nodegroup.yaml looks up only the kind the new substrate
223
+
uses, so the old kind stops being rendered and Helm deletes it — exactly the
224
+
failure that loop exists to prevent.
225
+
226
+
A pool that disagrees with its parent is the other one. The README says the
227
+
two must match; nothing checked it. A proxmox pool under a KubeVirt cluster
228
+
gets machines no ProxmoxCluster owns, and its workers never join.
229
+
230
+
Inert offline, like assertNoForeignPool and assertParentVersion above.
{{- fail (printf"kubernetes-nodes: pool %q already runs %s workers and substrate is now %q, which renders a %s. Switching rolls every worker to the other hypervisor and drops the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down instead."$groupName$liveKind (.Values.substrate| default "kubevirt") $wantTemplateKind) -}}
{{- fail (printf"kubernetes-nodes: pool %q is set to substrate %q but its parent cluster %q runs on %s. A pool's substrate must match its cluster's: its machines would be owned by no infrastructure cluster and its workers would never join."$groupName (.Values.substrate| default "kubevirt") $clusterName$parentKind) -}}
Copy file name to clipboardExpand all lines: packages/apps/kubernetes-nodes/values.schema.json
+7-1Lines changed: 7 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -264,7 +264,13 @@
264
264
"substrate": {
265
265
"description": "Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is not a supported in-place change: create a new pool instead. Changing it rolls the pool, because the worker machine template is named by a hash of its content.",
Copy file name to clipboardExpand all lines: packages/apps/kubernetes-nodes/values.yaml
+1Lines changed: 1 addition & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -132,6 +132,7 @@ talos:
132
132
##
133
133
134
134
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is not a supported in-place change: create a new pool instead. Changing it rolls the pool, because the worker machine template is named by a hash of its content.
135
+
## @immutable
135
136
substrate: kubevirt
136
137
137
138
## @typedef {struct} ProxmoxNetwork - Network attachment for a Proxmox worker VM.
{{- fail (printf"proxmox.dnsServers[%d] is %q: entries must be IPv4 or IPv6 addresses. Talos takes addresses here, and this list is written into a shell heredoc, so anything else is both invalid for Talos and unsafe to interpolate."$i$s) -}}
232
+
{{- end -}}
233
+
{{- end -}}
234
+
{{- end -}}
235
+
236
+
{{- /*
237
+
substrate decides which infrastructure provider this cluster's Cluster object
238
+
points at, and it is safe to choose only once. Switching it on a live cluster
239
+
renders no KubevirtCluster, no kccm, no kubevirt-csi controller and no
240
+
in-tenant -csi HelmRelease — 16 objects in a default render — so Flux
241
+
uninstalls the CSI driver behind every PVC the tenant already has, and the
242
+
same render adds a ProxmoxCluster and turns the apiserver Service into a
243
+
LoadBalancer. CAPI does not stop it: its Cluster webhook forbids removing
244
+
spec.infrastructureRef but admits a change of kind. The `## @immutable` marker
245
+
on the value is dashboard-only, which is why this reads the live object.
246
+
247
+
Inert offline, like the other lookup guards here: with no apiserver the lookup
248
+
is empty and the render proceeds, so `helm template` and the unit tests are
249
+
unaffected and a first install has nothing to compare against.
{{- fail (printf"kubernetes: cluster %q already runs on %s and substrate is now %q, which renders a %s. Switching the substrate of a live cluster removes the cloud-controller-manager, the CSI controller and the in-tenant CSI HelmRelease, so Flux uninstalls the driver behind every existing PVC. Create a new cluster on the other substrate and migrate the workloads instead.".Release.Name$liveKind (.Values.substrate| default "kubevirt") $wantKind) -}}
Copy file name to clipboardExpand all lines: packages/apps/kubernetes/values.schema.json
+7-1Lines changed: 7 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -36,7 +36,13 @@
36
36
"substrate": {
37
37
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.",
0 commit comments