Skip to content

Commit 4653a05

Browse files
committed
fix(kubernetes,kubernetes-nodes): close three substrate holes found in review
Three separate faults, all of them invisible in a passing test run. 1. proxmox.dnsServers reached the reconcile Job's shell unescaped. The Job applies the worker TalosConfigTemplate through an unquoted heredoc, so every byte of its body is subject to parameter expansion and command substitution at Job runtime. The INVARIANT comment above that block says each tenant-controlled value must be shell-escaped or pattern-validated; the new nameserver list was neither. `quote` there is YAML quoting. Rendered with `$(id -u)` and a backtick pair in dnsServers, extracted the way hack/talos-reconcile-heredoc_test.bats does and run under sh, the shell executed both and put their output in the worker's resolver list. The Job's pod holds the parent cluster's Talos token, cluster id, cluster secret and bootstrap token in its environment, and its Role can read the Talos CA and patch the parent KamajiControlPlane. Each entry is now validated as an IPv4 or IPv6 address at render time, in both charts. Validation rather than escaping because Talos takes addresses here: an address cannot carry a metacharacter, so the property holds by construction, and the ordinary mistake of writing a hostname is caught in the same move. IPv6 is matched on its character set rather than its full grammar — what matters is which bytes may reach the shell, and the apiserver rejects a malformed address anyway. 2. Nothing stopped a live cluster or pool from being switched between substrates. `substrate` had no immutability of any kind, and the `## @immutable` marker this adds is dashboard-enforced only, so both charts now read the live object. Switching a cluster removes sixteen objects from a default render: the KubevirtCluster, the cloud-controller-manager, the kubevirt-csi controller and the in-tenant -csi HelmRelease with all their RBAC. Flux then uninstalls the CSI driver behind every PVC the tenant has. CAPI does not prevent it — its Cluster webhook forbids removing spec.infrastructureRef but admits a change of kind. Switching a pool rolls every worker to the other hypervisor and drops the machine template the draining MachineSet still references, because the retention loop looks up only the kind the new substrate uses. A pool whose substrate differs from its parent's was not rejected either, though the README says the two must match. Both guards are inert offline, like assertNoForeignPool and assertParentVersion beside them: with no apiserver the lookup is empty and a first install renders. 3. The worker WorkloadMonitor reported a proxmox pool as broken. The controller lists Pods matching the monitor's selector and sets Operational=false while fewer than minReplicas are ready. On kubevirt those Pods are the virt-launcher Pods. A proxmox worker is a VM on another machine with no Pod here, so the count is always zero and any threshold pinned the pool — and through the lineage labels its whole application — to WorkloadsReady=False while every node was healthy. The monitor still renders for the dashboard and the lineage; what it no longer carries on proxmox is a threshold counted in Pods. Two smaller things. The wait-loop log line is now written per substrate, which restores the property the PR description claims: a default kubevirt render of kubernetes-nodes is byte-identical to main, so no existing pool gets a new reconcile Job on upgrade. And the comment claiming a rename of $kubevirtmachinetemplate would roll every live worker VM is replaced: the hash is taken over the rendered template, not the variable name — renaming it leaves the render byte-identical, as checked. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent 31cdb26 commit 4653a05

13 files changed

Lines changed: 162 additions & 8 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/templates/_helpers.tpl‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,3 +189,64 @@ string this admits parses to a positive duration.
189189
{{- end -}}
190190
{{- $value -}}
191191
{{- end -}}
192+
193+
{{- /*
194+
Nameservers are addresses, and this one is the only tenant-controlled value in
195+
these charts that reaches the reconcile Job's unquoted heredoc, where the shell
196+
expands what it finds. `quote` there is YAML quoting and does nothing about
197+
$(...) or backticks: rendered with a substitution in it, the Job's shell ran it
198+
and put the output in the worker's resolver list. Validating each entry as an
199+
address closes that by construction — an address has no metacharacters — and it
200+
also catches the ordinary mistake of writing a hostname where Talos takes only
201+
an IP. IPv6 is matched on its character set rather than its full grammar: the
202+
point here is which bytes may reach the shell, not whether every zero-run is
203+
written correctly, and the apiserver rejects a malformed address anyway.
204+
*/ -}}
205+
{{- define "kubernetes-nodes.assertDnsServersAreAddresses" -}}
206+
{{- $v4 := `^((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])$` -}}
207+
{{- range $i, $e := (default (list) .Values.proxmox.dnsServers) -}}
208+
{{- $s := $e | toString -}}
209+
{{- if not (or (regexMatch $v4 $s) (and (contains ":" $s) (regexMatch `^[0-9A-Fa-f:]+$` $s))) -}}
210+
{{- fail (printf "proxmox.dnsServers[%d] is %q: entries must be IPv4 or IPv6 addresses. Talos takes addresses here, and this list is written into a shell heredoc, so anything else is both invalid for Talos and unsafe to interpolate." $i $s) -}}
211+
{{- end -}}
212+
{{- end -}}
213+
{{- end -}}
214+
215+
{{- /*
216+
Two ways a pool can end up on the wrong substrate, both silent, both guarded
217+
here against the live cluster rather than against the `## @immutable` marker,
218+
which is dashboard-only.
219+
220+
Switching an existing pool rolls every worker to the other hypervisor and, on
221+
the way, drops the machine template the draining MachineSet still references:
222+
the retention loop in nodegroup.yaml looks up only the kind the new substrate
223+
uses, so the old kind stops being rendered and Helm deletes it — exactly the
224+
failure that loop exists to prevent.
225+
226+
A pool that disagrees with its parent is the other one. The README says the
227+
two must match; nothing checked it. A proxmox pool under a KubeVirt cluster
228+
gets machines no ProxmoxCluster owns, and its workers never join.
229+
230+
Inert offline, like assertNoForeignPool and assertParentVersion above.
231+
*/ -}}
232+
{{- define "kubernetes-nodes.assertSubstrateMatches" -}}
233+
{{- $clusterName := include "kubernetes-nodes.clusterName" . -}}
234+
{{- $groupName := include "kubernetes-nodes.groupName" . -}}
235+
{{- $isProxmox := eq (.Values.substrate | default "kubevirt") "proxmox" -}}
236+
{{- $wantTemplateKind := ternary "ProxmoxMachineTemplate" "KubevirtMachineTemplate" $isProxmox -}}
237+
{{- $wantClusterKind := ternary "ProxmoxCluster" "KubevirtCluster" $isProxmox -}}
238+
{{- $md := lookup "cluster.x-k8s.io/v1beta1" "MachineDeployment" .Release.Namespace (printf "%s-%s" $clusterName $groupName) -}}
239+
{{- if $md -}}
240+
{{- $liveKind := dig "spec" "template" "spec" "infrastructureRef" "kind" "" $md -}}
241+
{{- if and $liveKind (ne $liveKind $wantTemplateKind) -}}
242+
{{- fail (printf "kubernetes-nodes: pool %q already runs %s workers and substrate is now %q, which renders a %s. Switching rolls every worker to the other hypervisor and drops the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down instead." $groupName $liveKind (.Values.substrate | default "kubevirt") $wantTemplateKind) -}}
243+
{{- end -}}
244+
{{- end -}}
245+
{{- $cluster := lookup "cluster.x-k8s.io/v1beta1" "Cluster" .Release.Namespace $clusterName -}}
246+
{{- if $cluster -}}
247+
{{- $parentKind := dig "spec" "infrastructureRef" "kind" "" $cluster -}}
248+
{{- if and $parentKind (ne $parentKind $wantClusterKind) -}}
249+
{{- fail (printf "kubernetes-nodes: pool %q is set to substrate %q but its parent cluster %q runs on %s. A pool's substrate must match its cluster's: its machines would be owned by no infrastructure cluster and its workers would never join." $groupName (.Values.substrate | default "kubevirt") $clusterName $parentKind) -}}
250+
{{- end -}}
251+
{{- end -}}
252+
{{- end -}}

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
{{- $groupName := include "kubernetes-nodes.groupName" . }}
3232
{{- include "kubernetes-nodes.assertNoForeignPool" . }}
3333
{{- include "kubernetes-nodes.assertParentVersion" . }}
34+
{{- include "kubernetes-nodes.assertSubstrateMatches" . }}
3435
{{- $kubevirtmachinetemplateNames := list }}
3536
{{- /* Substrate selects which infrastructure provider renders this pool's
3637
machine template. Everything downstream of the template — autoscaler
@@ -566,9 +567,10 @@ spec:
566567
{{- $_ := set $context "group" $group }}
567568
{{- $_ := set $context "groupName" $groupName }}
568569
{{- $_ := set $context "clusterName" $clusterName }}
569-
{{- /* The variable keeps its kubevirt name so the rendered bytes of the
570-
default path are unchanged: this template is content-hash named, and a
571-
rename here would roll every live worker VM on every existing cluster. */}}
570+
{{- /* The $kubevirtmachinetemplate* variables now hold either kind; the names
571+
are historical. Renaming them is safe — the content hash below is taken
572+
over the rendered template, not over the variable name — and is left out
573+
of this change only to keep the diff to the substrate. */}}
572574
{{- $machineTemplateKind := ternary "ProxmoxMachineTemplate" "KubevirtMachineTemplate" $isProxmox }}
573575
{{- $kubevirtmachinetemplate := include (ternary "proxmoxmachinetemplate" "kubevirtmachinetemplate" $isProxmox) $context }}
574576
{{- $kubevirtmachinetemplateHash := $kubevirtmachinetemplate | sha256sum | trunc 6 }}
@@ -709,7 +711,19 @@ metadata:
709711
name: {{ $clusterName }}-{{ $groupName }}
710712
namespace: {{ $.Release.Namespace }}
711713
spec:
714+
{{- /* The controller counts PODS matching the selector below and reports
715+
Operational=false while fewer than minReplicas of them are ready. On
716+
kubevirt those Pods exist — they are the virt-launcher Pods, and the
717+
VMI template carries the deployment-name label further down. A proxmox
718+
worker is a VM on another machine with no Pod here at all, so the count
719+
is always zero and any minReplicas above zero would pin the pool, and
720+
through the lineage labels its whole application, to WorkloadsReady=False
721+
while every node is healthy. The object still renders: the dashboard and
722+
the lineage read it. What it cannot carry on proxmox is a threshold
723+
counted in Pods. */}}
724+
{{- if ne ($.Values.substrate | default "kubevirt") "proxmox" }}
712725
minReplicas: {{ $group.minReplicas }}
726+
{{- end }}
713727
kind: kubernetes
714728
type: worker
715729
selector:

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
{{- if eq (.Values.substrate | default "kubevirt") "proxmox" }}
2+
{{- include "kubernetes-nodes.assertDnsServersAreAddresses" . }}
3+
{{- end }}
14
{{- /*
25
Main-phase Job (NOT a Helm hook) that produces this pool's TalosConfigTemplate
36
and patches the parent KamajiControlPlane.spec.network.certSANs with the live
@@ -199,7 +202,16 @@ spec:
199202
- sh
200203
- -ec
201204
- |
202-
echo "[$(date -u +%H:%M:%S)] talos-reconcile/${GROUP_NAME}: waiting for apiserver Service address, Talos CA, K8s CA, management CoreDNS"
205+
{{- /* The wording differs per substrate because the address does:
206+
a ClusterIP on kubevirt, the Service's LoadBalancer address
207+
on proxmox. Keeping the kubevirt line exactly as it was also
208+
keeps this Job's content hash — and so its name — unchanged
209+
for every pool that is not on proxmox. */}}
210+
{{- if eq ($.Values.substrate | default "kubevirt") "proxmox" }}
211+
echo "[$(date -u +%H:%M:%S)] talos-reconcile/${GROUP_NAME}: waiting for apiserver Service LoadBalancer address, Talos CA, K8s CA, management CoreDNS"
212+
{{- else }}
213+
echo "[$(date -u +%H:%M:%S)] talos-reconcile/${GROUP_NAME}: waiting for Service ClusterIP, Talos CA, K8s CA, management CoreDNS"
214+
{{- end }}
203215
for i in $(seq 1 120); do
204216
{{- /* Which address the workers must resolve the apiserver to.
205217
On kubevirt they are Pods in this cluster and the ClusterIP

‎packages/apps/kubernetes-nodes/values.schema.json‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,13 @@
264264
"substrate": {
265265
"description": "Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is not a supported in-place change: create a new pool instead. Changing it rolls the pool, because the worker machine template is named by a hash of its content.",
266266
"type": "string",
267-
"default": "kubevirt"
267+
"default": "kubevirt",
268+
"x-kubernetes-validations": [
269+
{
270+
"rule": "self == oldSelf",
271+
"message": "substrate is immutable"
272+
}
273+
]
268274
},
269275
"proxmox": {
270276
"description": "Proxmox substrate settings.",

‎packages/apps/kubernetes-nodes/values.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ talos:
132132
##
133133

134134
## @param {string} substrate="kubevirt" - Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is not a supported in-place change: create a new pool instead. Changing it rolls the pool, because the worker machine template is named by a hash of its content.
135+
## @immutable
135136
substrate: kubevirt
136137

137138
## @typedef {struct} ProxmoxNetwork - Network attachment for a Proxmox worker VM.

‎packages/apps/kubernetes/templates/_helpers.tpl‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,3 +210,52 @@ string this admits parses to a positive duration.
210210
{{- end -}}
211211
{{- $value -}}
212212
{{- end -}}
213+
214+
{{- /*
215+
Nameservers are addresses, and this one is the only tenant-controlled value in
216+
these charts that reaches the reconcile Job's unquoted heredoc, where the shell
217+
expands what it finds. `quote` there is YAML quoting and does nothing about
218+
$(...) or backticks: rendered with a substitution in it, the Job's shell ran it
219+
and put the output in the worker's resolver list. Validating each entry as an
220+
address closes that by construction — an address has no metacharacters — and it
221+
also catches the ordinary mistake of writing a hostname where Talos takes only
222+
an IP. IPv6 is matched on its character set rather than its full grammar: the
223+
point here is which bytes may reach the shell, not whether every zero-run is
224+
written correctly, and the apiserver rejects a malformed address anyway.
225+
*/ -}}
226+
{{- define "kubernetes.assertDnsServersAreAddresses" -}}
227+
{{- $v4 := `^((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])$` -}}
228+
{{- range $i, $e := (default (list) .Values.proxmox.dnsServers) -}}
229+
{{- $s := $e | toString -}}
230+
{{- if not (or (regexMatch $v4 $s) (and (contains ":" $s) (regexMatch `^[0-9A-Fa-f:]+$` $s))) -}}
231+
{{- fail (printf "proxmox.dnsServers[%d] is %q: entries must be IPv4 or IPv6 addresses. Talos takes addresses here, and this list is written into a shell heredoc, so anything else is both invalid for Talos and unsafe to interpolate." $i $s) -}}
232+
{{- end -}}
233+
{{- end -}}
234+
{{- end -}}
235+
236+
{{- /*
237+
substrate decides which infrastructure provider this cluster's Cluster object
238+
points at, and it is safe to choose only once. Switching it on a live cluster
239+
renders no KubevirtCluster, no kccm, no kubevirt-csi controller and no
240+
in-tenant -csi HelmRelease — 16 objects in a default render — so Flux
241+
uninstalls the CSI driver behind every PVC the tenant already has, and the
242+
same render adds a ProxmoxCluster and turns the apiserver Service into a
243+
LoadBalancer. CAPI does not stop it: its Cluster webhook forbids removing
244+
spec.infrastructureRef but admits a change of kind. The `## @immutable` marker
245+
on the value is dashboard-only, which is why this reads the live object.
246+
247+
Inert offline, like the other lookup guards here: with no apiserver the lookup
248+
is empty and the render proceeds, so `helm template` and the unit tests are
249+
unaffected and a first install has nothing to compare against.
250+
*/ -}}
251+
{{- define "kubernetes.assertSubstrateUnchanged" -}}
252+
{{- $isProxmox := eq (.Values.substrate | default "kubevirt") "proxmox" -}}
253+
{{- $wantKind := ternary "ProxmoxCluster" "KubevirtCluster" $isProxmox -}}
254+
{{- $live := lookup "cluster.x-k8s.io/v1beta1" "Cluster" .Release.Namespace .Release.Name -}}
255+
{{- if $live -}}
256+
{{- $liveKind := dig "spec" "infrastructureRef" "kind" "" $live -}}
257+
{{- if and $liveKind (ne $liveKind $wantKind) -}}
258+
{{- fail (printf "kubernetes: cluster %q already runs on %s and substrate is now %q, which renders a %s. Switching the substrate of a live cluster removes the cloud-controller-manager, the CSI controller and the in-tenant CSI HelmRelease, so Flux uninstalls the driver behind every existing PVC. Create a new cluster on the other substrate and migrate the workloads instead." .Release.Name $liveKind (.Values.substrate | default "kubevirt") $wantKind) -}}
259+
{{- end -}}
260+
{{- end -}}
261+
{{- end -}}

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,9 @@
4949
{{- if not .Values.proxmox.dnsServers }}
5050
{{- fail "substrate proxmox requires proxmox.dnsServers: the ProxmoxCluster schema makes it mandatory" }}
5151
{{- end }}
52+
{{- include "kubernetes.assertDnsServersAreAddresses" . }}
5253
{{- end }}
54+
{{- include "kubernetes.assertSubstrateUnchanged" . }}
5355
{{- $infraKind := ternary "ProxmoxCluster" "KubevirtCluster" $isProxmox }}
5456
{{- /*
5557
When $etcd is empty, the parent Tenant application has not populated

‎packages/apps/kubernetes/values.schema.json‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,13 @@
3636
"substrate": {
3737
"description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.",
3838
"type": "string",
39-
"default": "kubevirt"
39+
"default": "kubevirt",
40+
"x-kubernetes-validations": [
41+
{
42+
"rule": "self == oldSelf",
43+
"message": "substrate is immutable"
44+
}
45+
]
4046
},
4147
"proxmox": {
4248
"description": "Proxmox substrate settings.",

0 commit comments

Comments
 (0)