Skip to content

Commit ecf0e27

Browse files
myasnikovdaniilmyasnikovdaniil
committed
feat(kubernetes): boot worker disk by cloning the shared golden image
Point each new worker's disk-system DataVolume at a CDI storage-layer clone of the golden Talos image in cozy-public instead of a per-worker HTTP import, removing the per-worker Image Factory dependency and the kubernetes-* e2e node-join flake where N workers race to reach the image cache ClusterIP across the sandbox CNI (#3231). The clone path omits the usePopulator=false annotation so CDI's populator does a CSI clone (live-verified: temp PVC CSI-cloned + resized to the worker diskSize, RWX, no host-assisted upload pod), not a pod-network copy. Existing node groups are never migrated: the KubevirtMachineTemplate is content-hashed, so both the source and that annotation are pinned to the live KMT's values when a MachineDeployment for the group already exists (render and hash unchanged, no worker roll); only a brand-new group with a matching golden clones, else HTTP. lookup() is empty under helm template, so renders and unit tests take the HTTP path (verified: KMT byte-identical old-vs-new). Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent d162115 commit ecf0e27

1 file changed

Lines changed: 52 additions & 1 deletion

File tree

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,19 +89,70 @@ spec:
8989
{{- end }}
9090
runStrategy: Always
9191
dataVolumeTemplates:
92+
{{- /* Prefer a CDI storage-layer clone of the shared golden Talos image in
93+
cozy-public (packages/system/kubernetes-worker-image) over a per-worker
94+
HTTP import: the clone is a CSI copy, so N workers no longer each stream
95+
the OS image across the pod network — the source of the kubernetes-*
96+
e2e node-join flake (cozystack/cozystack#3231) and a per-worker Image
97+
Factory dependency in production. The golden is named
98+
talos-worker-<schematicID>-<version>; keep in sync with that package's
99+
templates/dv.yaml.
100+
101+
Existing node groups are never migrated: this whole template is
102+
content-hashed to name the KubevirtMachineTemplate (see the KMT block
103+
below), so changing the source (or the usePopulator annotation below)
104+
would roll every worker. When a MachineDeployment for this group already
105+
exists, re-emit the source type its live KMT already uses (http stays
106+
http, clone stays clone) so the render — and the hash — is unchanged.
107+
Only a brand new group (no live MD) adopts the clone, and only when the
108+
matching golden exists; otherwise it falls back to the HTTP import.
109+
lookup() is empty under `helm template`, so renders/unit-tests always
110+
take the HTTP path. */}}
111+
{{- $httpURL := printf "%s/image/%s/%s/openstack-amd64.raw.xz" ($.Values.talos.imageFactoryURL | trimSuffix "/") $.Values.talos.schematicID $.Values.talos.version }}
112+
{{- $goldenName := printf "talos-worker-%s-%s" $.Values.talos.schematicID $.Values.talos.version }}
113+
{{- $liveSourceType := "" }}
114+
{{- $liveMD := lookup "cluster.x-k8s.io/v1beta1" "MachineDeployment" $.Release.Namespace (printf "%s-%s" $.Release.Name .groupName) }}
115+
{{- if $liveMD }}
116+
{{- $ref := dig "spec" "template" "spec" "infrastructureRef" (dict) $liveMD }}
117+
{{- if and (dig "name" "" $ref) (eq (dig "kind" "" $ref) "KubevirtMachineTemplate") }}
118+
{{- $liveKMT := lookup "infrastructure.cluster.x-k8s.io/v1alpha1" "KubevirtMachineTemplate" $.Release.Namespace (dig "name" "" $ref) }}
119+
{{- range (dig "spec" "template" "spec" "virtualMachineTemplate" "spec" "dataVolumeTemplates" (list) $liveKMT) }}
120+
{{- if eq (dig "metadata" "name" "" .) "disk-system" }}
121+
{{- if hasKey (dig "spec" "source" (dict) .) "pvc" }}{{- $liveSourceType = "pvc" }}{{- else if hasKey (dig "spec" "source" (dict) .) "http" }}{{- $liveSourceType = "http" }}{{- end }}
122+
{{- end }}
123+
{{- end }}
124+
{{- end }}
125+
{{- end }}
126+
{{- $useClone := eq $liveSourceType "pvc" }}
127+
{{- if not $liveMD }}{{- if lookup "cdi.kubevirt.io/v1beta1" "DataVolume" "cozy-public" $goldenName }}{{- $useClone = true }}{{- end }}{{- end }}
92128
- metadata:
93129
name: disk-system
130+
{{- /* usePopulator=false stays ONLY on the HTTP/import path (and thus on
131+
every existing group, pinned to its live source above): it preserves
132+
the legacy import behaviour and keeps the rendered DV — hence the KMT
133+
hash — byte-identical, so existing workers never roll. The clone path
134+
omits it so CDI's populator performs a storage-layer CSI clone of the
135+
golden (verified on dev10: temp PVC CSI-cloned + resized, no
136+
host-assisted upload pod) instead of a slower pod-network copy. */}}
137+
{{- if not $useClone }}
94138
annotations:
95139
cdi.kubevirt.io/storage.usePopulator: "false"
140+
{{- end }}
96141
spec:
97142
{{- /* Use the openstack/configdrive Talos image: KubeVirt CAPK
98143
injects bootstrap data via cloudInitConfigDrive (OpenStack
99144
metadata service shape), and the openstack platform image
100145
is the one that reads it. The nocloud-amd64 image silently
101146
ignores configdrive and never applies the machineconfig. */}}
102147
source:
148+
{{- if $useClone }}
149+
pvc:
150+
namespace: cozy-public
151+
name: {{ $goldenName }}
152+
{{- else }}
103153
http:
104-
url: {{ $.Values.talos.imageFactoryURL | trimSuffix "/" }}/image/{{ $.Values.talos.schematicID }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz
154+
url: {{ $httpURL }}
155+
{{- end }}
105156
{{- if hasKey .group "ephemeralStorage" }}
106157
{{- fail (printf "nodeGroup %q: ephemeralStorage is no longer supported and should have been automatically migrated to diskSize by platform migration 41. If you see this error after upgrading, the migration did not run — check the cozystack-migration-hook Job logs in cozy-system." .groupName) }}
107158
{{- end }}

0 commit comments

Comments
 (0)