Skip to content

Commit 98d5c8b

Browse files
myasnikovdaniilmyasnikovdaniil
authored
feat(kubernetes-nodes)!: render registry mirrors and their TLS settings as Talos documents
A pool could mirror a registry but could not tell its workers to trust the mirror's CA or skip verification, so an in-cluster mirror served by an internal CA was unusable: the worker's pulls failed on x509. talos.registryMirrors now renders one RegistryMirrorConfig document per host and the new talos.registryConfig one RegistryTLSConfig per endpoint that sets tls.ca or tls.insecureSkipVerify. The deprecated machine.registries section is no longer rendered. Both document kinds exist from Talos 1.12, so a pool that resolves to an older release and would render one now fails the render, naming the pool and the release. That is breaking for a pool on Talos below 1.12 that already sets registryMirrors: it rendered before and does not now, while its running workers are untouched. Pools without mirrors and TLS settings render what they rendered before. What the old passthrough handed to the node unchecked is refused at render: switches take true or false, endpoints are a list of non-empty strings, and tls.ca must be a plain PEM certificate, through the shared validator that also refuses private key material. Credentials and client identity are refused rather than ignored, because the rendered text is stored in the tenant namespace. A tls block that is false, 0, an empty string or an empty list is refused too instead of being read as empty, and so is a host or endpoint written as a URL, which Talos never matches to a pull. A pool that sets registryMirrors gets a new TalosConfigTemplate and its workers roll, because the rendered machine config changes. The pool chart's own text now says that only talos.version is kept in step with the parent chart. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent cc807af commit 98d5c8b

9 files changed

Lines changed: 870 additions & 47 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 4 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎hack/talos-reconcile-heredoc_test.bats‎

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,11 @@
2626
# escaped the value reaches the heredoc and is escaped for backslash,
2727
# dollar and backtick. Assert it survives a real shell as a
2828
# LITERAL. This covers talos.installerRepository and
29-
# talos.registryMirrors, both genuinely free-form.
29+
# talos.registryMirrors and the endpoint names of
30+
# talos.registryConfig, all genuinely free-form.
3031
# validated the value is refused at render before it can reach the heredoc.
31-
# Assert the refusal. This covers talos.schematicID and
32+
# Assert the refusal. This covers talos.registryConfig tls.ca,
33+
# talos.schematicID and
3234
# talos.version, which also land in the worker DataVolume name and
3335
# the image URL, where an unquoted YAML scalar makes escaping the
3436
# wrong tool. templates/nodegroup.yaml pattern-checks them. It also
@@ -147,6 +149,68 @@ VALS
147149
rm -rf "$work"
148150
}
149151

152+
@test "kubernetes-nodes worker TalosConfigTemplate heredoc keeps a hostile registryConfig endpoint name literal" {
153+
work=$(mktemp -d)
154+
cat > "$work/vals.yaml" <<'VALS'
155+
cluster: myk8s
156+
_cluster:
157+
cluster-domain: cozy.local
158+
version: "v1.35"
159+
minReplicas: 0
160+
maxReplicas: 3
161+
instanceType: ""
162+
diskSize: 20Gi
163+
storageClass: replicated
164+
roles: [ingress-nginx]
165+
resources: {cpu: "2", memory: 4Gi}
166+
VALS
167+
helm template kubernetes-nodes-myk8s-md0 packages/apps/kubernetes-nodes -n tenant-test -f "$work/vals.yaml" \
168+
--set 'talos.registryConfig.m$(id)`x`y\.example\.org.tls.insecureSkipVerify=true' \
169+
--show-only templates/talos-reconcile-job.yaml \
170+
| yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' \
171+
> "$work/cmd.sh"
172+
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
173+
awk '
174+
/(^|=\"?\$\()cat <<EOF/ { print "cat <<EOF"; inblock=1; next }
175+
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
176+
inblock { print }
177+
' "$work/cmd.sh" > "$work/heredoc.sh"
178+
grep -q '^cat <<EOF$' "$work/heredoc.sh" || { echo "could not extract the kubernetes-nodes heredoc" >&2; rm -rf "$work"; exit 1; }
179+
out=$(sh "$work/heredoc.sh" 2>"$work/err") || { echo "kubernetes-nodes heredoc shell exited non-zero" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
180+
[ -n "$out" ] || { echo "kubernetes-nodes heredoc emitted no output" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
181+
printf '%s' "$out" | grep -q 'kind: TalosConfigTemplate' || { echo "kubernetes-nodes heredoc output is not the TalosConfigTemplate" >&2; rm -rf "$work"; exit 1; }
182+
printf '%s' "$out" | grep -qF 'name: m$(id)`x`y.example.org' || { echo "kubernetes-nodes hostile registryConfig endpoint name not preserved literally" >&2; rm -rf "$work"; exit 1; }
183+
rm -rf "$work"
184+
}
185+
186+
@test "kubernetes-nodes refuses a registryConfig CA that is not a plain certificate at render" {
187+
work=$(mktemp -d)
188+
cat > "$work/vals.yaml" <<'VALS'
189+
cluster: myk8s
190+
_cluster:
191+
cluster-domain: cozy.local
192+
version: "v1.35"
193+
minReplicas: 0
194+
maxReplicas: 3
195+
instanceType: ""
196+
diskSize: 20Gi
197+
storageClass: replicated
198+
roles: [ingress-nginx]
199+
resources: {cpu: "2", memory: 4Gi}
200+
VALS
201+
# tls.ca is validated by cozy-lib.tls.validateCACert, so a shell metacharacter
202+
# cannot reach the heredoc through it. The check is that the render refuses it.
203+
if helm template kubernetes-nodes-myk8s-md0 packages/apps/kubernetes-nodes \
204+
-n tenant-test -f "$work/vals.yaml" \
205+
--set 'talos.registryConfig.mirror\.example\.org.tls.ca=MIIB$(id)`x`y' >/dev/null 2>"$work/err"; then
206+
echo "render accepted a hostile registryConfig ca, so the value reaches the heredoc" >&2
207+
rm -rf "$work"; exit 1
208+
fi
209+
grep -q 'must contain a complete PEM certificate block' "$work/err" \
210+
|| { echo "render failed on the registryConfig ca for some other reason:" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
211+
rm -rf "$work"
212+
}
213+
150214
# The proxmox substrate adds the only tenant-controlled value in this heredoc
151215
# that is not shell-escaped: proxmox.dnsServers. `quote` there is YAML quoting
152216
# and leaves $(...) and backticks for the Job's shell to run — rendered with a

0 commit comments

Comments
 (0)