Skip to content

Commit 020c545

Browse files
IvanHuntersIvanHunters
committed
feat(kubernetes-nodes): add talos.registryMirrors, in parity with the parent chart
The kubernetes-nodes worker-pool chart pulls the same ghcr.io/siderolabs/kubelet image as the parent kubernetes chart, and its talos values are kept byte-parity in sync with it. The registryMirrors knob added to the parent was missing here, leaving worker pools provisioned as KubernetesNodes with no way to route the kubelet image pull through a mirror in an air-gapped or flaky-egress environment (#3513). Add the same talos.registryMirrors passthrough to the worker TalosConfigTemplate (machine.registries.mirrors), using a Helm comment (not a YAML comment) and the same backslash/dollar/backtick escaping so free-form values stay literal in the unquoted reconcile-Job heredoc. Empty by default: golden render parity with the parent is preserved (render-parity.sh green). Regenerated schema, README, Go types, deepcopy and the kubernetes-nodes-rd cozyrd. Extends the heredoc execution test to this chart. Refs #3513 Signed-off-by: IvanHunters <[email protected]>
1 parent 102e694 commit 020c545

9 files changed

Lines changed: 148 additions & 11 deletions

File tree

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎hack/talos-reconcile-heredoc_test.bats‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,3 +79,37 @@ CHART=packages/apps/kubernetes
7979
printf '%s' "$out" | grep -q 'mirrors:' && { echo "default unexpectedly rendered a mirrors block" >&2; rm -rf "$work"; exit 1; }
8080
rm -rf "$work"
8181
}
82+
83+
@test "kubernetes-nodes worker TalosConfigTemplate heredoc emits with a hostile registryMirrors value" {
84+
work=$(mktemp -d)
85+
cat > "$work/vals.yaml" <<'VALS'
86+
cluster: myk8s
87+
_cluster:
88+
cluster-domain: cozy.local
89+
version: "v1.35"
90+
minReplicas: 0
91+
maxReplicas: 3
92+
instanceType: ""
93+
diskSize: 20Gi
94+
storageClass: replicated
95+
roles: [ingress-nginx]
96+
resources: {cpu: "2", memory: 4Gi}
97+
VALS
98+
helm template kubernetes-nodes-myk8s-md0 packages/apps/kubernetes-nodes -n tenant-test -f "$work/vals.yaml" \
99+
--set 'talos.registryMirrors.ghcr\.io.endpoints[0]=http://m$(id)`x`y' \
100+
--show-only templates/talos-reconcile-job.yaml \
101+
| yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' \
102+
> "$work/cmd.sh"
103+
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
104+
awk '
105+
/^cat <<EOF \| kubectl apply/ { print "cat <<EOF"; inblock=1; next }
106+
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
107+
inblock { print }
108+
' "$work/cmd.sh" > "$work/heredoc.sh"
109+
grep -q '^cat <<EOF$' "$work/heredoc.sh" || { echo "could not extract the kubernetes-nodes heredoc" >&2; rm -rf "$work"; exit 1; }
110+
out=$(sh "$work/heredoc.sh" 2>"$work/err") || { echo "kubernetes-nodes heredoc shell exited non-zero" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
111+
[ -n "$out" ] || { echo "kubernetes-nodes heredoc emitted no output" >&2; cat "$work/err" >&2; rm -rf "$work"; exit 1; }
112+
printf '%s' "$out" | grep -q 'kind: TalosConfigTemplate' || { echo "kubernetes-nodes heredoc output is not the TalosConfigTemplate" >&2; rm -rf "$work"; exit 1; }
113+
printf '%s' "$out" | grep -qF 'http://m$(id)`x`y' || { echo "kubernetes-nodes hostile endpoint not preserved literally" >&2; rm -rf "$work"; exit 1; }
114+
rm -rf "$work"
115+
}

‎packages/apps/kubernetes-nodes/README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -61,13 +61,14 @@ the parent cluster `kubernetes-<cluster>` in the same namespace, plus a
6161

6262
### Talos Worker Image
6363

64-
| Name | Description | Type | Value |
65-
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
66-
| `talos` | Talos worker image configuration. Keep in sync with the parent cluster's `talos`. | `object` | `{}` |
67-
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
68-
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
69-
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
70-
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
64+
| Name | Description | Type | Value |
65+
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ |
66+
| `talos` | Talos worker image configuration. Keep in sync with the parent cluster's `talos`. | `object` | `{}` |
67+
| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` |
68+
| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` |
69+
| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` |
70+
| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` |
71+
| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Keep in sync with the parent kubernetes chart. See cozystack/cozystack#3513. | `object` | `{}` |
7172

7273

7374
### Images

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,6 +325,12 @@ spec:
325325
rbac: true
326326
kubePrism:
327327
enabled: false
328+
{{- with $.Values.talos.registryMirrors }}
329+
registries:
330+
mirrors:
331+
{{- /* registryMirrors is free-form tenant-facing input rendered into the unquoted reconcile-Job heredoc above, so the value is escaped (backslash, dollar, backtick) to render as a literal and never be shell-expanded or command-substituted at Job runtime. This is a Helm comment: it is stripped at render, so its own text never reaches the heredoc. See cozystack cozystack#3513. */}}
332+
{{- toYaml . | replace "\\" "\\\\" | replace "$" "\\$" | replace "`" "\\`" | nindent 28 }}
333+
{{- end }}
328334
cluster:
329335
id: ${CLUSTER_ID}
330336
secret: ${CLUSTER_SECRET}

‎packages/apps/kubernetes-nodes/tests/registry_mirrors_test.yaml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
suite: worker registry mirrors (talos.registryMirrors)
2+
3+
# The kubernetes-nodes worker pool pulls the same ghcr.io/siderolabs/kubelet image
4+
# as the parent kubernetes chart and its talos values are kept in sync with it, so
5+
# it carries the same talos.registryMirrors knob rendered into the worker
6+
# TalosConfigTemplate's machine.registries.mirrors. Empty by default (byte-identical
7+
# render); a set value routes worker image pulls through a mirror. Free-form
8+
# tenant-facing input is shell-escaped because it is emitted into the reconcile-Job's
9+
# unquoted `cat <<EOF | kubectl apply` heredoc. See cozystack/cozystack#3513.
10+
11+
templates:
12+
- templates/talos-reconcile-job.yaml
13+
release:
14+
name: kubernetes-nodes-myk8s-md0
15+
namespace: tenant-test
16+
set:
17+
cluster: myk8s
18+
_cluster:
19+
cluster-domain: cozy.local
20+
version: "v1.35"
21+
minReplicas: 0
22+
maxReplicas: 3
23+
instanceType: ""
24+
diskSize: 20Gi
25+
storageClass: replicated
26+
roles:
27+
- ingress-nginx
28+
resources:
29+
cpu: "2"
30+
memory: 4Gi
31+
32+
tests:
33+
- it: default renders no machine-level mirrors block
34+
asserts:
35+
- notMatchRegex:
36+
path: spec.template.spec.containers[0].command[2]
37+
pattern: 'mirrors:'
38+
documentSelector:
39+
path: kind
40+
value: Job
41+
42+
- it: talos.registryMirrors renders machine.registries.mirrors into the worker TalosConfigTemplate
43+
set:
44+
talos:
45+
registryMirrors:
46+
ghcr.io:
47+
endpoints:
48+
- http://ghcr-mirror.kube-system.svc
49+
asserts:
50+
- matchRegex:
51+
path: spec.template.spec.containers[0].command[2]
52+
pattern: 'mirrors:'
53+
documentSelector:
54+
path: kind
55+
value: Job
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].command[2]
58+
pattern: 'ghcr\.io:'
59+
documentSelector:
60+
path: kind
61+
value: Job
62+
- matchRegex:
63+
path: spec.template.spec.containers[0].command[2]
64+
pattern: 'http://ghcr-mirror\.kube-system\.svc'
65+
documentSelector:
66+
path: kind
67+
value: Job
68+
69+
- it: registryMirrors values are shell-escaped so they cannot execute in the reconcile-Job heredoc
70+
set:
71+
talos:
72+
registryMirrors:
73+
ghcr.io:
74+
endpoints:
75+
- http://mirror$(id)
76+
asserts:
77+
- matchRegex:
78+
path: spec.template.spec.containers[0].command[2]
79+
pattern: 'http://mirror\\\$\(id\)'
80+
documentSelector:
81+
path: kind
82+
value: Job

‎packages/apps/kubernetes-nodes/values.schema.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,7 @@
170170
"required": [
171171
"imageFactoryURL",
172172
"installerRepository",
173+
"registryMirrors",
173174
"schematicID",
174175
"version"
175176
],
@@ -184,6 +185,12 @@
184185
"type": "string",
185186
"default": "factory.talos.dev/installer"
186187
},
188+
"registryMirrors": {
189+
"description": "Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Keep in sync with the parent kubernetes chart. See cozystack/cozystack#3513.",
190+
"type": "object",
191+
"default": {},
192+
"x-kubernetes-preserve-unknown-fields": true
193+
},
187194
"schematicID": {
188195
"description": "Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.",
189196
"type": "string",

‎packages/apps/kubernetes-nodes/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,13 +87,15 @@ version: "v1.35"
8787
## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.
8888
## @field {string} imageFactoryURL - Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.
8989
## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `<installerRepository>/<schematicID>:<version>`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.
90+
## @field {object} registryMirrors - Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Keep in sync with the parent kubernetes chart. See cozystack/cozystack#3513.
9091

9192
## @param {Talos} talos - Talos worker image configuration. Keep in sync with the parent cluster's `talos`.
9293
talos:
9394
version: "v1.13.6"
9495
schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"
9596
imageFactoryURL: "https://factory.talos.dev"
9697
installerRepository: "factory.talos.dev/installer"
98+
registryMirrors: {}
9799

98100
##
99101
## @section Images

0 commit comments

Comments
 (0)