Skip to content

Commit a860df1

Browse files
themoriartithemoriarti
committed
fix(kubernetes-nodes): keep the TalosConfigTemplate out of the filesystem
The reconcile Job runs with readOnlyRootFilesystem and no volumes, so writing the manifest to /tmp died with "can't create /tmp/tct.yaml" — the Job never applied a TalosConfigTemplate, workers never joined, and both kubernetes e2e suites timed out waiting on their machinedeployments. That was on the KubeVirt path, which this series is supposed to leave untouched. The manifest now lives in a shell variable, which keeps the container hardened as it was and has the property the retry needs anyway: the second apply sends exactly what the first one did, rather than re-rendering it. Caught by e2e on the pull request. Nothing in the unit suites could have seen it — they assert on the script's text, and the text was correct; what was wrong was the filesystem it assumed. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent 2a2f06c commit a860df1

2 files changed

Lines changed: 12 additions & 5 deletions

File tree

‎hack/talos-reconcile-heredoc_test.bats‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ VALS
5151
> "$work/cmd.sh"
5252
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
5353
awk '
54-
/^cat <<EOF([[:space:]]*\||[[:space:]]*>)/ { print "cat <<EOF"; inblock=1; next }
54+
/(^|=\"?\$\()cat <<EOF/ { print "cat <<EOF"; inblock=1; next }
5555
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
5656
inblock { print }
5757
' "$work/cmd.sh" > "$work/heredoc.sh"
@@ -86,7 +86,7 @@ VALS
8686
> "$work/cmd.sh"
8787
[ -s "$work/cmd.sh" ] || { echo "kubernetes-nodes render produced no Job command" >&2; rm -rf "$work"; exit 1; }
8888
awk '
89-
/^cat <<EOF([[:space:]]*\||[[:space:]]*>)/ { print "cat <<EOF"; inblock=1; next }
89+
/(^|=\"?\$\()cat <<EOF/ { print "cat <<EOF"; inblock=1; next }
9090
inblock && /^EOF$/ { print "EOF"; inblock=0; next }
9191
inblock { print }
9292
' "$work/cmd.sh" > "$work/heredoc.sh"

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,13 @@ spec:
249249
# Apply the TalosConfigTemplate. ownerReference back to the parent
250250
# KamajiControlPlane makes the standard Kubernetes garbage collector
251251
# clean it up when the parent cluster is deleted.
252-
cat <<EOF > /tmp/tct.yaml
252+
{{- /* Held in a variable, not a file: the container runs with
253+
readOnlyRootFilesystem, so /tmp is not writable and an
254+
earlier revision of this died on `can't create
255+
/tmp/tct.yaml`. Keeping the manifest in one variable also
256+
means the retry below applies exactly what the first
257+
attempt did. */}}
258+
TCT_MANIFEST="$(cat <<EOF
253259
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
254260
kind: TalosConfigTemplate
255261
metadata:
@@ -393,6 +399,7 @@ spec:
393399
service:
394400
disabled: true
395401
EOF
402+
)"
396403
397404
# The CRD marks spec immutable, so apply fails the moment any input
398405
# this render depends on has genuinely changed -- the apiserver
@@ -405,12 +412,12 @@ spec:
405412
# message is narrow on purpose: silently deleting and recreating on
406413
# an unrecognised error would turn an unrelated fault into worker
407414
# churn.
408-
if ! APPLY_OUT="$(kubectl apply -f /tmp/tct.yaml 2>&1)"; then
415+
if ! APPLY_OUT="$(printf '%s\n' "$TCT_MANIFEST" | kubectl apply -f - 2>&1)"; then
409416
case "$APPLY_OUT" in
410417
*"is immutable"*|*"Spec is immutable"*)
411418
echo "[$(date -u +%H:%M:%S)] TalosConfigTemplate ${RELEASE}-${GROUP_NAME} spec changed; replacing the immutable object"
412419
kubectl delete talosconfigtemplate "${RELEASE}-${GROUP_NAME}" -n "$NS"
413-
kubectl apply -f /tmp/tct.yaml
420+
printf '%s\n' "$TCT_MANIFEST" | kubectl apply -f -
414421
;;
415422
*)
416423
echo "$APPLY_OUT" >&2

0 commit comments

Comments
 (0)