Skip to content

Commit 048c029

Browse files
authored
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it (#4971)
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. `dump_diagnostics` then read whatever sat at that path, so a run that failed before `fetch_tenant_kubeconfig` reported the previous run's tenant as its own. - The file now lives under `RUNNER_TEMP`, which the runner exports into every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it for a local run, and `TMPDIR` is the last fallback. The workflow sets no path of its own: the `runner` context is not available in job-level `env`. - `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell, since the workflow's `umask 077` covers only the staging step. - `apply-tenant`, the suite's first step, removes a leftover before doing anything else, so diagnostics in this run can only see this run's file. - The suite's `finally` removes it, before the `keep_tenant` exit, and the workflow's "Remove staged credentials" step removes it again for a run killed before `finally`. Verification: `sh -n` and shellcheck on the script; actionlint on `e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported, as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases with a stub `kubectl`: the fetched file has no group or other bits under `umask 022`, `apply-tenant` removes a planted leftover, and the suite's `finally` (run out of the Chainsaw file on the `keep_tenant` path) removes the file, so `diagnostics` after it prints no tenant section. Each goes red with its line (`umask 077`, the leftover `rm`, the `finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9, teardown contract 8, step1b 4, talos template 12), also under `hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have cannot dispatch the workflow at the moment, so the workflow steps themselves are unexercised beyond actionlint. Closes #4806 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Proxmox end-to-end test runs store tenant credentials in a temporary work directory and create kubeconfig files with restricted permissions. * Stale tenant kubeconfig files are cleared before a new tenant fixture is applied. Cleanup removes the file even when tenant data is retained or a run is interrupted, preventing credentials from appearing in later diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3abdff5 + 3d97e91 commit 048c029

4 files changed

Lines changed: 95 additions & 4 deletions

File tree

‎.github/workflows/e2e-proxmox.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,4 +160,8 @@ jobs:
160160
161161
- name: Remove staged credentials
162162
if: always()
163-
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"
163+
# The tenant kubeconfig is the suite's, fetched by run-proxmox.sh into
164+
# RUNNER_TEMP, which the runner exports into every step (the runner
165+
# context is not allowed in job-level env); its finally removes it too,
166+
# but a run killed mid-step never reaches that.
167+
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY" "$RUNNER_TEMP/proxmox-e2e-tenant.kubeconfig"

‎hack/e2e-chainsaw/_lib/run-proxmox.sh‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,16 @@ CLUSTER="${COZY_PROXMOX_CLUSTER:-k8s-e2e}"
2222
PVE_SSH="${COZY_PVE_SSH:?COZY_PVE_SSH is required: the ssh destination of the Proxmox host, user@host}"
2323
PVE_SSH_KEY="${COZY_PVE_SSH_KEY:-$HOME/.ssh/id_ed25519}"
2424
STORAGE="${COZY_PROXMOX_STORAGE:?COZY_PROXMOX_STORAGE is required, the PVE storage id for the PVC probe}"
25-
TENANT_KUBECONFIG="${TMPDIR:-/tmp}/proxmox-e2e-tenant.kubeconfig"
25+
# Where this run keeps the tenant's super-admin kubeconfig. Per run rather than
26+
# a fixed path under /tmp: the file is a credential for the whole tenant, so it
27+
# must not outlive the run on a shared runner, and dump_diagnostics reads
28+
# whatever is at this path, so a file left by a previous run would be reported
29+
# as this run's tenant. On the workflow's runner this is RUNNER_TEMP, which the
30+
# runner exports into every step and empties between jobs; apply-tenant removes
31+
# a leftover before it does anything else, and the suite's finally removes the
32+
# file when the run ends.
33+
WORKDIR="${COZY_PROXMOX_WORKDIR:-${RUNNER_TEMP:-${TMPDIR:-/tmp}}}"
34+
TENANT_KUBECONFIG="$WORKDIR/proxmox-e2e-tenant.kubeconfig"
2635
FIXTURE="$(cd "$(dirname "$0")/.." && pwd)/kubernetes-proxmox/tenant.yaml"
2736

2837
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
@@ -77,6 +86,10 @@ render_tenant() {
7786
}
7887

7988
apply_tenant() {
89+
# First step of the suite, so this is where a kubeconfig left by an earlier
90+
# run on this runner is removed: from here on the path holds this run's file
91+
# or nothing, and dump_diagnostics cannot pick up another tenant.
92+
rm -f "$TENANT_KUBECONFIG"
8093
render_tenant | kubectl apply -f -
8194
}
8295

@@ -85,8 +98,16 @@ apply_tenant() {
8598
# carries the address a runner outside the management cluster can reach; .svc
8699
# is the in-cluster Service the management-side controllers use.
87100
fetch_tenant_kubeconfig() {
88-
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
89-
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
101+
# Written under umask 077 in a subshell of its own: the workflow sets that
102+
# umask only in its staging step, and a chainsaw script step runs with the
103+
# runner's default, under which the file would be readable by every local
104+
# user. No pipefail in POSIX sh, so a failed read leaves an empty file, which
105+
# the size check below turns into a failure rather than an empty kubeconfig.
106+
(
107+
umask 077
108+
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
109+
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
110+
)
90111
[ -s "$TENANT_KUBECONFIG" ] || die "tenant kubeconfig is empty — did the control plane come up?"
91112
}
92113

‎hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,12 @@ spec:
124124
- script:
125125
content: |
126126
set -eu
127+
# The tenant's super-admin kubeconfig that run-proxmox.sh fetched for
128+
# the checks above; the same path the script derives. Removed before
129+
# the keep_tenant exit below, because the file is a credential and
130+
# keeping the tenant is not a reason to keep that on the runner. The
131+
# workflow's cleanup step removes it again, for a run killed here.
132+
rm -f "${COZY_PROXMOX_WORKDIR:-${RUNNER_TEMP:-${TMPDIR:-/tmp}}}/proxmox-e2e-tenant.kubeconfig"
127133
# keep_tenant is honoured here as well as in the workflow's own
128134
# teardown, or the suite would delete what the operator asked to keep.
129135
if [ "${COZY_KEEP_TENANT:-false}" = "true" ]; then

‎hack/proxmox-e2e-fixture_test.bats‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,63 @@ RUNNER="$REPO_ROOT/hack/e2e-chainsaw/_lib/run-proxmox.sh"
7676
rm -rf "$stub"
7777
case "$out" in *"is gone from main-pool"*) ;; *) echo "$out" >&2; exit 1 ;; esac
7878
}
79+
80+
# The tenant's super-admin kubeconfig is a credential for the whole tenant, kept
81+
# under RUNNER_TEMP for one run. The workflow sets no path of its own, so these
82+
# cases go through the same RUNNER_TEMP fallback the runner provides, with the
83+
# override cleared.
84+
KUBECONFIG_NAME=proxmox-e2e-tenant.kubeconfig
85+
SUITE="$REPO_ROOT/hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled"
86+
87+
@test "tenant-node writes the tenant kubeconfig without group or other bits under umask 022" {
88+
work="$(mktemp -d)"
89+
stub="$(mktemp -d)"
90+
# The secret read answers a kubeconfig and the first tenant read fails, which
91+
# ends the check right after the fetch with the file in place.
92+
cat > "$stub/kubectl" <<'STUB'
93+
#!/bin/sh
94+
case "$*" in *"get secret"*) printf '%s' YXBpVmVyc2lvbjogdjEK; exit 0 ;; esac
95+
exit 1
96+
STUB
97+
chmod +x "$stub/kubectl"
98+
rc=0
99+
(umask 022; PATH="$stub:$PATH" RUNNER_TEMP="$work" COZY_PROXMOX_WORKDIR= COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s "$RUNNER" tenant-node) >/dev/null 2>&1 || rc=$?
100+
mode="$(ls -l "$work/$KUBECONFIG_NAME" | cut -c1-10)"
101+
rm -rf "$work" "$stub"
102+
[ "$rc" -ne 0 ]
103+
[ "$mode" = "-rw-------" ] || { echo "FAIL: the tenant kubeconfig was written $mode" >&2; exit 1; }
104+
}
105+
106+
@test "apply-tenant removes a tenant kubeconfig an earlier run left behind" {
107+
work="$(mktemp -d)"
108+
stub="$(mktemp -d)"
109+
printf 'an earlier run\n' > "$work/$KUBECONFIG_NAME"
110+
printf '#!/bin/sh\ncat >/dev/null\n' > "$stub/kubectl"
111+
chmod +x "$stub/kubectl"
112+
PATH="$stub:$PATH" RUNNER_TEMP="$work" COZY_PROXMOX_WORKDIR= COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s "$RUNNER" apply-tenant
113+
left=no
114+
[ ! -e "$work/$KUBECONFIG_NAME" ] || left=yes
115+
rm -rf "$work" "$stub"
116+
[ "$left" = no ] || { echo "FAIL: apply-tenant left the earlier run's kubeconfig in place" >&2; exit 1; }
117+
}
118+
119+
@test "the suite's finally removes the tenant kubeconfig, so diagnostics after it prints no tenant section" {
120+
# keep_tenant exits the finally before its deletes, and the kubeconfig has to
121+
# be gone on that path as well. Diagnostics reads whatever is at the path, so
122+
# a file left there would be reported as the next run's tenant.
123+
work="$(mktemp -d)"
124+
stub="$(mktemp -d)"
125+
printf 'this run\n' > "$work/$KUBECONFIG_NAME"
126+
printf '#!/bin/sh\necho "stub $*"\n' > "$stub/kubectl"
127+
printf '#!/bin/sh\nexit 0\n' > "$stub/ssh"
128+
chmod +x "$stub/kubectl" "$stub/ssh"
129+
finally="$(yq -r '.spec.steps[] | select(.name == "replacement-converges-unattended") | .finally[0].script.content' "$SUITE")"
130+
[ -n "$finally" ]
131+
PATH="$stub:$PATH" RUNNER_TEMP="$work" COZY_PROXMOX_WORKDIR= COZY_KEEP_TENANT=true sh -c "$finally" >/dev/null
132+
out="$(PATH="$stub:$PATH" RUNNER_TEMP="$work" COZY_PROXMOX_WORKDIR= COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s "$RUNNER" diagnostics 2>&1)"
133+
left=no
134+
[ ! -e "$work/$KUBECONFIG_NAME" ] || left=yes
135+
rm -rf "$work" "$stub"
136+
[ "$left" = no ] || { echo "FAIL: the suite's finally left the tenant kubeconfig in place" >&2; exit 1; }
137+
case "$out" in *"tenant nodes and pods"*) echo "$out" >&2; exit 1 ;; esac
138+
}

0 commit comments

Comments
 (0)