Skip to content

Commit 2af6053

Browse files
themoriartimyasnikovdaniil
themoriarti
authored and
myasnikovdaniil
committed
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. dump_diagnostics then read whatever sat at that path, so a run that failed before the fetch reported the previous run's tenant as its own. The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points at runner.temp. It is written under umask 077, removed at the start of apply-tenant so this run never sees a leftover, removed again in the suite's finally, and once more in the workflow's credential cleanup step for a run killed before finally runs. Closes #4806 Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM (cherry picked from commit 6123850) Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent a6a157d commit 2af6053

3 files changed

Lines changed: 37 additions & 4 deletions

File tree

‎.github/workflows/e2e-proxmox.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,11 @@ jobs:
4949
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
5050
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
5151
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
52+
# Where run-proxmox.sh keeps the tenant's super-admin kubeconfig for the
53+
# run. runner.temp is emptied by the runner between jobs, and the cleanup
54+
# step below removes the file as well; see the script for why neither is
55+
# left to /tmp.
56+
COZY_PROXMOX_WORKDIR: ${{ runner.temp }}
5257
steps:
5358
- name: Checkout
5459
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@@ -157,4 +162,6 @@ jobs:
157162
158163
- name: Remove staged credentials
159164
if: always()
160-
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"
165+
# The tenant kubeconfig is the suite's, fetched by run-proxmox.sh; its
166+
# finally removes it too, but a run killed mid-step never reaches that.
167+
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY" "$COZY_PROXMOX_WORKDIR/proxmox-e2e-tenant.kubeconfig"

‎hack/e2e-chainsaw/_lib/run-proxmox.sh‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,15 @@ CLUSTER="${COZY_PROXMOX_CLUSTER:-k8s-e2e}"
2222
PVE_SSH="${COZY_PVE_SSH:?COZY_PVE_SSH is required: the ssh destination of the Proxmox host, user@host}"
2323
PVE_SSH_KEY="${COZY_PVE_SSH_KEY:-$HOME/.ssh/id_ed25519}"
2424
STORAGE="${COZY_PROXMOX_STORAGE:?COZY_PROXMOX_STORAGE is required, the PVE storage id for the PVC probe}"
25-
TENANT_KUBECONFIG="${TMPDIR:-/tmp}/proxmox-e2e-tenant.kubeconfig"
25+
# Where this run keeps the tenant's super-admin kubeconfig. Per run rather than
26+
# a fixed path under /tmp: the file is a credential for the whole tenant, so it
27+
# must not outlive the run on a shared runner, and dump_diagnostics reads
28+
# whatever is at this path, so a file left by a previous run would be reported
29+
# as this run's tenant. The workflow points this at runner.temp, which the
30+
# runner empties between jobs; apply-tenant removes a leftover before it does
31+
# anything else, and the suite's finally removes the file when the run ends.
32+
WORKDIR="${COZY_PROXMOX_WORKDIR:-${RUNNER_TEMP:-${TMPDIR:-/tmp}}}"
33+
TENANT_KUBECONFIG="$WORKDIR/proxmox-e2e-tenant.kubeconfig"
2634
FIXTURE="$(cd "$(dirname "$0")/.." && pwd)/kubernetes-proxmox/tenant.yaml"
2735

2836
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
@@ -77,6 +85,10 @@ render_tenant() {
7785
}
7886

7987
apply_tenant() {
88+
# First step of the suite, so this is where a kubeconfig left by an earlier
89+
# run on this runner is removed: from here on the path holds this run's file
90+
# or nothing, and dump_diagnostics cannot pick up another tenant.
91+
rm -f "$TENANT_KUBECONFIG"
8092
render_tenant | kubectl apply -f -
8193
}
8294

@@ -85,8 +97,16 @@ apply_tenant() {
8597
# carries the address a runner outside the management cluster can reach; .svc
8698
# is the in-cluster Service the management-side controllers use.
8799
fetch_tenant_kubeconfig() {
88-
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
89-
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
100+
# Written under umask 077 in a subshell of its own: the workflow sets that
101+
# umask only in its staging step, and a chainsaw script step runs with the
102+
# runner's default, under which the file would be readable by every local
103+
# user. No pipefail in POSIX sh, so a failed read leaves an empty file, which
104+
# the size check below turns into a failure rather than an empty kubeconfig.
105+
(
106+
umask 077
107+
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
108+
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
109+
)
90110
[ -s "$TENANT_KUBECONFIG" ] || die "tenant kubeconfig is empty — did the control plane come up?"
91111
}
92112

‎hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,12 @@ spec:
124124
- script:
125125
content: |
126126
set -eu
127+
# The tenant's super-admin kubeconfig that run-proxmox.sh fetched for
128+
# the checks above; the same path the script derives. Removed before
129+
# the keep_tenant exit below, because the file is a credential and
130+
# keeping the tenant is not a reason to keep that on the runner. The
131+
# workflow's cleanup step removes it again, for a run killed here.
132+
rm -f "${COZY_PROXMOX_WORKDIR:-${RUNNER_TEMP:-${TMPDIR:-/tmp}}}/proxmox-e2e-tenant.kubeconfig"
127133
# keep_tenant is honoured here as well as in the workflow's own
128134
# teardown, or the suite would delete what the operator asked to keep.
129135
if [ "${COZY_KEEP_TENANT:-false}" = "true" ]; then

0 commit comments

Comments
 (0)