Skip to content

Commit 7ea4d8d

Browse files
themoriartithemoriarti
committed
test(kubernetes): add the Proxmox e2e suite, parked disabled
Every assertion in it is a bug this integration hit rather than a shape someone imagined: the ProxmoxCluster that never reaches Ready without an IPAM provider, the linked-clone default whose regression costs 10.2 GiB per worker and fails nothing, the providerID formats that leave a Machine unpaired with its Node, the topology label the CSI node plugin dies without, a PVC that is Bound with no disk behind it, and a replacement that deadlocks when the controller which removes departed Nodes is stranded on one. It ships as chainsaw-test.yaml.disabled because suite discovery matches only chainsaw-test.yaml, which makes it unselectable by the lanes that have no hypervisor rather than merely unselected. select-e2e.sh needs no change for that, so it did not get one. The workflow is never triggered by pull_request and never cancels in progress. The first is because a self-hosted runner executes what it checks out while holding a hypervisor token; the second is because a killed run leaks VMs. The helper is not a port of run-kubernetes.sh: that script's diagnostic half is KubeVirt objects that cannot exist here. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent e84f376 commit 7ea4d8d

5 files changed

Lines changed: 476 additions & 0 deletions

File tree

‎.github/workflows/e2e-proxmox.yaml‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
name: E2E (proxmox)
2+
3+
# The Proxmox substrate's own lane, separate from every other e2e workflow for
4+
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
5+
# the oracle-vm-* runners the other lanes use will never have one.
6+
#
7+
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
8+
# whatever the workflow checks out, and this one holds credentials to a
9+
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
10+
# hand both to its author. Manual dispatch and a schedule are the whole trigger
11+
# surface, and both come from someone who already has write access.
12+
on:
13+
workflow_dispatch:
14+
inputs:
15+
keep_tenant:
16+
description: "Leave the tenant in place after the run, for debugging"
17+
type: boolean
18+
default: false
19+
schedule:
20+
# Nightly, offset from the upstream lanes so a shared hypervisor is not
21+
# asked to do two things at once.
22+
- cron: "40 2 * * *"
23+
24+
permissions:
25+
contents: read
26+
27+
concurrency:
28+
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
29+
# over all three. Never cancel in progress: a killed run leaks VMs.
30+
group: e2e-proxmox
31+
cancel-in-progress: false
32+
33+
jobs:
34+
e2e:
35+
name: Proxmox substrate
36+
runs-on: [self-hosted, proxmox]
37+
timeout-minutes: 90
38+
env:
39+
COZY_PROXMOX_NS: tenant-e2e-proxmox
40+
COZY_PROXMOX_CLUSTER: k8s-e2e
41+
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE || 'vm-disks1' }}
42+
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH || '[email protected]' }}
43+
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
44+
steps:
45+
- name: Checkout
46+
uses: actions/checkout@v4
47+
with:
48+
persist-credentials: false
49+
50+
- name: Stage credentials
51+
env:
52+
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
53+
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
54+
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
55+
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
56+
PVE_URL: ${{ vars.PROXMOX_URL }}
57+
PVE_REGION: ${{ vars.PROXMOX_REGION }}
58+
run: |
59+
set -eu
60+
umask 077
61+
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
62+
printf '%s' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
63+
echo "KUBECONFIG=$GITHUB_WORKSPACE/.kubeconfig" >> "$GITHUB_ENV"
64+
65+
# The credentials the tenant's CCM and CSI read. A Secret rather than
66+
# values so no token is ever written into a rendered HelmRelease.
67+
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
68+
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
69+
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
70+
create secret generic proxmox-credentials \
71+
--from-literal=url="$PVE_URL" \
72+
--from-literal=token_id="$PVE_TOKEN_ID" \
73+
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
74+
--from-literal=region="$PVE_REGION" \
75+
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
76+
77+
- name: Enable the suite
78+
# It ships disabled so the normal lanes cannot select it; this is the
79+
# one place that is allowed to turn it on.
80+
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
81+
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
82+
83+
- name: Run the Proxmox suite
84+
run: |
85+
chainsaw test \
86+
--config hack/e2e-chainsaw/.chainsaw.yaml \
87+
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
88+
89+
- name: Tear down
90+
# always(), because the whole point of the teardown is the failing run:
91+
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
92+
# them.
93+
if: always() && !inputs.keep_tenant
94+
run: |
95+
set -eu
96+
kubectl delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
97+
kubectl delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
98+
kubectl delete tenant.apps.cozystack.io e2e-proxmox -n tenant-root --ignore-not-found
99+
100+
- name: Remove staged credentials
101+
if: always()
102+
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"

‎docs/agents/e2e-testing.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,16 @@ The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per a
1010

1111
Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.
1212

13+
## The Proxmox lane
14+
15+
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: suite discovery is `find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml`, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
16+
17+
Two properties of that workflow are security boundaries rather than preferences. It is never triggered by `pull_request`, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.
18+
19+
It does not install Cozystack. Every other lane builds a sandbox per run; this one asserts against an existing cluster and an existing hypervisor, which is a weaker guarantee and is stated in the suite header rather than left to be discovered.
20+
21+
The imperative half lives in `_lib/run-proxmox.sh` and is deliberately not a port of `_lib/run-kubernetes.sh`: that script's diagnostics are virt-launcher Pods, VirtualMachineInstances and guest consoles, none of which exist on an external hypervisor.
22+
1323
## Conventions
1424

1525
### 1. No retries on deterministic steps; retry only pure infrastructure

‎hack/e2e-chainsaw/_lib/run-proxmox.sh‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
#!/bin/sh
2+
# Imperative half of the kubernetes-proxmox suite.
3+
#
4+
# Chainsaw's declarative asserts cover everything that is a Kubernetes object in
5+
# THIS cluster, and the suite uses them for exactly that. What is left needs a
6+
# shell: the hypervisor is reached over SSH and answers in `qm` output, and the
7+
# tenant cluster is reached with a kubeconfig that only exists once its control
8+
# plane is up. Those are the two things below.
9+
#
10+
# Deliberately not a port of _lib/run-kubernetes.sh. That script's diagnostic
11+
# half is virt-launcher Pods, VirtualMachineInstances and guest consoles —
12+
# objects with no counterpart on an external hypervisor — so sharing it would
13+
# mean carrying a thousand lines that can never fire here.
14+
#
15+
# Every subcommand exits non-zero on failure and says which check failed. There
16+
# is no `|| echo` anywhere in this file on purpose: the pattern turns a failed
17+
# command into a green run, and this repository has been bitten by it.
18+
set -eu
19+
20+
NS="${COZY_PROXMOX_NS:-tenant-e2e-proxmox}"
21+
CLUSTER="${COZY_PROXMOX_CLUSTER:-k8s-e2e}"
22+
PVE_SSH="${COZY_PVE_SSH:?COZY_PVE_SSH is required, e.g. root@192.168.20.1}"
23+
PVE_SSH_KEY="${COZY_PVE_SSH_KEY:-$HOME/.ssh/id_ed25519}"
24+
STORAGE="${COZY_PROXMOX_STORAGE:?COZY_PROXMOX_STORAGE is required, the PVE storage id for the PVC probe}"
25+
TENANT_KUBECONFIG="${TMPDIR:-/tmp}/proxmox-e2e-tenant.kubeconfig"
26+
27+
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
28+
die() { printf '[%s] FAILED: %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; exit 1; }
29+
30+
pve() { ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i "$PVE_SSH_KEY" "$PVE_SSH" "$@"; }
31+
32+
# The Machine carries the VM id in the ProxmoxMachine it references. Reading it
33+
# from the cluster rather than guessing keeps this working when a replacement
34+
# lands on a different id.
35+
vmid_of_machine() {
36+
kubectl -n "$NS" get proxmoxmachine -o jsonpath='{.items[0].spec.virtualMachineID}'
37+
}
38+
39+
# --- checks -----------------------------------------------------------------
40+
41+
# A full clone is a copy of the template's disk; a linked clone references it and
42+
# stores only the increment. Nothing fails when this is wrong — the pool comes
43+
# up either way — so the only place it can be caught is here, by the shape of the
44+
# volume name: a linked clone points at `base-<template>-disk-N/`.
45+
check_linked_clone() {
46+
vmid=$(vmid_of_machine)
47+
[ -n "$vmid" ] || die "no ProxmoxMachine with a virtualMachineID in $NS"
48+
log "VM $vmid: checking the boot disk is a linked clone"
49+
50+
bootdisk=$(pve "qm config $vmid" | sed -n 's/^scsi0: *//p')
51+
[ -n "$bootdisk" ] || die "VM $vmid has no scsi0"
52+
53+
case "$bootdisk" in
54+
*:base-*-disk-*/*) log "linked clone confirmed: $bootdisk" ;;
55+
*) die "VM $vmid boot disk is a full copy, not a linked clone: $bootdisk" ;;
56+
esac
57+
}
58+
59+
# The tenant kubeconfig exists only after Kamaji has issued it. Fetching it is
60+
# also the first proof that the control plane came up at all.
61+
fetch_tenant_kubeconfig() {
62+
kubectl -n "$NS" get secret "${CLUSTER}-admin-kubeconfig" \
63+
-o jsonpath='{.data.super-admin\.svc}' | base64 -d > "$TENANT_KUBECONFIG"
64+
[ -s "$TENANT_KUBECONFIG" ] || die "tenant kubeconfig is empty — did the control plane come up?"
65+
}
66+
67+
# providerID and the topology labels are applied by the cloud-controller-manager,
68+
# and the CSI node plugin refuses to start without the labels. Asserting them
69+
# here is what separates "the node joined" from "the node is usable".
70+
check_tenant_node() {
71+
fetch_tenant_kubeconfig
72+
log "checking the tenant node is Ready and initialised by the CCM"
73+
74+
node=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
75+
[ -n "$node" ] || die "tenant cluster has no nodes"
76+
77+
provider=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" -o jsonpath='{.spec.providerID}')
78+
region=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" \
79+
-o jsonpath='{.metadata.labels.topology\.kubernetes\.io/region}')
80+
81+
[ -n "$provider" ] || die "node $node has no providerID — the CCM did not initialise it"
82+
[ -n "$region" ] || die "node $node has no topology region label — the CSI node plugin cannot start"
83+
log "node $node: providerID=$provider region=$region"
84+
}
85+
86+
# Bound is not proof: a StorageClass can bind a volume that never becomes a disk.
87+
# The disk on the hypervisor is the proof, so the check reads `qm config` and
88+
# then removes what it created.
89+
check_pvc_round_trip() {
90+
fetch_tenant_kubeconfig
91+
vmid=$(vmid_of_machine)
92+
log "binding a PVC and looking for its disk on VM $vmid"
93+
94+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl apply -f - <<'EOF'
95+
apiVersion: v1
96+
kind: PersistentVolumeClaim
97+
metadata:
98+
name: e2e-csi-probe
99+
spec:
100+
accessModes: [ReadWriteOnce]
101+
storageClassName: proxmox
102+
resources:
103+
requests:
104+
storage: 1Gi
105+
---
106+
apiVersion: v1
107+
kind: Pod
108+
metadata:
109+
name: e2e-csi-probe
110+
spec:
111+
containers:
112+
- name: c
113+
image: busybox:1.36
114+
command: ["sh", "-c", "sleep 600"]
115+
volumeMounts:
116+
- name: v
117+
mountPath: /data
118+
volumes:
119+
- name: v
120+
persistentVolumeClaim:
121+
claimName: e2e-csi-probe
122+
EOF
123+
124+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl wait --for=condition=Ready pod/e2e-csi-probe --timeout=5m \
125+
|| die "the probe Pod never became Ready — the volume did not attach"
126+
127+
if ! pve "qm config $vmid" | grep -q "^scsi[1-9].*$STORAGE:"; then
128+
pve "qm config $vmid" | sed 's/^/ qm-config: /' >&2
129+
die "PVC bound but no disk is attached to VM $vmid on storage $STORAGE"
130+
fi
131+
log "disk attached on the hypervisor"
132+
133+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pod e2e-csi-probe --wait=true
134+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pvc e2e-csi-probe --wait=true
135+
}
136+
137+
# The regression this exists for: with a blanket toleration the CCM Pod outlived
138+
# the Node it ran on, and since that controller is what removes a departed Node,
139+
# a replacement deadlocked until an operator deleted the Node by hand. "Without
140+
# intervention" is the whole assertion, so nothing here touches the cluster
141+
# beyond the initial delete.
142+
check_replacement_converges() {
143+
deadline=$(( $(date +%s) + 900 ))
144+
old=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}')
145+
[ -n "$old" ] || die "no Machine to replace in $NS"
146+
log "deleting Machine $old and waiting for an unattended replacement"
147+
148+
kubectl -n "$NS" delete machine "$old" --wait=false
149+
150+
while [ "$(date +%s)" -lt "$deadline" ]; do
151+
name=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
152+
phase=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
153+
noderef=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.nodeRef.name}' 2>/dev/null || true)
154+
if [ -n "$name" ] && [ "$name" != "$old" ] && [ "$phase" = "Running" ] && [ -n "$noderef" ]; then
155+
log "replacement $name reached Running with nodeRef $noderef"
156+
return 0
157+
fi
158+
sleep 15
159+
done
160+
161+
kubectl -n "$NS" get machine,machinedeployment -o wide >&2 || true
162+
die "no replacement reached Running with a nodeRef within 15m — the CCM Pod is probably stranded on the departed Node"
163+
}
164+
165+
# Runs from the suite's catch. Cheap reads first, so a truncated dump still
166+
# carries the ones that discriminate between failure modes.
167+
dump_diagnostics() {
168+
log "=== Machines and their infrastructure ==="
169+
kubectl -n "$NS" get machine,proxmoxmachine,machinedeployment,proxmoxcluster -o wide || true
170+
log "=== capmox controller ==="
171+
kubectl -n cozy-cluster-api logs -l control-plane=controller-manager --tail=80 \
172+
--selector=cluster.x-k8s.io/provider=infrastructure-proxmox || true
173+
log "=== VMs on the hypervisor ==="
174+
pve "qm list" || true
175+
if [ -s "$TENANT_KUBECONFIG" ]; then
176+
log "=== tenant nodes and pods ==="
177+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o wide || true
178+
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get pods -A -o wide || true
179+
fi
180+
}
181+
182+
case "${1:-}" in
183+
linked-clone) check_linked_clone ;;
184+
tenant-node) check_tenant_node ;;
185+
pvc-round-trip) check_pvc_round_trip ;;
186+
replacement) check_replacement_converges ;;
187+
diagnostics) dump_diagnostics ;;
188+
*) die "usage: $0 {linked-clone|tenant-node|pvc-round-trip|replacement|diagnostics}" ;;
189+
esac

0 commit comments

Comments
 (0)