Skip to content

Commit d54e3e4

Browse files
committed
test(e2e): capture worker VM/CSR state on tenant node-join failure
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with fewer than two nodes joined, the failure path dumped only `describe nodes` and `get hr`. That cannot tell apart the two node-join failure sub-modes: a worker VM that never booted (virt-launcher Pending or OOMKilled) versus a VM that booted fine but whose kubelet never registered a Node. The tenant cilium-operator HelmRelease reports "InProgress" identically in both, so the current dump is not enough to diagnose the failure or design a fix. Dump scoped, guarded diagnostics before failing fast: - worker VM/VMI/virt-launcher state on the management cluster, which is what actually splits the two sub-modes; - tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal a boot-but-no-Node stall. In-guest Talos/kubelet capture is deliberately left out: talosctl needs a tenant-cluster talosconfig the runner does not have (the tenant workers use a separate Talos CA from the management sandbox), so the gap is documented in the failure path rather than shipped as a call that would fail mTLS and capture nothing. Every capture is guarded with `|| true` so a capture failure never masks the real exit. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent fc86252 commit d54e3e4

1 file changed

Lines changed: 48 additions & 3 deletions

File tree

‎hack/e2e-apps/run-kubernetes.sh‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -301,9 +301,54 @@ EOF
301301
sleep 5
302302
done
303303
'; then
304-
# Dump debug info and fail fast — no point running LB/NFS tests without Ready nodes
305-
kubectl --kubeconfig "tenantkubeconfig-${test_name}" describe nodes
306-
kubectl -n tenant-test get hr
304+
# Node-join failed: fewer than 2 tenant nodes became Ready inside the 12m
305+
# deadline. Dump scoped diagnostics that split the failure sub-modes, then
306+
# fail fast — no point running LB/NFS tests without Ready nodes.
307+
#
308+
# The tenant's cilium-operator HR reports "InProgress" here purely because
309+
# zero worker Nodes joined, so the HelmRelease condition alone cannot tell
310+
# apart (2a) the worker VM never booted (virt-launcher Pending/OOMKilled)
311+
# from (2b) the VM booted fine but its kubelet never registered a Node
312+
# (Talos/CSR/DNS/routing). The captures below make that distinction legible;
313+
# (2b) is the failure mode a follow-up fix has to target, and it cannot be
314+
# designed without this artifact. Every capture is guarded with `|| true`
315+
# so a capture failure never masks the real `exit 1`.
316+
echo "=== node-join failed: fewer than 2 tenant nodes Ready within 12m — diagnostics follow ==="
317+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" describe nodes || true
318+
kubectl -n tenant-test get hr || true
319+
320+
# (a) Worker VM / VMI / virt-launcher state on the MANAGEMENT cluster. A VMI
321+
# stuck Pending or a virt-launcher pod OOMKilled/Pending is mode 2a; a
322+
# Running+Ready VMI with a healthy virt-launcher is mode 2b. This is the key
323+
# split. Full resource names (not the `vm` alias) to avoid short-name
324+
# ambiguity, matching cozy_wait_tenant_drained above.
325+
echo "=== (a) tenant worker VM/VMI/virt-launcher state (management cluster, ns tenant-test) ==="
326+
kubectl -n tenant-test get virtualmachines.kubevirt.io,virtualmachineinstances.kubevirt.io -o wide || true
327+
kubectl -n tenant-test describe virtualmachineinstances.kubevirt.io || true
328+
kubectl -n tenant-test get pods -l kubevirt.io=virt-launcher -o wide || true
329+
kubectl -n tenant-test describe pods -l kubevirt.io=virt-launcher || true
330+
331+
# (c) Tenant kubelet CSRs + the talos-csr-signer sidecar log. A mode-2b node
332+
# boots but blocks on a kubelet-serving/-client CSR that is never submitted
333+
# or never approved; the pending CSR list (tenant cluster) plus the signer
334+
# sidecar log (in the Kamaji apiserver pod on the management cluster) show
335+
# which side stalled.
336+
echo "=== (c) tenant CSRs + talos-csr-signer sidecar log ==="
337+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" get csr || true
338+
kubectl -n tenant-test logs -l kamaji.clastix.io/name="kubernetes-${test_name}" \
339+
-c talos-csr-signer --tail=200 --prefix || true
340+
341+
# (b) In-guest Talos/kubelet state from the worker VMs is intentionally NOT
342+
# captured here. talosctl needs a client talosconfig for the TENANT cluster,
343+
# and the runner has none: the tenant workers are provisioned with their own
344+
# Talos PKI whose CA differs from the sandbox's /workspace/talosconfig (which
345+
# cozyreport.sh uses to reach the MANAGEMENT nodes only), and the chart
346+
# materialises no tenant client talosconfig Secret. Pointing talosctl at the
347+
# worker IPs with the management talosconfig would just fail mTLS and capture
348+
# nothing, so it is skipped rather than shipped as a misleading no-op. (a) +
349+
# (c) carry the 2a-vs-2b split; adding real in-guest capture later requires
350+
# wiring a tenant talosconfig into the runner first.
351+
echo "=== (b) in-guest Talos/kubelet capture skipped: no tenant talosconfig on the runner (a/c cover the 2a-vs-2b split) ==="
307352
exit 1
308353
fi
309354
kubectl --kubeconfig "tenantkubeconfig-${test_name}" get nodes -o wide

0 commit comments

Comments
 (0)