Repository navigation
Description
Summary
The node-join failure diagnostics block in hack/e2e-chainsaw/_lib/run-kubernetes.sh (everything under "=== node-join failed: fewer than 2 tenant nodes Ready within 18m ===", line 874 at 605030b) issues about a dozen kubectl calls with neither an outer timeout nor --request-timeout: describe nodes through the tenant kubeconfig, get hr, describe virtualmachineinstances, describe pods -l kubevirt.io=virt-launcher, the importer-pod log loop, the image-cache diagnosis, get csr, and the CSR-signer logs. Verified by count: eleven unbounded calls in the block's first stretch alone, zero bounded ones.
Why it matters
This block runs precisely when the cluster is misbehaving, and the very first call, describe nodes through the tenant kubeconfig, targets the component most likely to be unresponsive in a node-join failure. On a wedged tenant apiserver the block hangs until the Chainsaw op is killed, and everything scheduled after it is lost rather than truncated, including the tenant crust-gather snapshot, the largest artifact the failure path exists to produce. The op-timeout arithmetic in both kubernetes-{latest,previous}/chainsaw-test.yaml comments enumerates the bounded collectors precisely and says nothing about this block, so the stated budget carries an unmeasured term.
The gap has a history that makes it easy to miss: the bounding work that fixed the report collector established the timeout -k idiom, and the newest reads on this failure path are the strictest, while the oldest are unbounded. A skim reads as "bounding is handled here".
Raising the op timeout: 50m is not a fix on its own: a larger ceiling multiplied by an unbounded term is still unbounded.
Fix shape
Give each read the same per-call bound the newer collectors on this path already use, sized so the block's worst case fits inside the op budget with the tenant snapshot still reachable after it, and let a read that fails or times out say so in the log instead of stalling the artifacts that follow.
Activity
fix(e2e): bound the tenant failure diagnostics per read and as a phase (
lexfrei commentedon Sep 23, 2026
Fixed by #3676. Every read in the node-join diagnostics block goes through
cozy_diag_readnow, which bounds each read and the whole phase. Unbounded reads on other failure paths are tracked in #3666.