Skip to content

Commit 61edf33

Browse files
committed
fix(e2e): capture both ends of the worker byte path on node-join failure
A tenant worker that misses the node-Ready deadline while its kubelet image pull crawls leaves two mechanisms behind one symptom, and the artifact could not tell them apart. Either the link is slow, or the pull keeps restarting and discarding what it fetched; from inside the guest both are a progress line that barely moves. Per-worker network counters, read from the kubelet's cAdvisor endpoint the CPU throttling capture already uses, count every byte that crossed into the Pod including the ones a restart threw away, so the gap between that and the progress the guest reported separates the two. Which row carries it is not obvious and getting it wrong inverts the reading, so the collector states it. cAdvisor publishes one row per interface in the Pod netns, and a bridge-bound worker has four: the one still named eth0 is a dummy holding the Pod IP, carries no traffic, and reports zero -- the row a reader reaches for first, where zero reads as a worker that received nothing. The renamed NIC's receive is the number that answers; the bridge counts the same bytes again and the tap is named from the host's end. No interface predicate is applied, so all four reach the artifact and can be told apart. Two further properties of the family fail by capturing nothing rather than loudly: the dropped-packet series are spelled packets_dropped, and every row carries an empty container label because cAdvisor publishes them on the Pod sandbox, so a filter borrowed from the CPU counters beside it matches nothing at all. Both are stated where the filter is. The fixture's rows are real kubelet output relabelled; its interface names are read from the KubeVirt version this repo ships rather than sampled, since no virt-launcher Pod was available. The guest side gains the service state machine and the link table. Talos creates a service's log buffer when its runner opens the log writer, as it starts the process, so a kubelet still waiting on its volumes, on time sync, or on the container runtime reporting healthy never reached its runner, has no log at all, and the capture recorded an rpc error naming a log that was never registered: true about the log, silent about the service, while the state and the awaited condition were the finding. The link table is read as yaml because MTU lives in the spec rather than among the printed columns, and it is the value that decides whether a path-MTU reading of a stalled transfer survives. Both carry a tighter bound than the two reads beside them, being single tabular RPCs rather than streams, and the walk they sit in is the term the phase budget sizes itself against. A worker whose reads all failed now says so once for the worker. Each log already carried its own error and exit code, which is accurate per file and adds up to a directory indistinguishable from a worker that had little to report. The phase budget drops from 8m to 7m to pay for the two added guest reads, because the inequality keeping the tenant snapshot reachable had ten seconds of slack. A shorter budget declines more, and what it declines first is what is gated last: the mirror capture, whose subject is partly recoverable from its own request counts and the host snapshot, and the image-cache re-probe, which has no substitute and was already the first to go. The second cost lands on the guest captures, which are gated after the new counters and reach their gate with both the shortened budget and a listing plus a read per node already spent -- up to 100s at the read bound and the three-node cap, a ceiling those numbers impose rather than a duration measured on a live cluster. Sharing that read would mean the two captures becoming one walk, which is left out for scope rather than merit: the phase gate is a deadline rather than a per-collector allowance, so one walk admitted at the first of the two gates would cost less and collect more. The cost is stated rather than netted away, and the source says which way the argument runs. Both captures run one shared body, which also settles what the two copies had already broken once: the audit that catches an unbounded read pins each collector by something only it produces, and two collectors issuing byte-identical reads left neither with anything of its own, so stubbing one out of the audit went unnoticed. It is pinned by the report directory each writes. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 1c00ddb commit 61edf33

7 files changed

Lines changed: 1670 additions & 297 deletions

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 516 additions & 268 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ spec:
1919
try:
2020
- script:
2121
# ~25m Kamaji bringup + worker join + LB/NFS/ouroboros assertions.
22-
# The on-failure diagnostics phase carries its own 8m wall-clock budget
22+
# The on-failure diagnostics phase carries its own 7m wall-clock budget
2323
# (COZY_DIAG_PHASE_BUDGET in _lib/run-kubernetes.sh): once it is spent, the
2424
# collectors that have not started are declined out loud rather than
2525
# attempted, so the in-process tenant crust-gather snapshot, the one the
@@ -33,6 +33,7 @@ spec:
3333
# than the reads that discriminate between failure modes:
3434
# node-join diagnostics reads
3535
# worker CPU throttling counters
36+
# worker network counters
3637
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3738
# guest Talos capture
3839
# ghcr-mirror state, access log and warm-up Job

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ spec:
1313
- name: tenant-kubernetes-previous-version
1414
try:
1515
- script:
16-
# The on-failure diagnostics phase carries its own 8m wall-clock budget
16+
# The on-failure diagnostics phase carries its own 7m wall-clock budget
1717
# (COZY_DIAG_PHASE_BUDGET in _lib/run-kubernetes.sh): once it is spent, the
1818
# collectors that have not started are declined out loud rather than
1919
# attempted, so the in-process tenant crust-gather snapshot, the one the
@@ -27,6 +27,7 @@ spec:
2727
# than the reads that discriminate between failure modes:
2828
# node-join diagnostics reads
2929
# worker CPU throttling counters
30+
# worker network counters
3031
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
3132
# guest Talos capture
3233
# ghcr-mirror state, access log and warm-up Job

‎hack/run-kubernetes-cpu-throttle_test.bats‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -478,7 +478,15 @@ run_capture() {
478478
# copy of it and falsifies silently; a cap the code carries as a literal can
479479
# be pinned against the code itself, which is what this does.
480480
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
481-
fn=$(awk '/^cozy_capture_tenant_worker_cpu_throttle\(\)/,/^}/' "$lib")
481+
# The walk and the read moved into the body this capture now shares with the
482+
# network counters, so the subject of these assertions is the code the
483+
# collector runs rather than the function that carries its name -- which is
484+
# now a six-line call. Reading only the named function would leave every
485+
# assertion below vacuously true against a body it no longer contains.
486+
fn=$(awk '/^cozy_capture_tenant_worker_cpu_throttle\(\)/,/^}/' "$lib"
487+
awk '/^_cozy_capture_worker_cadvisor\(\)/,/^}/' "$lib"
488+
awk '/^_cozy_cadvisor_node_stream\(\)/,/^}/' "$lib"
489+
awk '/^_cozy_cadvisor_worker_nodes\(\)/,/^}/' "$lib")
482490
case "$fn" in
483491
*'max_nodes=3'*) ;;
484492
*) echo "expected the collector to cap the walk at the 3 nodes the comments state" >&2; return 1 ;;
@@ -1085,7 +1093,7 @@ container_cpu_cfs_throttled_periods_total{container="compute",namespace="tenant-
10851093
# Both empty-capture arms open with "the kubelet answered", so the line above
10861094
# cannot tell them apart on its own. This one is about a node carrying nothing
10871095
# of the namespace, and the sibling arm's wording is what it must not be.
1088-
assert_file_contains 'no CPU series for a tenant-test container' "$capture"
1096+
assert_file_contains 'no CPU series for a tenant-test worker' "$capture"
10891097
assert_file_lacks_pattern 'none of them from a worker Pod' "$capture"
10901098
rm -rf "$tmp"
10911099
}

0 commit comments

Comments
 (0)