Skip to content

Commit 527d4ea

Browse files
committed
test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it
Two instruments for the node-join failure the kubernetes-* suites keep hitting. Neither changes what the suites assert. The worker CPU capture now reads container_cpu_usage_seconds_total beside the CFS counters, and reads them twice a fixed interval apart instead of once. The throttled counters say a container met its ceiling and cannot say how much CPU it got, which are the two candidate explanations for a stalled worker: a guest held at a one-core ceiling, and a guest whose vCPU thread is rarely scheduled at all. Both leave throttled periods behind. Every counter involved accumulates from container start, so a single reading is an average over the whole uptime and no rate at all, and the profile under investigation is a guest that freezes for tens of seconds and then runs flat out: the average comes out low, the instantaneous figure comes out at the cap, and only the second separates it from a guest that is simply capped. The sandbox nodes are a new subject. The worker captures describe a cgroup, so a guest that is runnable and never scheduled looks from there like a guest that asked for nothing. Whether the sandbox node itself was given its turn is recorded only in the steal column of its own /proc/stat: no kubelet endpoint carries it, and crust-gather cannot reach it because PodSecurity refuses its node-shell Pod here, so the read goes over the Talos API, which is already how the report reaches those nodes for dmesg. The rows are captured verbatim with their column legend beside them rather than reduced to a percentage, because /proc/stat has no header, steal and guest sit next to each other, and guest is large on a node running VMs and already counted inside user. Both subjects are read on either side of one shared wait, so the pair pays it once. The wait is not the interval either subject's counters span, though: each pass walks its own nodes and the other subject's pass falls between a subject's two readings, and on the run this exists for those passes are the slow part. Each capture therefore stamps both ends of its read, so a rate comes from subtracting stamps rather than from assuming the knob, and the width of those brackets is the uncertainty. That wait is a measurement interval rather than a stand-in for a condition nobody wrote a wait for, so the e2e conventions carve it out by name instead of marking it as a fixed timeout to be replaced. The pair is gated below the serial console rather than above it. Its ceiling is sixteen bounded reads plus the interval, which is most of the phase budget, and admission gates only when a collector may start, so above the console a run that hit those bounds would leave the console never started rather than merely cut short. The console is the only capture that survives a worker which never reached apid, which is the shape this failure usually takes. That overrun is inside the term the budget derivation already allows for one late admission, and a guard holds the pair's ceiling against it. No collector here may leave an empty directory, since an empty directory reads as a healthy node: a missing binary, a missing talosconfig, a listing that never answered, a node with no address and a read that was cut off each land as a named note or a COLLECTION-FAILED.txt. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 3dd142a commit 527d4ea

8 files changed

Lines changed: 1682 additions & 170 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ A Chainsaw `assert` polls until the resource matches its expected shape or the p
3737
3838
- Condition checks use the **filter-as-list** form `(conditions[?type == 'Ready'])`. Chainsaw v0.2.15 throws "field not found" on the indexed `(...)[0]` form.
3939
- Imperative checks that no Kubernetes API condition models — S3 reachability through a port-forward, an external LB HTTP path, MetalLB advertisement — stay in a `script` step. A bare `sleep N` inside such a script is allowed **only** when nothing better models the wait; annotate it with a `TODO(e2e-replace-fixed-timeouts):` comment. The bootstrap BATS (`hack/e2e-install-cozystack.bats`, `hack/e2e-chainsaw/_lib/run-kubernetes.sh`) carries the sanctioned `until kubectl get` exceptions.
40+
- A **measurement interval** is not one of those waits and does not take the marker. The rule above governs a `sleep` standing in for a condition nobody wrote a wait for, and the marker means "replace this once something models it". A gap between two readings of a cumulative counter models nothing and is the quantity itself: the diagnostics phase in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the tenant worker CPU counters and the sandbox nodes' `/proc/stat` on either side of one `COZY_DIAG_RATE_INTERVAL` wait, because a single reading of a counter that accumulates from container start is an average over the whole uptime and no rate at all. There is no event to wait for instead, so a `TODO` there would promise a replacement that cannot exist. Carve-outs are named here rather than judged case by case, and this is the only measurement-interval one.
4041

4142
### 3. Let Chainsaw own cleanup; no test-level EXIT/RETURN traps
4243

@@ -153,7 +154,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
153154

154155
1. No new retry loop unless the step is pure infra (image pull / VM boot / network).
155156
2. Resource readiness uses a Chainsaw `assert` (not an imperative `until kubectl get …; kubectl wait`); condition checks use the filter-as-list form `(conditions[?type == 'Ready'])`.
156-
3. Imperative-only waits live in a `script` step; any bare `sleep` carries a `TODO(e2e-replace-fixed-timeouts):` justification.
157+
3. Imperative-only waits live in a `script` step; any bare `sleep` carries a `TODO(e2e-replace-fixed-timeouts):` justification, unless it is a measurement interval, which §2 carves out by name and which takes no marker because no event can replace it.
157158
4. No test-level `EXIT`/`RETURN` trap — rely on Chainsaw cleanup; a self-contained trap is allowed inside a single `script` step (port-forward / temp dir), and inside an explicit subshell within a BATS `@test`, where it does not displace the runner's own handler. A BATS file that still carries test-level traps declares the count itself in `# EXIT-TRAP DEBT: N`; do not ask for one to be removed without checking that line, because the count is exact in both directions.
158159
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
159160
6. Standard HR-Ready assert timeout is **5–6m**; longer waits (harbor 10m, NFS 10m, VM image pulls, platform-wide install 15m) are justified in-line.

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 450 additions & 72 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,12 @@ spec:
3232
# so that a budget running out drops the collectors that cost minutes rather
3333
# than the reads that discriminate between failure modes:
3434
# node-join diagnostics reads
35-
# worker CPU throttling counters
3635
# worker network counters
3736
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
37+
# worker CPU usage and throttling counters, two samples
38+
# sandbox node CPU time, two samples
39+
# (those two alternate rather than run to completion in turn: both
40+
# take their first reading, one wait passes, both take their second)
3841
# guest Talos capture
3942
# ghcr-mirror state, access log and warm-up Job
4043
# talos-image-cache diagnosis

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,12 @@ spec:
2626
# so that a budget running out drops the collectors that cost minutes rather
2727
# than the reads that discriminate between failure modes:
2828
# node-join diagnostics reads
29-
# worker CPU throttling counters
3029
# worker network counters
3130
# serial-console family (~5m50s at worst, ~3m30s at the pool's minimum)
31+
# worker CPU usage and throttling counters, two samples
32+
# sandbox node CPU time, two samples
33+
# (those two alternate rather than run to completion in turn: both
34+
# take their first reading, one wait passes, both take their second)
3235
# guest Talos capture
3336
# ghcr-mirror state, access log and warm-up Job
3437
# talos-image-cache diagnosis

0 commit comments

Comments
 (0)