Skip to content

Commit ac92b6b

Browse files
committed
fix(e2e): cap the bucket verify S3 reads so the step honors its timeout
A Chainsaw `script` op runs its shell under exec.CommandContext with stdout and stderr wired to in-process buffers. When the op timeout fires it SIGKILLs the shell process alone, not its descendants, and cmd.Wait() does not return until every process that inherited the pipe fds closes them. An mc that keeps retrying an S3 call after the shell is gone holds the pipe open, so the op runs to mc's own exhaustion instead of stopping at its ceiling. This is why the bucket suite's verify-every-bucket-takes-a-write step ran 606s under a 300s ceiling. verify-s3-access shares that shape, so wrap each S3 and apiserver read in its own `timeout`: a hung read dies at a known budget and the op timeout stays a backstop. The 30s kubectl cap matches the terminal `k` wrapper the redis suite carries. The two `mc cp` PUTs take 45s, the budget verify-every-bucket-takes-a-write already gives a write, because a working PUT can still wait while the master grows a volume for a freshly created bucket; 15s left too little headroom on that grow and could turn a slow-but-healthy write red on an unrelated PR. The 15s cap on the mc reads has no terminal precedent: the backup-access preflight caps `mc cp` at 15s too, but inside a retry loop where a 124 costs one attempt rather than the step, so it is not one. Land each secret's encoded value in a file before base64 -d so a timed-out kubectl read (exit 124) stops the step under set -e rather than masking as an empty-credential mc alias set further down. The readonly-upload deny probe branched on mc directly (if mc cp ...), which suspends set -e, so a timeout there (exit 124) was indistinguishable from the access-denied non-zero the probe expects: a hung deny path would exit 124, take the false branch, and reach a green cleanup, the exact window this change bounds. Capture the status instead: exit 0 is the readonly regression, 124 is a timed-out probe, both fail; any other non-zero is the expected denial. The suite's own catch read (the `kubectl get pods` in namespace cozy-objectstorage-controller) carried neither an op timeout nor an inner cap, so it inherits timeouts.exec (5m) and can hold the pipe the same way; give it a 30s inner cap and a 1m op timeout so the rule the rest of the file follows holds for it too. The .chainsaw.yaml notes still read the op ceiling as a reliable cut, the old SIGKILL as a de-facto cap and a smaller timeout as guaranteeing a kill; correct them, since the shell-only kill plus the pipe wait are why an inner cap on every read is required rather than optional. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 88a7ea0 commit ac92b6b

3 files changed

Lines changed: 102 additions & 23 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
116116
- **`COZY_REPORT_POD_TAIL` bounds each container log** at 2000 lines by default, and a log that was read successfully records which bound it was written under, since `--tail` drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; `-1` restores kubectl's own default of the whole log. `COZY_REPORT_PREFER_NS` (default `tenant-`) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end.
117117
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 has no `previous` option — it does accept `name`/`namespace`/`selector`/`container`/`tail`/`timeout`/`cluster`/`clusters`) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the e2e BATS suites) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a `COZY_PREVLOG_TAIL` that was rejected and fell back to 200 (a zero or leading-zero tail asks kubectl for nothing, which would otherwise be reported as the container having logged nothing), the tail bound each kept log was written under — named in `capture-notes.txt` beside the logs rather than inside them, since `--tail` drops the oldest lines and nothing in the output says so while a log read in full is a complete artifact that a trailing prose line would stop any parser from reading, and suppressed on `-1` because no bound was applied then — a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster. Each of those lines the script itself emits is written to `snapshots/<test>/previous-logs/capture-notes.txt` as well as to the job log, and the directory is created before the first of them, so a reader who has only the uploaded artifact can still tell a capture that found nothing from one that never ran. The outer backstop firing is the exception: that line comes from the callers, after the script has already been killed, and reaches the job log only. `hack/cozytest.sh` arms this capture, and the other two legs, only for suites whose own filename or whose parent directory carries the `e2e-` prefix, because all three read cluster state and a unit runner has `kubectl` on PATH with no cluster behind it, so a red unit test used to walk into the kubectl-gated legs on its way out, and into the crust-gather one as well on any machine where that binary happened to be installed.
118118
- **The data-plane collector is the third capture leg, and the only one that reads forwarding state off the node.** `hack/e2e-capture-dataplane.sh` runs after the crust-gather snapshot, from `hack/cozytest.sh` for the e2e BATS suites and from the global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` for the Chainsaw ones, and it collects what an object-state snapshot structurally cannot: the cilium endpoint and BPF conntrack view from an affected pod's node, the kernel conntrack table that is the authoritative one for the host-to-local-pod path under `enable-host-legacy-routing`, and the LoadBalancer datapath. Its two callers bound it differently, 600s from `cozytest.sh` and 300s from the Chainsaw catch, so anything added to what it collects has to fit the smaller of the two rather than the one you happened to test under. It holds the same notes contract as the previous-logs collector, with one stated exception: every read carries its own ceiling, and why a read produced nothing goes to `capture-notes.txt` beside the capture rather than only to the job log, because an empty result and a read that never answered are the same bytes on disk and only the note tells them apart; the two EndpointSlice reads in the LoadBalancer path are the exception and report nothing either way, so a `<none>` backend line there does not distinguish an endpointless Service from a read that never answered.
119-
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to be killed — it reports `SCRIPT | ERROR | signal: killed` and yields nothing. That kill is strictly worse than an inner `timeout` firing: SIGKILL takes out the process group, so the collector dies mid-write **and** every later step in the script is skipped, whereas an inner `timeout` exits 124, leaves a partial capture on disk, and lets the rest of the catch run. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
119+
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to hit the ceiling — it reports `SCRIPT | ERROR | signal: killed` and yields little. That kill is strictly worse than an inner `timeout` firing, and worse in a way the process tree makes precise: the ceiling SIGKILLs the **shell process alone**, not its process group. Chainsaw runs the shell under `exec.CommandContext` (`pkg/engine/operations/script/operation.go`), whose default cancel is `Process.Kill()` on the shell PID, so every later step in the script is skipped but any collector the shell already spawned is orphaned rather than killed — it keeps running, and it keeps the script's stdout/stderr pipe open. Because that op wires stdout/stderr to in-process buffers, `cmd.Wait()` does not return until the pipe reaches EOF, which needs every inherited fd closed, so the op does not even come back **at** its ceiling until the orphan exits on its own — the ceiling stops being a reliable cut (#4268, where the bucket suite's `verify-every-bucket-takes-a-write` step ran 606s under a 300s ceiling because `mc` kept retrying after the shell was gone). An inner `timeout` instead kills that read at a known budget, exits 124, leaves a partial capture on disk, and lets the rest of the catch run — which is why every read must carry its own cap rather than lean on the op ceiling. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
120120
- **When the sum cannot fit the op, bound the collection phase as a phase rather than raising the op.** The rule above assumes the inner budgets add up to something an op timeout can sit above. On a failure path that has accumulated collectors it stops being true — the tenant-Kubernetes node-join path in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` sums past what is left of its 67m op once the bringup ahead of it and the snapshot behind it are counted, and several of its reads carry only `--request-timeout`, which bounds one HTTP request rather than a client retrying against a wedged apiserver, so parts of it have no ceiling to add up in the first place. Raising the op ceiling does not fix that shape, it postpones it: a larger ceiling multiplied by a sum that grows with every collector added is the same problem one collector later. What works instead is a wall-clock budget for the whole phase (`COZY_DIAG_PHASE_BUDGET`, with `cozy_diag_phase_start` / `cozy_diag_phase_has_time`): once it is spent, the collectors that have not started are declined out loud instead of attempted, so whatever must run after the phase — the tenant crust-gather snapshot, in that case — is never queued behind work begun past the deadline. Two consequences worth carrying to the next collector. The budget bounds when a collector may *start*, not when the phase ends, so it is not a worst case and the comment next to it must not be written as one. And because what runs last is what gets declined, source order becomes load-bearing: put the cheap reads that discriminate between failure modes ahead of the collectors that cost minutes, or the budget will spend itself on the expensive ones and drop the evidence the path exists to produce.
121121
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
122122
- **A collector that only runs on failure answers in one direction.** Its output has no measured value to be read against, so a claim like "this boot was slow" rests on an impression of what a healthy run looks like rather than on a number from one. Where the evidence is destroyed by teardown — the tenant worker's guest serial console goes with the Pod that holds the stream — the baseline has to be taken on the passing run or it does not exist at all, which is why `hack/e2e-chainsaw/_lib/run-kubernetes.sh` captures that console on both paths and records which one it was. A passing-path collector is held to two rules the failure path does not need. It must not fail the suite, since everything the suite exists to prove has already passed by the time it runs — and its own cost counts, not just its exit status, so a collector that costs minutes declines itself out loud when the operation's ceiling no longer has room for what it is about to spend, rather than starting a walk that the ceiling cuts off mid-write. That bounds the collector's contribution and nothing else: work already on the path with no ceiling of its own stays unbounded, and no reserve makes it fit by arithmetic. And what it could not collect **at all** has to reach the **job log**, not only the report, because a green run's artifact is the one nobody downloads. Per-read outcomes stay beside the read in the report, as everywhere else in that collector: a note about one console is only usable next to that console, while a walk that collected nothing is a fact about the run.

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 26 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -45,19 +45,30 @@ spec:
4545
# Every catch script below carries an explicit `timeout:`. A Chainsaw script
4646
# op is bounded by `timeouts.exec` (5m above) unless it overrides it, and the
4747
# inner wall-clock budgets here sum well past that — so without the override
48-
# Chainsaw SIGKILLs the whole script partway through and the catch reports
49-
# `SCRIPT | ERROR | signal: killed` having collected nothing. That kill is
50-
# strictly worse than an inner `timeout` firing: SIGKILL takes out the process
51-
# group, so the collector dies mid-write AND every later step in the script is
52-
# skipped, whereas an inner `timeout` leaves a partial capture on disk and
53-
# lets the remaining steps run. Keep each op's `timeout:` above the sum of the
54-
# inner budgets it contains.
48+
# the op ceiling fires and the catch reports `SCRIPT | ERROR | signal: killed`
49+
# having collected little. That kill is strictly worse than an inner `timeout`
50+
# firing, and worse in a way the process tree makes precise: the ceiling
51+
# SIGKILLs the shell process alone — Chainsaw runs it under exec.CommandContext
52+
# whose default cancel is Process.Kill() on the shell PID, not a process-group
53+
# kill — so every later step in the script is skipped, but any collector the
54+
# shell already spawned is orphaned rather than killed and keeps running. It
55+
# also keeps the script's stdout/stderr pipe open, and cmd.Wait() does not
56+
# return until that pipe closes, so the op does not even come back at its
57+
# ceiling until the orphan exits on its own (#4268). An inner `timeout` instead
58+
# kills that read at a known budget, exits 124, leaves a partial capture on
59+
# disk and lets the remaining steps run — which is why every read below is
60+
# itself wrapped in `timeout`, not left to the op ceiling. Keep each op's
61+
# `timeout:` above the sum of the inner budgets it contains.
5562
#
5663
# Accepted tradeoff, so the arithmetic does not have to be rediscovered: this
57-
# raises the worst-case catch from ~5m per failed test (the old SIGKILL was the
58-
# de-facto cap) to ~19m, tests run sequentially, and the E2E job is capped at
59-
# 215 timeout-minutes. A cascade reddening many suites can therefore spend a
60-
# large share of the job in diagnostics. That is deliberate — the old ceiling
64+
# raises the worst-case catch to ~19m per failed test. The old design had no
65+
# inner caps and leaned on the op ceiling, which does not reliably cut a busy
66+
# script: it SIGKILLs the shell alone, orphaned children hold the pipe open,
67+
# and the op overruns (#4268 measured 606s under a 300s ceiling), so the
68+
# nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run
69+
# sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade
70+
# reddening many suites can therefore spend a large share of the job in
71+
# diagnostics. That is deliberate — the old ceiling
6172
# bought its speed by collecting nothing at all — and it is bounded: both legs
6273
# only approach their ceilings on a genuinely degraded cluster, a systemic
6374
# failure usually reds the install gate and skips the E2E step entirely, and
@@ -66,8 +77,10 @@ spec:
6677
# grace window is consumed by the earlier `Collect report` step, so both
6778
# cozyreport.tgz and the job log are lost (measured on runs 29557003429 and
6879
# 29568562337). If this ever does start truncating runs, bound the number of
69-
# snapshotted tests rather than shrinking these timeouts back to a size that
70-
# guarantees a kill.
80+
# snapshotted tests rather than shrinking these timeouts back down: the op
81+
# ceiling firing does not cleanly cut a busy script (it SIGKILLs the shell
82+
# alone and orphaned children hold the pipe open, so the op overruns), so a
83+
# smaller timeout does not guarantee a kill.
7184
error:
7285
catch:
7386
# Ordered cheapest-and-most-decisive first: if anything downstream overruns,

0 commit comments

Comments
 (0)