You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9196b97
Browse filesBrowse the repository at this point in the historyBrowse files
fix(e2e): bound the dataplane capture collector by wall clock (#4640)
## What this PR does
The e2e data-plane collector limited itself by counting pods and
LoadBalancers, but both of its callers stop it by the clock.
`hack/cozytest.sh` gives it 600s and the Chainsaw catch gives it 300s.
One affected pod could already run past the smaller limit, and the
LoadBalancer capture, which runs last, was the part that got lost. I
moved the collector to a wall-clock budget and fixed the other issues
filed against it.
- Every kubectl call now goes through `dp_run`. It cuts the call's
timeout to what is left of `COZY_DATAPLANE_BUDGET` and starts nothing
once the budget is spent. The default is 270s, and `cozytest.sh` passes
570s. The pod section stops 90s early, so the LoadBalancer section
always gets time to start. A call the deadline cut says so, a call cut
by its own bound names that bound, and a refused call says it was not
started and names its section; exec blocks carry that line inside the
block. The node capture and the LoadBalancer probe used to blame an
unanswered lookup or a missing cni-server when the budget stopped a
call. Now they name the budget (#3671).
- A static test fails on any kubectl call that bypasses `dp_run` and
lists the lines. It matches by command position, so a call inside a
double-quoted `$(...)` is still checked (#3694).
- Per-pod and per-LoadBalancer capture files join namespace and name
with `_`, which neither can contain. A second run into the same
directory appends, with the run separator `capture-notes.txt` already
uses (#3679).
- `capture_node` and `capture_lb_node` no longer write their callers'
loop variables. A static test fails when a function assigns a name that
a `read -r` loop or another scope owns (#3675).
- The tests that shrank every timeout to one second now cut only the
read under test. A `timeout` shim decides by argv, so a slow machine
can't cut a read the test needs answered (#3685).
- `cilium-dbg bpf lb list` runs unfiltered in every affected node's
capture, not only for a failing LoadBalancer (#3689).
The call-site comment in `.chainsaw.yaml` and the collector paragraph in
`docs/agents/e2e-testing.md` now match the code. Every behaviour change
has a test in `hack/capture-dataplane.bats` that failed on the code
before it.
Two parts of the issues live elsewhere. The runner stopping a bats file
at its first failing test is #4620, and a conntrack read filtered by
service VIP is #4623.
From #4629 I fixed the items on lines this PR already changes: the node
capture now gets the run separator, a failed speaker log read writes a
note, the endpoint tcpdump says when it falls back to the geneve
interface, and the per-read timeout knobs are validated (a zero there
would otherwise look like a spent budget). The rest of #4629 stays open:
`ovn-lflows.txt` and `lb-speaker-logs.txt` are still replaced on each
run, and two comments still describe older behaviour.
- Closes#3694
- Closes#3671
- Closes#3679
- Closes#3675
- Closes#3685
- Closes#3689
- Refs #4629
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(e2e): bound the failure-path data-plane collector by a wall-clock budget, keep its per-pod and per-LoadBalancer captures from overwriting each other, and capture the cilium service map for ClusterIP failures.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Improvements**
* Data-plane diagnostics now follow a configurable time budget,
reserving time for LoadBalancer captures and stopping operations when
the budget is spent.
* Capture reports distinguish failed, successful, unknown, and
budget-limited checks. LoadBalancers with successful probes are skipped
for heavy capture.
* Repeated captures are preserved together, and limits help control
LoadBalancer diagnostic collection.
* Diagnostics include additional network details from relevant nodes
during probe replay.
* **Documentation**
* Updated guidance covers capture time limits and how unanswered
EndpointSlice reads are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -131,7 +131,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
131
131
- **Every bound that the report's pod collection hits is named in the artifact, not only in the job log.** The pod walk stops at `COZY_REPORT_PODS_MAX` pods or `COZY_REPORT_PODS_BUDGET` seconds and leaves `COLLECTION-TRUNCATED.txt` naming whichever fired; a pod it cut off part way through leaves `CONTAINER-LOGS-TRUNCATED.txt`; a pod list that never came back leaves `COLLECTION-FAILED.txt` rather than the empty tree a healthy cluster produces; a missing local `timeout` leaves `COLLECTION-UNBOUNDED.txt`, because the reads still run, just without a ceiling, and a missing dependency on the machine writing the report is not a statement about the cluster; a kind this cluster does not serve at all — `kubectl get vm -A` on an install without KubeVirt, which exits non-zero — leaves `KIND-NOT-SERVED.txt` rather than a failure marker, since that is the cluster answering correctly and a marker that fires on healthy installs is one a reader learns to skip; and a knob value the collector could not use is named too, appended to `COLLECTION-TRUNCATED.txt` when there is one, written to `COLLECTION-NOTES.txt` when there is not, and appended to `COLLECTION-FAILED.txt` instead when the pod list itself never returned, so the note always sits beside the failure it accompanies rather than in a file the reader has no reason to open, and a bound that turns out to be the default is not mistaken for the bound that was asked for. A pod whose container list came back as a prefix — kubectl named some containers and then failed — leaves `CONTAINER-LIST-INCOMPLETE.txt` beside the logs it did manage to read, because the containers it never named have no file at all and their absence would otherwise read as containers that were silent. A read that succeeded while kubectl also wrote to stderr — a deprecation notice, a partial-result warning — leaves that message in `READ-WARNINGS.txt` beside the object rather than inside it, since `pod.yaml` and the logs are machine-read and a warning line in them costs the parser the file, while dropping it altogether would be the silent loss of evidence this tree exists to prevent. A pod directory whose containers produced any log at all also carries `CONTAINER-LOGS-BOUND.txt`, naming the `--tail` bound those logs were written under (a pod whose every container was silent has no bounded log to describe, so the file is absent), because `--tail` drops the oldest lines and nothing in a log says so; it sits beside the logs rather than inside them, since a log read in full is a complete artifact and the platform's controllers log JSON per line, so a trailing prose line would break every parser that could read the file before. A pod whose container list could not be read leaves `logs-UNAVAILABLE.txt` in its own directory, carrying kubectl's reason, because a pod directory holding `pod.yaml` and `describe.txt` and no logs at all otherwise reads as a pod whose containers were silent. The VM, VMI and services loops carry the same pair of bounds as constants rather than knobs, so a failed install that leaves every LoadBalancer without an address collects the first of them and names the rest instead of running the report step past its own `timeout-minutes: 30`, which would lose the tarball rather than truncate it, since the collector writes it on its last line. That step ceiling is the outer backstop for a read with no bound of its own; it does not save this report, it saves the steps after it, the other uploads included. It carries `continue-on-error: true` alongside, because the step's `|| true` is its standing statement that collecting diagnostics is never fatal and a step killed at its ceiling never reaches that `|| true` — without it a passing run whose collection ran long would be published as a failed one, and every later `failure()` in the job would start meaning "the tests failed or the report was slow". That tolerance also hides the overrun, since the step then renders green, so a following step turns the collect step's `outcome` into a `::warning::` annotation — a bound that fires is recorded, and the outer bound is not exempt from the rule the inner ones follow. The LINSTOR reads that run a command inside a pod carry a ceiling too — they are the only reads whose duration a workload sets rather than the apiserver, and a wedged controller holds the stream open with nothing on either end to close it — and they record a cutoff in `COLLECTION-FAILED.txt` beside their output, which for the two that stream a gzipped tarball is the only place it can go, since a line of prose inside one is an archive nothing opens; the four that produce a table carry the marker in the file as well. A LINSTOR table that got nothing at all — the wedged controller killed before it wrote a byte — says so in the file rather than shipping at zero length, and a read whose stderr could not be captured says that too instead of reporting silence. A LINSTOR table cut short carries the marker inside itself whether the clock or the command ended it, in the same `# [cozyreport]` form every other in-file marker uses so one anchored grep finds them all, and the note about a satellite pod list that came back short lands under `linstor/error-reports/` beside the bundles rather than in the shared Kubernetes file, like every other `.txt` in the tarball, because a short node list whose explanation sits one directory up reads as a small cluster; the two archives cannot take one, since a line of prose inside a gzip stream is a file nothing opens. A cut-off bundle goes one of two ways and the note names both: a stream that got members across is kept and truncated, so a short member list there is the cutoff rather than a satellite with few reports, while a stream killed before its first byte leaves zero bytes and the emptiness check at the call site removes it. The reader runs before that check, so it says which two outcomes are possible instead of asserting one. The four LINSTOR reads whose output is a table keep the remote command's whole message in the file itself when the command produced no table of its own — a partial table plus an error keeps the message beside the file instead, so the table stays a table — because `kubectl exec` appends its own `command terminated with exit code N` last — so the last line there is the one line that never carries the cause, and a listing sent away from its own error is a zero-byte file saying nothing. That note states what the status leaves open instead of resolving it: `kubectl exec` returns the remote command's own exit status, so a 124 there is either the collector's deadline firing or the command itself exiting 124, and nothing separates the two. Every other read in the collector carries a ceiling too, gates included — an `if kubectl get deploy …` that decides whether a module runs is a read, and an unbounded one ends the whole report before the module it guards. A gate cut off short is recorded rather than treated as an answer, because skipping the module produces the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. What a per-read ceiling does **not** give you is a ceiling on the report: the pod, VM, VMI and services walks carry a count cap and a time budget, and every other per-object walk in the collector carries neither, so their cost is the cluster's to set and a failed install with a hundred not-Ready HelmReleases spends four hundred bounded reads there, four per release. Described as "the walks outside those four" rather than listed by name, because a list of them drifts and reads as complete while it does so — which is the defect this same change removed from the collector's own header. The step's `timeout-minutes` is what bounds that, not the per-read wrapper. Whichever bound stopped a walk says so in `COLLECTION-TRUNCATED.txt`, count and clock alike, because objects removed by the clock while the count note stood alone would read as a cap that never fired. The read that feeds each loop is bounded too: the budget cannot start until rows arrive, so an apiserver hanging on the cluster-wide list would otherwise spend the whole collection step with the deadline never consulted. The budget is a deadline for starting a read rather than for finishing the walk, and it is rechecked between containers but not between a container's current and previous log, so the ceiling is the budget plus one pod's object reads plus both log reads of one container, which is the price of never leaving a half-written file behind.
132
132
- **`COZY_REPORT_POD_TAIL` bounds each container log** at 2000 lines by default, and a log that was read successfully records which bound it was written under, since `--tail` drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; `-1` restores kubectl's own default of the whole log. `COZY_REPORT_PREFER_NS` (default `tenant-`) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end.
133
133
- **Capture the PREVIOUS container instance, not just the current one.** For a crash-looping pod `kubectl logs` (and the Chainsaw `podLogs` collector, which in v0.2.15 has no `previous` option — it does accept `name`/`namespace`/`selector`/`container`/`tail`/`timeout`/`cluster`/`clusters`) shows the Nth restart — a fresh replay of the same startup path — while the evidence that explains the failure sits in the immediately preceding instance, reachable only via `--previous` and lost for good once the kubelet garbage-collects that container. `hack/e2e-capture-previous-logs.sh` closes this once for every suite from the global `error.catch` (and from `hack/cozytest.sh` for the e2e BATS suites) rather than per-suite: it discovers containers with `restartCount > 0` at runtime — init containers included, since an interrupted bootstrap lives there — orders the app suites' `tenant-test` namespace first (a best-effort ordering only — it never restricts the capture, and for a suite whose pods live in a child tenant namespace it simply degrades to input order), and dumps each to the job log and to `snapshots/<test>/previous-logs/`. Containers that never restarted are skipped, so a healthy run emits one line, not a page of "previous terminated container not found". Every way the capture can come up short is named rather than inferred: the `COZY_PREVLOG_MAX` overflow, a malformed cap, a `COZY_PREVLOG_TAIL` that was rejected and fell back to 200 (a zero or leading-zero tail asks kubectl for nothing, which would otherwise be reported as the container having logged nothing), the tail bound each kept log was written under — named in `capture-notes.txt` beside the logs rather than inside them, since `--tail` drops the oldest lines and nothing in the output says so while a log read in full is a complete artifact that a trailing prose line would stop any parser from reading, and suppressed on `-1` because no bound was applied then — a pod list that never returned (which is NOT the same as "nothing restarted" and must not be reported as such), a read that timed out versus one kubectl itself refused (kubelet GC, RBAC, an unreachable node — different problems that a blanket "kubelet GC" line would hide), a previous instance that produced no output, and a capture the outer backstop cut short. A partial log is kept and marked truncated, never deleted — it is the only copy of something already gone from the cluster. Each of those lines the script itself emits is written to `snapshots/<test>/previous-logs/capture-notes.txt` as well as to the job log, and the directory is created before the first of them, so a reader who has only the uploaded artifact can still tell a capture that found nothing from one that never ran. The outer backstop firing is the exception: that line comes from the callers, after the script has already been killed, and reaches the job log only. `hack/cozytest.sh` arms this capture, and the other two legs, only for suites whose own filename or whose parent directory carries the `e2e-` prefix, because all three read cluster state and a unit runner has `kubectl` on PATH with no cluster behind it, so a red unit test used to walk into the kubectl-gated legs on its way out, and into the crust-gather one as well on any machine where that binary happened to be installed.
134
-
- **The data-plane collector is the third capture leg, and the only one that reads forwarding state off the node.** `hack/e2e-capture-dataplane.sh` runs after the crust-gather snapshot, from `hack/cozytest.sh` for the e2e BATS suites and from the global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` for the Chainsaw ones, and it collects what an object-state snapshot structurally cannot: the cilium endpoint and BPF conntrack view from an affected pod's node, the kernel conntrack table that is the authoritative one for the host-to-local-pod path under `enable-host-legacy-routing`, and the LoadBalancer datapath. Its two callers bound it differently, 600s from `cozytest.sh` and 300s from the Chainsaw catch, so anything added to what it collects has to fit the smaller of the two rather than the one you happened to test under. It holds the same notes contract as the previous-logs collector, with one stated exception: every read carries its own ceiling, and why a read produced nothing goes to `capture-notes.txt` beside the capture rather than only to the job log, because an empty result and a read that never answered are the same bytes on disk and only the note tells them apart; the two EndpointSlice reads in the LoadBalancer path are the exception and report nothing either way, so a `<none>` backend line there does not distinguish an endpointless Service from a read that never answered.
134
+
- **The data-plane collector is the third capture leg, and the only one that reads forwarding state off the node.** `hack/e2e-capture-dataplane.sh` runs after the crust-gather snapshot, from `hack/cozytest.sh` for the e2e BATS suites and from the global `error.catch` in `hack/e2e-chainsaw/.chainsaw.yaml` for the Chainsaw ones, and it collects what an object-state snapshot structurally cannot: the cilium endpoint and BPF conntrack view from an affected pod's node, the kernel conntrack table that is the authoritative one for the host-to-local-pod path under `enable-host-legacy-routing`, and the LoadBalancer datapath. Its two callers bound it differently, 600s from `cozytest.sh` and 300s from the Chainsaw catch, so the collector keeps its own wall-clock budget (`COZY_DATAPLANE_BUDGET`) under each: 270s by default, under the Chainsaw one, and 570s passed by `cozytest.sh`. Every call is cut to what is left, nothing starts once it is spent, and the last 90s are kept for the LoadBalancer section. It holds the same notes contract as the previous-logs collector: every read carries its own ceiling, and why a read produced nothing goes to `capture-notes.txt` beside the capture rather than only to the job log, because an empty result and a read that never answered are the same bytes on disk and only the note tells them apart. The two EndpointSlice reads in the LoadBalancer path follow it too, and write `<unknown>` rather than `<none>` when the read did not answer.
135
135
- **A catch script needs an explicit `timeout:` that exceeds the inner budgets it contains.** A Chainsaw `script` op is bounded by `timeouts.exec` (5m in `.chainsaw.yaml`) unless it overrides it, so a catch whose own `timeout -k …` wrappers sum past that is *structurally guaranteed* to hit the ceiling — it reports `SCRIPT | ERROR | signal: killed` and yields little. That kill is strictly worse than an inner `timeout` firing, and worse in a way the process tree makes precise: the ceiling SIGKILLs the **shell process alone**, not its process group. Chainsaw runs the shell under `exec.CommandContext` (`pkg/engine/operations/script/operation.go`), whose default cancel is `Process.Kill()` on the shell PID, so every later step in the script is skipped but any collector the shell already spawned is orphaned rather than killed — it keeps running, and it keeps the script's stdout/stderr pipe open. Because that op wires stdout/stderr to in-process buffers, `cmd.Wait()` does not return until the pipe reaches EOF, which needs every inherited fd closed, so the op does not even come back **at** its ceiling until the orphan exits on its own — the ceiling stops being a reliable cut (#4268, where the bucket suite's `verify-every-bucket-takes-a-write` step ran 606s under a 300s ceiling because `mc` kept retrying after the shell was gone). An inner `timeout` instead kills that read at a known budget, exits 124, leaves a partial capture on disk, and lets the rest of the catch run — which is why every read must carry its own cap rather than lean on the op ceiling. Size the op timeout above the sum of the inner budgets and let the inner ones be what fires.
136
136
- **When the sum cannot fit the op, bound the collection phase as a phase rather than raising the op.** The rule above assumes the inner budgets add up to something an op timeout can sit above. On a failure path that has accumulated collectors it stops being true — the tenant-Kubernetes node-join path in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` sums past what is left of its 67m op once the bringup ahead of it and the snapshot behind it are counted, and several of its reads carry only `--request-timeout`, which bounds one HTTP request rather than a client retrying against a wedged apiserver, so parts of it have no ceiling to add up in the first place. Raising the op ceiling does not fix that shape, it postpones it: a larger ceiling multiplied by a sum that grows with every collector added is the same problem one collector later. What works instead is a wall-clock budget for the whole phase (`COZY_DIAG_PHASE_BUDGET`, with `cozy_diag_phase_start` / `cozy_diag_phase_has_time`): once it is spent, the collectors that have not started are declined out loud instead of attempted, so whatever must run after the phase — the tenant crust-gather snapshot, in that case — is never queued behind work begun past the deadline. Two consequences worth carrying to the next collector. The budget bounds when a collector may *start*, not when the phase ends, so it is not a worst case and the comment next to it must not be written as one. And because what runs last is what gets declined, source order becomes load-bearing: put the cheap reads that discriminate between failure modes ahead of the collectors that cost minutes, or the budget will spend itself on the expensive ones and drop the evidence the path exists to produce.
137
137
- **Set `crust-gather --duration` explicitly.** It is crust-gather's own budget for the collection phase and it defaults to **60s**; on elapse it abandons the collection and skips its finish step, leaving a partial tree with nothing to say the rest is missing. The outer wall-clock must exceed it, because the API discovery crust-gather runs *first* is not covered by that budget at all — with an unhealthy aggregated APIService (the state a failed e2e is usually in) aggregated discovery times out and the fallback re-walks every group, which can consume the entire outer timeout before a single object is written. Do not diagnose a killed snapshot by bumping the outer number alone; the two budgets bound different phases. Send the collector's output to a log inside the snapshot rather than `/dev/null` so "complete or truncated?" is answerable from the artifact.
0 commit comments