Skip to content

Commit c5e175a

Browse files
authored
test(e2e): record the runner VM shape and region in the e2e report (#3889)
## What this PR does Records which machine each e2e run landed on, in the job log and in `cozyreport.tgz` under `sandbox-host/runner-identity.txt`: the OCI shape, both region keys, the availability and fault domain, the numbers of the `shapeConfig` sizing object, and the runner VM's own kernel release, online CPU count and processor model. An e2e lane asks for a runner class, not for a machine. A flex shape is filled from whatever capacity the pool has at the time, so one lane can be served by two processor generations, and the region is a preference with a fallback. None of that reached the report, so a red run and a green one could not be split by the hardware they ran on. That is the first thing worth knowing about a flake that some runs have and others do not, and today the answer is nowhere in the artifact. `hack/e2e-runner-identity.sh` takes the read on the runner VM, before the sandbox container is created, because the instance metadata service answers on a link-local address the container has no guaranteed route to. An "unreachable" answered from inside the container would look exactly like a capture that had tried. The call sits in `packages/core/testing/Makefile`'s `apply` target rather than in a workflow step: every e2e lane reaches the sandbox through `make prepare-env`, so one call covers all of them, on the passing path and the failing one, with no per-workflow step to keep in step. `hack/cozyreport.sh` folds the file into the report from inside the container, without going near the endpoint itself. What it holds to: - **The root instance document is never requested.** The keys are fetched one at a time. That document carries the instance and compartment OCIDs, the image OCID, the display name, the hostname and the user-supplied metadata bag, and the report is a public artifact, so asking per key makes the privacy property something the wire settles rather than something a filter has to be trusted for. It also matches where the shape sizing is actually published: `shapeConfig` has its own subpath and is not in the root document at all, so a reader hunting it inside that document would have reported the size of every machine as unknown and blamed the service for it. From the sizing object only the numbers are kept, and only under keys shaped and sized like field names, which holds for keys nobody has seen yet since a size is a number, an identifier is a string, and a key is printed into the record exactly as a value is; that object is refused whole unless it arrives as one complete JSON object, so a truncated answer cannot be read in part and published as a sizing record; and a per-key answer that is not one scalar is named as such rather than printed, because a service answering a whole document to a key request is the one way this design could still publish what it never asked for. - **Never fatal, and never slow.** This runs inside `Prepare environment`, the one retried step in the job, so a diagnostic that failed it would cost a whole run. Every read is taken through a wall-clock ceiling and every failure is written into the record instead: a runner that is not an OCI instance leaves an `imds unreachable` line and an exit status of 0, and a wedged `lscpu` is cut off and named rather than holding the step. Where `timeout` is not on PATH the local reads run unbounded rather than not at all, and the record says so, which is the same trade `hack/cozyreport.sh` makes for its own. Exiting zero is not enough on its own, since a read that never returns never reaches that exit. The first key doubles as the probe and is held to answering the way this service does, so a machine that is not an OCI instance pays one ceiling rather than one per key whether the address is silent or something else is answering on it, and either way the run leaves an `imds unreachable` line to grep for. - **One filename either way.** When there is no record, the report writes that same file holding the reason there is none, because an unidentified runner and an unremarkable one otherwise leave the same absence. The record also goes to the job log, and that copy is what covers the one path where the artifact cannot: a run whose sandbox never came up produces no `cozyreport.tgz` at all, so the log is where the record lives. Tests are in `hack/runner-identity.bats`, 52 cases, green under both runner forms. They cover the sizing-object number reader (a renamed key, non-numeric fields counted rather than printed, a nested value, an escaped quote inside a string value, input that is not JSON), the scalar guard (including a whole document answered to a key request), the ceiling knob including the zero that would remove the bound outright, and the running script: which paths are requested and that the root document is not among them, the probe short-circuit, a single key failing without the service being called unreachable, a partial transfer of the sizing object, the bounded local reads and the cut-off arm, a write that would never return, the record reaching both sinks, and both arms of the report side. Verified against mawk and dash as well as the local defaults, since the runner image is neither gawk nor bash. Context: #3513 ### Screenshots No UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The `hack/` triggers it names are `hack/e2e-prepare-cluster.bats` (ansible-cozystack, talm), `hack/package.mk` and `hack/update-crd.sh` (ccp, external-apps-example) and `hack/upload-assets.sh` (website); none of those is touched. The new script and its suite are new files with no counterpart anywhere, `hack/cozyreport.sh` is not restated downstream, and the `packages/core/testing/Makefile` edit adds a diagnostic to the e2e sandbox bring-up without changing what `make prepare-env` does for a caller. The docs change is a bullet in `docs/agents/`, which no downstream repository mirrors. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): every e2e run now records the runner VM it landed on (OCI shape, both region keys, availability and fault domain, shape sizing numbers, kernel release, CPU count and processor model) in the job log and in cozyreport.tgz, so red and green runs can be compared by hardware ```
2 parents a1f3b54 + b76cb20 commit c5e175a

5 files changed

Lines changed: 2139 additions & 3 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
9797
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The `chainsaw test` run is one step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
9898
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
9999
- `make collect-report` → `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
100+
- **Which machine a run landed on is recorded before the sandbox container exists**, in `sandbox-host/runner-identity.txt`: the OCI shape, both region keys, the availability and fault domain, the numbers of the `shapeConfig` sizing object, and the runner VM's own kernel release, online CPU count and processor model. A lane asks for a runner class rather than for a machine — a flex shape is filled from whatever capacity the pool has, and the region is a preference with a fallback — so without this a red run and a green one cannot be compared by the hardware they ran on, which is the first question a flake that only some runs have will raise. `hack/e2e-runner-identity.sh` takes the read on the runner VM, from `packages/core/testing/Makefile`'s `apply` target, because the metadata service answers on a link-local address the sandbox container has no guaranteed route to, and an "unreachable" answered from in there would look exactly like a capture that had tried. Every e2e lane reaches the sandbox through `make prepare-env`, so the record is taken once per run in each of them, on the passing path as well as the failing one. It goes to the job log as well as to the file, and the log copy is not a convenience: a run whose sandbox never came up has no `cozyreport.tgz` at all, so on that one path the log is the only place the record exists.
101+
- **The keys are fetched one at a time and the root instance document never is.** That document carries the instance and compartment OCIDs, the image OCID, the display name, the hostname and the user-supplied metadata bag, and the report is a public artifact — so asking per key makes the privacy property something the wire settles rather than something a filter has to be trusted for. It also removes a guess: Oracle documents the shape sizing under its own `shapeConfig` subpath and the root document does not carry that object at all, so a reader hunting it inside the root document would have reported the size of every machine as unknown and blamed the service for it. From the sizing object only the numbers are kept, and only under keys shaped like field names, which makes the rule hold for keys nobody has seen yet: a size is a number, an identifier is a string, and a key is printed into the record exactly as a value is. That object is also refused whole unless it arrives as one complete JSON object, so a truncated answer or a top-level array cannot be read in part and published as a sizing record. A per-key answer that is not one scalar is named as such rather than printed, since a service answering a whole document to a key request is the one way the design could still publish what it never asked for. The first key doubles as the probe and is held to answering like this service does, so a machine that is not an OCI instance costs one ceiling rather than one per key whether the address is silent or something else is answering on it, and either way the run leaves an `imds unreachable` line to grep for. Every read is taken through a wall-clock ceiling, because a diagnostic that hangs costs the step its whole budget without ever reaching an exit status; where `timeout` is not on PATH the local reads run unbounded instead of not at all and the record says so in a `[bounds]` line, the same trade `hack/cozyreport.sh` makes for its own reads. None of it is fatal, since this runs inside the one retried step in the job — a runner that is not an OCI instance leaves an `imds unreachable` line and an exit status of 0. The report writes that filename either way, holding the record or the reason there is none, because an unidentified runner and an unremarkable one would otherwise leave the same absence.
100102
- **The report's per-pod evidence, under `kubernetes/pods/` in `cozyreport.tgz`, is selected by the READY column rather than by STATUS** — so a pod left at `0/1 Running` by a readiness probe that never passes is in scope, where before it was not. In scope is not the same as collected: the walk stops at `COZY_REPORT_PODS_MAX` pods or `COZY_REPORT_PODS_BUDGET` seconds, and what it did not reach is named in the artifact rather than dropped in silence, as the next point describes. Selecting on the STATUS column alone drops exactly that pod, and it is a common way for a suite to time out: the replica never joins, its HelmRelease sits in `InProgress`, and the artifact carries no trace of the pod the run died on. Terminal `Succeeded`/`Completed` pods stay excluded, since they legitimately end unready. Widening the selection and capping the walk are only safe together if the widening cannot displace what the uncapped walk always kept, so the collection order splits by evidence class before namespace: every pod that is not `Running` — a crash loop, `Pending`, `Error`, the class the collector took before this — is ordered ahead of every pod that is `Running` but unready, and `COZY_REPORT_PREFER_NS` orders within each of those two classes rather than across the whole selection. Otherwise a failed install with more not-Ready pods than the cap spends it on newly-in-scope tenant pods and drops the container logs of the platform pod that is actually crashing.
101103
- **Every bound that the report's pod collection hits is named in the artifact, not only in the job log.** The pod walk stops at `COZY_REPORT_PODS_MAX` pods or `COZY_REPORT_PODS_BUDGET` seconds and leaves `COLLECTION-TRUNCATED.txt` naming whichever fired; a pod it cut off part way through leaves `CONTAINER-LOGS-TRUNCATED.txt`; a pod list that never came back leaves `COLLECTION-FAILED.txt` rather than the empty tree a healthy cluster produces; a missing local `timeout` leaves `COLLECTION-UNBOUNDED.txt`, because the reads still run, just without a ceiling, and a missing dependency on the machine writing the report is not a statement about the cluster; a kind this cluster does not serve at all — `kubectl get vm -A` on an install without KubeVirt, which exits non-zero — leaves `KIND-NOT-SERVED.txt` rather than a failure marker, since that is the cluster answering correctly and a marker that fires on healthy installs is one a reader learns to skip; and a knob value the collector could not use is named too, appended to `COLLECTION-TRUNCATED.txt` when there is one, written to `COLLECTION-NOTES.txt` when there is not, and appended to `COLLECTION-FAILED.txt` instead when the pod list itself never returned, so the note always sits beside the failure it accompanies rather than in a file the reader has no reason to open, and a bound that turns out to be the default is not mistaken for the bound that was asked for. A pod whose container list came back as a prefix — kubectl named some containers and then failed — leaves `CONTAINER-LIST-INCOMPLETE.txt` beside the logs it did manage to read, because the containers it never named have no file at all and their absence would otherwise read as containers that were silent. A read that succeeded while kubectl also wrote to stderr — a deprecation notice, a partial-result warning — leaves that message in `READ-WARNINGS.txt` beside the object rather than inside it, since `pod.yaml` and the logs are machine-read and a warning line in them costs the parser the file, while dropping it altogether would be the silent loss of evidence this tree exists to prevent. A pod directory whose containers produced any log at all also carries `CONTAINER-LOGS-BOUND.txt`, naming the `--tail` bound those logs were written under (a pod whose every container was silent has no bounded log to describe, so the file is absent), because `--tail` drops the oldest lines and nothing in a log says so; it sits beside the logs rather than inside them, since a log read in full is a complete artifact and the platform's controllers log JSON per line, so a trailing prose line would break every parser that could read the file before. A pod whose container list could not be read leaves `logs-UNAVAILABLE.txt` in its own directory, carrying kubectl's reason, because a pod directory holding `pod.yaml` and `describe.txt` and no logs at all otherwise reads as a pod whose containers were silent. The VM, VMI and services loops carry the same pair of bounds as constants rather than knobs, so a failed install that leaves every LoadBalancer without an address collects the first of them and names the rest instead of running the report step past its own `timeout-minutes: 30`, which would lose the tarball rather than truncate it, since the collector writes it on its last line. That step ceiling is the outer backstop for a read with no bound of its own; it does not save this report, it saves the steps after it, the other uploads included. It carries `continue-on-error: true` alongside, because the step's `|| true` is its standing statement that collecting diagnostics is never fatal and a step killed at its ceiling never reaches that `|| true` — without it a passing run whose collection ran long would be published as a failed one, and every later `failure()` in the job would start meaning "the tests failed or the report was slow". That tolerance also hides the overrun, since the step then renders green, so a following step turns the collect step's `outcome` into a `::warning::` annotation — a bound that fires is recorded, and the outer bound is not exempt from the rule the inner ones follow. The LINSTOR reads that run a command inside a pod carry a ceiling too — they are the only reads whose duration a workload sets rather than the apiserver, and a wedged controller holds the stream open with nothing on either end to close it — and they record a cutoff in `COLLECTION-FAILED.txt` beside their output, which for the two that stream a gzipped tarball is the only place it can go, since a line of prose inside one is an archive nothing opens; the four that produce a table carry the marker in the file as well. A LINSTOR table that got nothing at all — the wedged controller killed before it wrote a byte — says so in the file rather than shipping at zero length, and a read whose stderr could not be captured says that too instead of reporting silence. A LINSTOR table cut short carries the marker inside itself whether the clock or the command ended it, in the same `# [cozyreport]` form every other in-file marker uses so one anchored grep finds them all, and the note about a satellite pod list that came back short lands under `linstor/error-reports/` beside the bundles rather than in the shared Kubernetes file, like every other `.txt` in the tarball, because a short node list whose explanation sits one directory up reads as a small cluster; the two archives cannot take one, since a line of prose inside a gzip stream is a file nothing opens. A cut-off bundle goes one of two ways and the note names both: a stream that got members across is kept and truncated, so a short member list there is the cutoff rather than a satellite with few reports, while a stream killed before its first byte leaves zero bytes and the emptiness check at the call site removes it. The reader runs before that check, so it says which two outcomes are possible instead of asserting one. The four LINSTOR reads whose output is a table keep the remote command's whole message in the file itself when the command produced no table of its own — a partial table plus an error keeps the message beside the file instead, so the table stays a table — because `kubectl exec` appends its own `command terminated with exit code N` last — so the last line there is the one line that never carries the cause, and a listing sent away from its own error is a zero-byte file saying nothing. That note states what the status leaves open instead of resolving it: `kubectl exec` returns the remote command's own exit status, so a 124 there is either the collector's deadline firing or the command itself exiting 124, and nothing separates the two. Every other read in the collector carries a ceiling too, gates included — an `if kubectl get deploy …` that decides whether a module runs is a read, and an unbounded one ends the whole report before the module it guards. A gate cut off short is recorded rather than treated as an answer, because skipping the module produces the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. What a per-read ceiling does **not** give you is a ceiling on the report: the pod, VM, VMI and services walks carry a count cap and a time budget, and every other per-object walk in the collector carries neither, so their cost is the cluster's to set and a failed install with a hundred not-Ready HelmReleases spends four hundred bounded reads there, four per release. Described as "the walks outside those four" rather than listed by name, because a list of them drifts and reads as complete while it does so — which is the defect this same change removed from the collector's own header. The step's `timeout-minutes` is what bounds that, not the per-read wrapper. Whichever bound stopped a walk says so in `COLLECTION-TRUNCATED.txt`, count and clock alike, because objects removed by the clock while the count note stood alone would read as a cap that never fired. The read that feeds each loop is bounded too: the budget cannot start until rows arrive, so an apiserver hanging on the cluster-wide list would otherwise spend the whole collection step with the deadline never consulted. The budget is a deadline for starting a read rather than for finishing the walk, and it is rechecked between containers but not between a container's current and previous log, so the ceiling is the budget plus one pod's object reads plus both log reads of one container, which is the price of never leaving a half-written file behind.
102104
- **`COZY_REPORT_POD_TAIL` bounds each container log** at 2000 lines by default, and a log that was read successfully records which bound it was written under, since `--tail` drops the oldest lines and a file that begins mid-stream looks exactly like a complete one; `-1` restores kubectl's own default of the whole log. `COZY_REPORT_PREFER_NS` (default `tenant-`) is collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform namespaces while the pod the suite died on falls off the end.

0 commit comments

Comments
 (0)