You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ede426d
Browse filesBrowse the repository at this point in the historyBrowse files
test(e2e): fail the suite whose cert-manager Secrets outlive its cleanup
cert-manager runs without --enable-certificate-owner-ref, so the Secret a
Certificate writes outlives the Certificate and the release, and a later
release of the same name finds and reuses the private key in it. The leak
guard only looked at pods, so a suite that deleted its app and left those
Secrets behind still passed.
A Secret is charged to a suite when it carries cert-manager's
certificate-name annotation, was created after the suite started, and the
root of its chain was too, where a Secret with no owners of its own is
walked as if its Certificate owned it. A platform Secret re-issued
mid-suite traces back to a Certificate the install created, while one
whose Certificate is gone is charged. Charged Secrets get the same budget
as pods and are reported under their own LEAKED-SECRETS marker, so the pod
markers read as before.
The list is read with a kubernetes.io/tls field selector and cut to
metadata inside the pipe from kubectl, so the keys it carries never reach
the shell.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -127,7 +127,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs
127
127
128
128
- On install failure, the CI step dumps `kubectl get hr -A -o wide`, a `describe` of each non-Ready HR, and sorted events, all under collapsible `::group::` blocks.
129
129
- On E2E failure, the CI step dumps `kubectl get hr -A`, sorted events, and COSI bucket/claim/access readiness columns (those CRDs ship no printer columns, so plain `get` shows only NAME/AGE). The whole suite run is one CI step, so this dump is at run level; Chainsaw's own per-`Test` `catch` blocks add scoped events/describe/podLogs for the failing suite.
130
-
- **A suite whose pods outlive it fails where it leaked, not where the memory ran out.** A suite can pass and still leave its workloads running — brokers of Kafka releases whose application was deleted, a TenantControlPlane stuck on its finalizer — and the first thing to notice used to be a later suite failing on `Insufficient memory`, hours on and under the wrong name. Chainsaw v0.2.15 has no hook that runs after a test's cleanup (its Configuration carries `error.catch`, which runs only on failure), so `hack/e2e-chainsaw-run.sh` runs one `chainsaw test` per suite directory and checks between them, with `hack/e2e-chainsaw/_lib/leak-guard.sh`. A pod is charged to the suite when it is not terminal, did not exist before the suite, and the root of its ownerReferences chain was created at or after the suite started on the apiserver's clock; a platform pod replaced mid-suite still traces back to an object the install created, so it is not charged. Charged pods get the default `timeouts.cleanup` of `.chainsaw.yaml` (5m), counted from when `chainsaw test` returns and so after any per-suite cleanup override has been spent, to disappear; past that the run prints `LEAKED-PODS suite=<dir>` with one line per pod and its root, raises a `leaked-pods` error annotation, and adds a failed `leak-guard` case to `chainsaw-report.xml`. Every suite that leaves nothing behind logs how long it took to clear, and leftovers of a suite that already failed are reported once as a warning, without the wait. The wait has three outcomes, and nothing is retried. A timeout or a spent budget gives the verdict from the last good pod list, including when the re-list after it fails: a leak fails the suite. Any other error on a wait or a re-list, before the budget is spent and before any timeout, stops the waiting and is reported as `LEAKED-PODS suite=<dir> count=N inconclusive="<error>"`, a warning rather than a failure, so an apiserver blip while waiting cannot turn into a leak red. A wait that fails only after running for its whole timeout counts as a timeout whatever kubectl printed, since kubectl reports a deadline that passes before its watch starts as the client's `context deadline exceeded`. An apiserver blip before a suite starts is different: if the reading taken ahead of the suite fails, that suite is not run and is reported as not run, which fails the run. A failed first reading after a suite leaves the suite unchecked, and it fails with a `leak-guard` annotation and a report case saying the cluster could not be read, not that it leaked. A suite with a known, filed leak is listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt` as `<suite-dir> <issue-ref>`; its leak prints as a warning and does not fail. The exemption covers the whole suite, so the PR that fixes the issue deletes the line, and an entry naming a directory that is not a suite stops the run before the first suite.
130
+
- **A suite whose pods outlive it fails where it leaked, not where the memory ran out.** A suite can pass and still leave its workloads running — brokers of Kafka releases whose application was deleted, a TenantControlPlane stuck on its finalizer — and the first thing to notice used to be a later suite failing on `Insufficient memory`, hours on and under the wrong name. Chainsaw v0.2.15 has no hook that runs after a test's cleanup (its Configuration carries `error.catch`, which runs only on failure), so `hack/e2e-chainsaw-run.sh` runs one `chainsaw test` per suite directory and checks between them, with `hack/e2e-chainsaw/_lib/leak-guard.sh`. A pod is charged to the suite when it is not terminal, did not exist before the suite, and the root of its ownerReferences chain was created at or after the suite started on the apiserver's clock; a platform pod replaced mid-suite still traces back to an object the install created, so it is not charged. A Secret that cert-manager wrote, marked by its `cert-manager.io/certificate-name` annotation, is charged by the same rule, because cert-manager runs without `--enable-certificate-owner-ref` and nothing deletes such a Secret with its Certificate (cozystack/cozystack#4704). It must have been created at or after the suite started, and unless it has owners of its own its walk starts at the Certificate the annotation names, so a platform Secret re-issued mid-suite is not charged and one whose Certificate is gone is. Charged pods and Secrets get the default `timeouts.cleanup` of `.chainsaw.yaml` (5m), counted from when `chainsaw test` returns and so after any per-suite cleanup override has been spent, to disappear; past that the run prints `LEAKED-PODS suite=<dir>` with one line per pod and its root, or `LEAKED-SECRETS suite=<dir>` with one line per Secret, raises a `leaked-pods` or `leaked-secrets` error annotation, and adds a failed `leak-guard` case to `chainsaw-report.xml`. Every suite that leaves nothing behind logs how long it took to clear, and leftovers of a suite that already failed are reported once as a warning, without the wait. The wait has three outcomes, and nothing is retried. A timeout or a spent budget gives the verdict from the last good pod list, including when the re-list after it fails: a leak fails the suite. Any other error on a wait or a re-list, before the budget is spent and before any timeout, stops the waiting and is reported as `LEAKED-PODS suite=<dir> count=N inconclusive="<error>"` (or `LEAKED-SECRETS`), a warning rather than a failure, so an apiserver blip while waiting cannot turn into a leak red. A wait that fails only after running for its whole timeout counts as a timeout whatever kubectl printed, since kubectl reports a deadline that passes before its watch starts as the client's `context deadline exceeded`. An apiserver blip before a suite starts is different: if the reading taken ahead of the suite fails, that suite is not run and is reported as not run, which fails the run. A failed first reading after a suite leaves the suite unchecked, and it fails with a `leak-guard` annotation and a report case saying the cluster could not be read, not that it leaked. A suite with a known, filed leak is listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt` as `<suite-dir> <issue-ref>`; its leak prints as a warning and does not fail. The exemption covers the whole suite, so the PR that fixes the issue deletes the line, and an entry naming a directory that is not a suite stops the run before the first suite.
131
131
- Inside a suite, attach scoped diagnostics with a top-level `catch:` block (`events`, `describe`, `podLogs`, `get`) rather than letting a bare assert failure through — see `hack/e2e-chainsaw/bucket/chainsaw-test.yaml` and `hack/e2e-chainsaw/harbor/chainsaw-test.yaml`. This replaces the BATS `... || { echo …; kubectl get/describe …; false; }` idiom.
132
132
- `make collect-report`→ `hack/cozyreport.sh` uploads a full `cozyreport.tgz` (operator/Flux/cert-manager logs, LINSTOR, Kamaji, Talos dmesg, COSI YAML) on every run, and the Chainsaw JUnit report (`chainsaw-report.xml`) is uploaded as its own artifact.
133
133
- **Which machine a run landed on is recorded before the sandbox container exists**, in `sandbox-host/runner-identity.txt`: the OCI shape, both region keys, the availability and fault domain, the numbers of the `shapeConfig` sizing object, and the runner VM's own kernel release, online CPU count and processor model. A lane asks for a runner class rather than for a machine — a flex shape is filled from whatever capacity the pool has, and the region is a preference with a fallback — so without this a red run and a green one cannot be compared by the hardware they ran on, which is the first question a flake that only some runs have will raise. `hack/e2e-runner-identity.sh` takes the read on the runner VM, from `packages/core/testing/Makefile`'s `apply` target, because the metadata service answers on a link-local address the sandbox container has no guaranteed route to, and an "unreachable" answered from in there would look exactly like a capture that had tried. Every e2e lane reaches the sandbox through `make prepare-env`, so the record is taken once per run in each of them, on the passing path as well as the failing one. It goes to the job log as well as to the file, and the log copy is not a convenience: a run whose sandbox never came up has no `cozyreport.tgz` at all, so on that one path the log is the only place the record exists.
0 commit comments