You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 365195d
Browse filesBrowse the repository at this point in the historyBrowse files
docs(agents): record both E2E test-selection escalation rules
The test-impact section listed only the path-pattern escalation and
described a package whose suite does not exist as "not selected", when
it in fact runs the whole suite. That second rule is the one deciding
most real pull requests, so a maintainer could not explain from the
docs why a small change ran everything.
Write it down: coverage is decided per changed path, one uncovered path
escalates the whole run, and a directory under hack/e2e-chainsaw/ with
no chainsaw-test.yaml defines no suite and is ignored. Also point at
the suite-name mapping a new package has to satisfy to stay narrow.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -67,12 +67,13 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
67
67
68
68
`hack/select-e2e.sh`walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`.
69
69
70
-
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or the E2E workflows escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite.
70
+
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or the E2E workflows escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a **switched-off** suite, whose `chainsaw-test.yaml` has been renamed to `chainsaw-test.yaml.disabled` (the `hack/e2e-chainsaw/backup/` demo), defines no runnable suite and its files are ignored entirely. Any other directory under `hack/e2e-chainsaw/` that carries no `chainsaw-test.yaml` — a shared fixtures directory, or a suite nested deeper than the discovery scan looks — escalates to the **full suite** instead, because Chainsaw discovers suites recursively and selecting nothing for it would skip E2E.
71
+
- Second escalation rule, and in practice the one that fires most: a changed package under `packages/{apps,system,extra}/` whose dependency graph reaches no existing Chainsaw suite runs the **full suite**, because nothing narrower would exercise it at all. Many `packages/system/` components are in this class. Coverage is decided **per changed path**, so a single uncovered path escalates the whole run however narrowly the other paths resolve; deciding it from the merged selection instead would let any other path that contributes a suite name swallow the escalation without a trace. A path owned by several PackageSources counts as covered when **any one** of them reaches a suite — `system/postgres-operator` belongs both to `cozystack.monitoring`, which reaches none, and to `cozystack.postgres-operator`, which reaches postgres and harbor, and a change to it selects those two rather than escalating.
71
72
- The `full-e2e` PR label forces the whole suite.
72
73
- The selection is passed to `make test-chainsaw CHAINSAW_SUITES="<names>"`, which runs `chainsaw test <names>` from `hack/e2e-chainsaw/` (empty = the whole suite).
73
74
- The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the **full** suite — so the release PR exercises the whole suite against the commit the tag will point at.
74
75
75
-
When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist will not be selected — add `hack/e2e-chainsaw/<app>/chainsaw-test.yaml` first.
76
+
When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist is not quietly skipped — it reaches no suite, so every diff touching it runs the full suite until `hack/e2e-chainsaw/<app>/chainsaw-test.yaml` is in place. Suite names come from `src_to_suites()`, the inverse of `select-install.sh`'s `suite_to_source()`: a source named neither `<suite>-application` nor `<suite>` needs an explicit entry in both, or its suite becomes unreachable and every change to its package escalates.
0 commit comments