Skip to content

Commit 365195d

Browse files
committed
docs(agents): record both E2E test-selection escalation rules
The test-impact section listed only the path-pattern escalation and described a package whose suite does not exist as "not selected", when it in fact runs the whole suite. That second rule is the one deciding most real pull requests, so a maintainer could not explain from the docs why a small change ran everything. Write it down: coverage is decided per changed path, one uncovered path escalates the whole run, and a directory under hack/e2e-chainsaw/ with no chainsaw-test.yaml defines no suite and is ignored. Also point at the suite-name mapping a new package has to satisfy to stay narrow. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 61fc80e commit 365195d

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,12 +67,13 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
6767

6868
`hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`.
6969

70-
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or the E2E workflows escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite.
70+
- Conservative escalation: edits to `packages/library/`, `packages/core/`, `api/`, `cmd/`, `internal/`, top-level `hack/*.sh|*.bats` helpers, the shared `hack/e2e-chainsaw/_lib/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or the E2E workflows escalate to the **full suite**. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a **switched-off** suite, whose `chainsaw-test.yaml` has been renamed to `chainsaw-test.yaml.disabled` (the `hack/e2e-chainsaw/backup/` demo), defines no runnable suite and its files are ignored entirely. Any other directory under `hack/e2e-chainsaw/` that carries no `chainsaw-test.yaml` — a shared fixtures directory, or a suite nested deeper than the discovery scan looks — escalates to the **full suite** instead, because Chainsaw discovers suites recursively and selecting nothing for it would skip E2E.
71+
- Second escalation rule, and in practice the one that fires most: a changed package under `packages/{apps,system,extra}/` whose dependency graph reaches no existing Chainsaw suite runs the **full suite**, because nothing narrower would exercise it at all. Many `packages/system/` components are in this class. Coverage is decided **per changed path**, so a single uncovered path escalates the whole run however narrowly the other paths resolve; deciding it from the merged selection instead would let any other path that contributes a suite name swallow the escalation without a trace. A path owned by several PackageSources counts as covered when **any one** of them reaches a suite — `system/postgres-operator` belongs both to `cozystack.monitoring`, which reaches none, and to `cozystack.postgres-operator`, which reaches postgres and harbor, and a change to it selects those two rather than escalating.
7172
- The `full-e2e` PR label forces the whole suite.
7273
- The selection is passed to `make test-chainsaw CHAINSAW_SUITES="<names>"`, which runs `chainsaw test <names>` from `hack/e2e-chainsaw/` (empty = the whole suite).
7374
- The coverage gap TIA opens on PRs is closed at release-cut time: the "Prepare release" commit bakes image digests into `packages/core/` (and other packages), which TIA escalates to the **full** suite — so the release PR exercises the whole suite against the commit the tag will point at.
7475

75-
When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist will not be selected — add `hack/e2e-chainsaw/<app>/chainsaw-test.yaml` first.
76+
When adding a new app package, confirm `select-e2e.sh` maps it correctly (it has a unit test, `hack/select-e2e_test.bats`). A new app whose suite directory does not yet exist is not quietly skipped — it reaches no suite, so every diff touching it runs the full suite until `hack/e2e-chainsaw/<app>/chainsaw-test.yaml` is in place. Suite names come from `src_to_suites()`, the inverse of `select-install.sh`'s `suite_to_source()`: a source named neither `<suite>-application` nor `<suite>` needs an explicit entry in both, or its suite becomes unreachable and every change to its package escalates.
7677

7778
#### What TIA does and does not do
7879

0 commit comments

Comments
 (0)