Skip to content

Commit 02fbe76

Browse files
authored
fix(ci): close three blind spots in the e2e test selectors (cozystack#4658)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The e2e selectors miss Chainsaw suites that have no `chainsaw-test.yaml` or `.yml` directly in their directory. Chainsaw v0.2.15 also runs a directory that holds only numbered step files such as `01-install.yaml`, and its discovery walks every subdirectory of the path the runner hands it. A suite in either layout ran only when nothing was selected. No narrowed selection named it, the full-suite list left it out, and `select-install.sh --validate` never asked for its mapping. Now `hack/select-e2e.sh` and `hack/select-install.sh` list a top-level directory as a suite when Chainsaw would load a test from it or from anywhere below it. No suite in the current tree uses either layout, so today's suite list stays the same. A file placed directly in `examples/backups/` (a shared `common.sh`, for example) selected nothing, because it fell through to the inert `examples/` rule. It now escalates to the full suite, same as `examples/backups/_lib/`. `README.md` there stays inert through the `*.md` rule. The `test-chainsaw` comment in `packages/core/testing/Makefile` said Chainsaw discovers every `chainsaw-test.yaml`, which was too narrow. I fixed it, and the matching sentences in `docs/agents/e2e-testing.md` too. Every behaviour change has a bats test that fails against the scripts on main. Known gaps I left for a follow-up, none of them a wrong answer today: - The `full_suite_list` helper in `hack/select-e2e_test.bats` repeats the selector's `find`/`sed`, so a bug in that shared pattern would pass every full-suite assertion. The step-file and nested tests compare against literal lists, which covers the new layouts. - The selectors are a little wider than Chainsaw: a broken `chainsaw-test.y*ml` symlink, or one with no Test in it, still lists its suite. That errs toward running more. - A few test comments describe what used to happen rather than why the rule exists. Closes cozystack#4630 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(ci): the e2e test selectors now see Chainsaw suites made of numbered step files or nested below a suite directory, and run the full suite for an edit to a file placed directly in examples/backups/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * E2E suite discovery recognizes supported manifest and numbered-step files at any depth and maps them to the top-level suite. * Changes to files directly under `examples/backups/` trigger full-suite selection. * Suite selection reports an error if discovery fails or finds no suites. * **Documentation** * Clarified supported suite layouts, disabled suite-file handling, discovery and escalation rules, and per-path coverage guidance. * **Tests** * Added coverage for nested suites, numbered steps, unmapped suite layouts, and backup-path selection. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 9485f91 + 5fde704 commit 02fbe76

6 files changed

Lines changed: 196 additions & 30 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together.
44

5-
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`, or a `chainsaw-test.yml`). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
5+
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`; §7 lists the other layouts Chainsaw accepts). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
66

77
## The core principle
88

@@ -12,7 +12,7 @@ Every convention below follows from that finding: **fail fast, fail loud, make t
1212

1313
## The Proxmox lane
1414

15-
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: suite discovery takes only a `chainsaw-test.yaml` or `chainsaw-test.yml` one directory down, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
15+
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: neither Chainsaw nor the selector counts a `.disabled` file as a test, and nothing else in the directory or below it is one, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
1616

1717
Two properties of that workflow are security boundaries rather than preferences. It is never triggered by `pull_request`, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.
1818

@@ -92,13 +92,13 @@ A parent HelmRelease that hit its wait timeout, uninstalled, and reinstalled is
9292

9393
### 7. Test-Impact Analysis (TIA), default-on
9494

95-
`hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory under `hack/e2e-chainsaw/` containing a `chainsaw-test.yaml`, or a `chainsaw-test.yml`, which Chainsaw loads when the `.yaml` is absent: a suite spelled that way runs, so the selector has to see it or it is never selected.
95+
`hack/select-e2e.sh` walks the `packages/core/platform/sources/*.yaml` dependency graph and runs only the Chainsaw suites affected by a diff. A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Chainsaw would find a Test: a `chainsaw-test.yaml`, or a `chainsaw-test.yml`, which Chainsaw loads when the `.yaml` is absent, or, with neither, numbered step files such as `01-install.yaml`, which Chainsaw turns into a Test on its own. Chainsaw walks every subdirectory of the directory it is handed, so a Test in a top-level directory or anywhere below it makes that directory a suite. Every one of those layouts runs, so the selector has to see it or it is never selected.
9696

97-
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/` and `examples/backups/_lib/` (the place for helpers shared by the backup walkthroughs), the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. The directory name is membership-tested against the suite list on the spot, so a directory holding no suite — shared material beside `_lib/`, or a suite nested deeper than discovery looks — escalates there and then, naming the file. `hack/e2e-apps/<name>.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. The files formerly kept there were wired to nothing after migration, and the path remains deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested the same way. Deferring either test to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason.
97+
- Conservative escalation: edits to `packages/library/`, `packages/core/`, the Go trees (`api/`, `cmd/`, `internal/`, `pkg/`), the codegen under `tools/`, `go.mod`/`go.sum`, top-level `hack/*.sh` helpers, the `hack/e2e-*.bats` the sandbox runs and the `hack/lib/` they source, `hack/*.mk`, `hack/buildkitd.toml`, `hack/e2e-*.yaml`, the shared `hack/e2e-chainsaw/_lib/`, `examples/backups/_lib/` (the place for helpers shared by the backup walkthroughs) and any file directly in `examples/backups/`, the chainsaw config `hack/e2e-chainsaw/.chainsaw.yaml`, the `Makefile`, or any workflow that runs the suite escalate to the **full suite**. Only `pull-requests.yaml` is executed from the pull request's own head, so escalating on the other three buys generic regression coverage rather than coverage of the edit; the trade and its measured cost are written next to `full_suite_pattern` rather than repeated here. A per-suite edit (any file under `hack/e2e-chainsaw/<app>/`) selects **only** that suite, with one exception: a `*.disabled` file selects nothing, because a parked suite is executed by nothing and editing it cannot regress a test. The directory name is membership-tested against the suite list on the spot, so a directory holding no suite — shared material beside `_lib/` — escalates there and then, naming the file. `hack/e2e-apps/<name>.bats` — the pre-Chainsaw per-app BATS suites — maps the same way, off the basename instead of the directory. The files formerly kept there were wired to nothing after migration, and the path remains deliberately mapped rather than marked inert: inert would bake the orphan status into the rule and go quietly wrong the day a lane runs those files again. The basename is membership-tested the same way. Deferring either test to the final intersection instead would leave the verdict to the rest of the diff — alone the selection empties and the backstop escalates, but beside any path that contributed a suite the escalation disappears and the run narrows to that suite in silence, which is the merge-before-escalate shape #3330 removed from the graph walk. Anything else under `hack/e2e-apps/` — a nested path, a non-`.bats` file — is shared material for those suites the way `hack/e2e-chainsaw/_lib/` is for the Chainsaw ones, and escalates for the same reason.
9898
- **Every path is classified, and an unclassified one escalates.** The selector has three selection outcomes — full suite, a scoped set, or nothing — and each is reached by an explicit rule; the fourth outcome in the bullet below is not a selection but a refusal to produce one. Several rules produce the empty one, which is worth knowing before reading an empty selection as "the inert list matched": documentation, dashboards and `*.md` are dropped first wherever they live, a `*.disabled` Chainsaw suite and a `examples/backups/<app>/` edit whose app has no suite are dropped where chainsaw and backup-example paths are matched, and `inert_config_pattern` covers repo meta, agent config, `hack/testdata/`, the helm-unittest fixture charts under `packages/tests/`, the three `.gitattributes` that hold nothing but `linguist-generated` markers, which reach no build, chart or test (enumerated rather than matched by filename, because the justification is what those files contain and the name does not carry it — `.gitattributes` can also set `filter`, `eol`, `working-tree-encoding` and `export-subst`, each of which changes what lands in the working tree and therefore what gets built, so a by-name rule would make such a file inert in silence) and the top-level `hack/*.bats` that are not `hack/e2e-*.bats`. That last one is the unit lane, and the root `Makefile` is the authority on the split: `BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))` keeps every non-E2E file for `make unit-tests`, while `packages/core/testing` runs the E2E-prefixed files, so the sandbox never executes a unit file and no Chainsaw suite can regress from an edit to one. Being inert here does not leave them untested: the `unit-tests` job is gated on the `plan` job's `code` output, which `pull-requests.yaml` computes as "any changed path outside `docs/`" and never from `select-e2e.sh`, so a bats-only pull request still runs the whole unit lane — plus install and the OpenAPI tests, since the `e2e` job also reads `code` rather than the selection. `full_suite_pattern` covers shared build inputs and the e2e harness, and the graph lookup handles `packages/(apps|system|extra)/`. A path matching none of them escalates to the full suite and logs why, because both lanes read an empty selection as "skip Chainsaw" and then post `E2E Tests` green — so a silent default is a green gate with nothing tested (#3392). When that escalation fires for something genuinely inert, add it to `inert_config_pattern`; do not relax the fall-through. The escalation list and the inert list are checked in that order, so a specific escalation beats a broad inert directory.
9999
- **Coverage is decided per changed path.** A path that resolves through the graph but reaches no runnable suite is covered by nothing, so it forces the full run on its own account, whatever the rest of the diff selected. Reading that off the merged selection instead let any other changed path contributing a suite name swallow the escalation, with nothing in the output to record that it happened: change a platform component, adjust the Chainsaw test beside it, and the component's escalation disappeared behind that one suite (#3330). The unit is the path rather than the source, because one path can belong to several PackageSources — `system/postgres-operator` belongs to two, one of which reaches no suite — and deciding per source would escalate on that half and run everything for every change to it.
100100
- **Two sources are excluded from the reverse-dependency walk as install-ordering hubs.** Every `*-application` source depends on `cozystack.cozystack-engine` so its `*-rd` HelmRelease waits for the ApplicationDefinition CRD, and the sources that depend on `cozystack.cozystack-basics` do so to get a namespace or a platform-wide policy in place first. Neither edge says the dependent's behaviour changes when the hub does, so carrying selection across them goes wrong in both directions: through the engine a `cert-manager` change fans out to every app in the tree, and through basics the platform's namespace-and-policy package narrows to whichever single suite happens to land downstream of it — 22 suites down to one on #3426, where the new site-router suite sits under `kubevirt-cdi`, which depends on basics. Dropping the reverse edges leaves both hubs reachable and stops them propagating; a change to either still runs everything, because its own group then reaches no suite and the per-path escalation fires on it. Adding a third one is a real decision, not bookkeeping: it says every existing edge into that source is ordering rather than behaviour, so read them before extending the list.
101-
- **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no `chainsaw-test.yaml` or `chainsaw-test.yml`. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
101+
- **There is a fourth outcome, and it is a failed step rather than a selection.** It fires when the suite list itself is unavailable: `find` cannot list `hack/e2e-chainsaw/`, or that directory holds no suite in any of the layouts above. There is no fallback left to take, because escalating would print an empty list and both lanes read an empty selection as "skip Chainsaw" before posting `E2E Tests` green — the outcome meaning "run everything" would deliver a run of nothing. So the selector names what broke on stderr and exits non-zero; both lanes run the step under `bash -e`, so the job fails and the required check goes red. A red `Select E2E tests` step is a real signal, not a flake: read its stderr line before rerunning. A `yq` failure is deliberately not in this class — what is missing there is the dependency graph, not the suite list, so the selector still has a safe answer and escalates to the full suite. That is conservative but quiet: the only sign is one stderr line inside a step that passes, so a selector running blind on every PR looks exactly like a project that escalates a lot.
102102
- **Every escalation names its cause on stderr.** "Run everything" is the same list of suite names whichever rule produced it, so a reason line is the only thing that distinguishes the branches that can reach it, and without one the answer to "why did this pull request run the whole suite" has to be re-derived by hand from the diff. Four of them used to be silent — the shared `hack/e2e-chainsaw/_lib/` and `.chainsaw.yaml` case, the `full_suite_pattern` match, a `packages/` path no PackageSource claims, and the empty-selection backstop — and `full_suite_pattern` is the commonest cause of a full run by a wide margin, so the usual answer was the one the log never carried. Each line names the path or the sources that caused it, and a unit test asserts each message, because a reason that regresses to silence changes no selection and is invisible to every other test. The backstop is the exception: every rule that selects a suite by name membership-tests it where the path is classified, so nothing reaches the backstop and no test asserts its line; it stays as the guard for a future rule that forgets to. The lines go to **stderr** and must stay there: stdout is the suite list and both lanes parse it, so a reason line on stdout would be read as a suite name.
103103
- **The one path the fall-through cannot catch is a new e2e workflow.** `inert_config_pattern` makes `.github/` inert as a directory, and the workflows that run the suite are exempted from it by name in `full_suite_pattern`. A workflow added under `.github/workflows/` therefore reads as inert until someone adds it to that enumeration, and the fall-through never sees it. That is why the list carries a `rg -l test-chainsaw .github/workflows/` reminder: the enumeration is the guard, not a convenience.
104104
- The `full-e2e` PR label forces the whole suite.

0 commit comments

Comments
 (0)