You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3d68d41
Browse filesBrowse the repository at this point in the historyBrowse files
fix(e2e): validate the leak allowlist against every suite layout (#4972)
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh`
accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already
see step-file suites and a Test nested below the suite directory
(#4630). A suite in one of those layouts could not be put on the leak
allowlist at all: validation failed and the run stopped before the first
suite.
- `lg_suite_has_test` validates an allowlist entry with the rule
`hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`,
`chainsaw-test.yml`, or a numbered step file, anywhere in or below the
suite directory, with the selector's type test (a step-file name counts
unless it is a directory, a `chainsaw-test.y*ml` counts whatever its
type, so Chainsaw reports a directory by that name); a parked
`chainsaw-test.yaml.disabled` matches none. Both sides carry a comment
pointing at the other so they move together.
- `hack/e2e-chainsaw/README.md` describes the three layouts once, under
Layout, and lists `kubernetes-proxmox` as the parked suite.
`docs/agents/e2e-testing.md` points at it.
- Unit tests: every layout is accepted, including a `chainsaw-test.yaml`
that is a directory; a parked, empty, or not-a-step-file directory is
refused, including `1a-install.yaml` (passes the `find` glob, not the
step-file regex) and a directory named `01-install.yaml`.
Left as they are, with the reasons:
- `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one
parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was
removed in 9cc3a9e.
- The top-level globs in `hack/e2e-chainsaw-run.sh` are wider than
Chainsaw's step-file pattern, but they only refuse a test at the top
level and fail closed.
- A file under `examples/backups/<other>/` that matches no suite is
documented in `select-e2e.sh` as a docs-only demo that stays ignored.
Verified with Bats and `hack/cozytest.sh` on
`hack/chainsaw-leak-guard_test.bats` (34 tests) and
`hack/select-e2e_test.bats` (65 tests; one case, "a path owned by
several sources counts as covered if any one is", fails on my machine on
main as well, from the local yq build, and does not touch this change).
Closes#4659
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Documentation**
* Clarified that Chainsaw suites may use `chainsaw-test.yaml` or `.yml`,
numbered step files, or either layout in a subdirectory.
* Updated guidance on suite selection and noted that disabled test files
are not selected or allowlisted.
* Identified the parked Kubernetes Proxmox suite and its workflow
availability.
* **Bug Fixes**
* Suite validation now recognizes tests in the documented layouts.
* Validation rejects empty directories, parked test files, invalid
step-file names, and directories named like step files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Backported-from: ba0b1ac
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together.
4
4
5
-
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`; §7 lists the other layouts Chainsaw accepts). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
5
+
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, usually with a `chainsaw-test.yaml`; the README there describes the layouts a suite can take, and §7 how they are selected). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Chainsaw finds a Test. Three layouts do that, and the selector (`hack/select-e2e.sh`) and the leak-guard allowlist both recognise the same three:
22
+
23
+
-`chainsaw-test.yaml`, the usual one, or `chainsaw-test.yml` when the `.yaml` is absent;
24
+
- numbered step files such as `01-install.yaml`, which Chainsaw turns into a Test on its own when neither of the above exists;
25
+
- either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.
26
+
27
+
A parked `chainsaw-test.yaml.disabled` is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.
28
+
29
+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor.
Each suite is a directory with a `chainsaw-test.yaml` (one or more `Test` docs) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
41
+
Each suite is a directory with one or more `Test` docs (in one of the layouts above, usually a `chainsaw-test.yaml`) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
34
42
35
43
Declarative suites assert on `status.conditions` and concrete fields. Inherently imperative suites keep their logic in `script` steps:
echo"leak-guard: allowlist names '$suite', which is not a suite directory under $root">&2
216
+
if! lg_suite_has_test"$root""$suite";then
217
+
echo"leak-guard: allowlist names '$suite', which is not a suite directory under $root (no chainsaw-test.yaml, chainsaw-test.yml or numbered step file in or below it)">&2
0 commit comments