Skip to content

Commit 3d68d41

Browse files
fix(e2e): validate the leak allowlist against every suite layout (#4972)
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh` accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already see step-file suites and a Test nested below the suite directory (#4630). A suite in one of those layouts could not be put on the leak allowlist at all: validation failed and the run stopped before the first suite. - `lg_suite_has_test` validates an allowlist entry with the rule `hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`, `chainsaw-test.yml`, or a numbered step file, anywhere in or below the suite directory, with the selector's type test (a step-file name counts unless it is a directory, a `chainsaw-test.y*ml` counts whatever its type, so Chainsaw reports a directory by that name); a parked `chainsaw-test.yaml.disabled` matches none. Both sides carry a comment pointing at the other so they move together. - `hack/e2e-chainsaw/README.md` describes the three layouts once, under Layout, and lists `kubernetes-proxmox` as the parked suite. `docs/agents/e2e-testing.md` points at it. - Unit tests: every layout is accepted, including a `chainsaw-test.yaml` that is a directory; a parked, empty, or not-a-step-file directory is refused, including `1a-install.yaml` (passes the `find` glob, not the step-file regex) and a directory named `01-install.yaml`. Left as they are, with the reasons: - `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was removed in 9cc3a9e. - The top-level globs in `hack/e2e-chainsaw-run.sh` are wider than Chainsaw's step-file pattern, but they only refuse a test at the top level and fail closed. - A file under `examples/backups/<other>/` that matches no suite is documented in `select-e2e.sh` as a docs-only demo that stays ignored. Verified with Bats and `hack/cozytest.sh` on `hack/chainsaw-leak-guard_test.bats` (34 tests) and `hack/select-e2e_test.bats` (65 tests; one case, "a path owned by several sources counts as covered if any one is", fails on my machine on main as well, from the local yq build, and does not touch this change). Closes #4659 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Documentation** * Clarified that Chainsaw suites may use `chainsaw-test.yaml` or `.yml`, numbered step files, or either layout in a subdirectory. * Updated guidance on suite selection and noted that disabled test files are not selected or allowlisted. * Identified the parked Kubernetes Proxmox suite and its workflow availability. * **Bug Fixes** * Suite validation now recognizes tests in the documented layouts. * Validation rejects empty directories, parked test files, invalid step-file names, and directories named like step files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: ba0b1ac
2 parents 53a1d2d + 7ec550b commit 3d68d41

5 files changed

Lines changed: 81 additions & 6 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together.
44

5-
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`; §7 lists the other layouts Chainsaw accepts). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
5+
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, usually with a `chainsaw-test.yaml`; the README there describes the layouts a suite can take, and §7 how they are selected). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
66

77
## The core principle
88

‎hack/chainsaw-leak-guard_test.bats‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,47 @@ baseline_of_platform() {
434434
rm -rf "$FIX"
435435
}
436436

437+
@test "an allowlist entry is accepted for every suite layout the selectors see" {
438+
# hack/select-e2e.sh enumerates a suite as a directory in or below which
439+
# Chainsaw finds a Test: chainsaw-test.yaml, chainsaw-test.yml, numbered step
440+
# files, or a Test nested in a subdirectory. The allowlist has to accept the
441+
# same set, or a suite that runs cannot be exempted and the run stops before
442+
# the first suite. That includes a chainsaw-test.y*ml that is a directory:
443+
# the selector keeps it so Chainsaw fails on it loudly, and refusing it here
444+
# would stop the run before Chainsaw could report it.
445+
new_fixture
446+
mkdir -p "$FIX/root/plain" "$FIX/root/yml" "$FIX/root/steps" "$FIX/root/nested/sub" "$FIX/root/dirtest/chainsaw-test.yaml" "$FIX/root/dirtestyml/chainsaw-test.yml"
447+
: >"$FIX/root/plain/chainsaw-test.yaml"
448+
: >"$FIX/root/yml/chainsaw-test.yml"
449+
: >"$FIX/root/steps/01-install.yaml"
450+
: >"$FIX/root/nested/sub/chainsaw-test.yaml"
451+
printf 'plain cozystack/cozystack#1\nyml cozystack/cozystack#1\nsteps cozystack/cozystack#1\nnested cozystack/cozystack#1\ndirtest cozystack/cozystack#1\ndirtestyml cozystack/cozystack#1\n' >"$FIX/allow"
452+
lg lg_allowlist_validate "$FIX/allow" "$FIX/root"
453+
rm -rf "$FIX"
454+
}
455+
456+
@test "a parked, empty or step-file-less suite directory cannot be allowlisted" {
457+
# A chainsaw-test.yaml.disabled is registered nowhere and runs nowhere, so an
458+
# exemption for it would exempt nothing today and a real suite the day the
459+
# suffix goes. A file that only looks like a step file is not one to Chainsaw
460+
# either: install-01.yaml has no digits before the dash, and 1a-install.yaml
461+
# passes the find glob but not the step-file regex behind it. A directory
462+
# with a step-file name is not a step file.
463+
new_fixture
464+
mkdir -p "$FIX/root/parked" "$FIX/root/empty" "$FIX/root/notsteps" "$FIX/root/globonly" "$FIX/root/dirsteps/01-install.yaml"
465+
: >"$FIX/root/parked/chainsaw-test.yaml.disabled"
466+
: >"$FIX/root/notsteps/install-01.yaml"
467+
: >"$FIX/root/globonly/1a-install.yaml"
468+
for suite in parked empty notsteps globonly dirsteps; do
469+
printf '%s cozystack/cozystack#1\n' "$suite" >"$FIX/allow"
470+
if lg lg_allowlist_validate "$FIX/allow" "$FIX/root" 2>/dev/null; then
471+
echo "an allowlist entry for '$suite' was accepted" >&2
472+
return 1
473+
fi
474+
done
475+
rm -rf "$FIX"
476+
}
477+
437478
@test "an allowlist whose last line has no newline still has that line validated" {
438479
new_fixture
439480
mkdir -p "$FIX/root/kafka"

‎hack/e2e-chainsaw/README.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,15 @@ hack/e2e-chainsaw/
1818
└── ...
1919
```
2020

21-
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`.
21+
A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Chainsaw finds a Test. Three layouts do that, and the selector (`hack/select-e2e.sh`) and the leak-guard allowlist both recognise the same three:
22+
23+
- `chainsaw-test.yaml`, the usual one, or `chainsaw-test.yml` when the `.yaml` is absent;
24+
- numbered step files such as `01-install.yaml`, which Chainsaw turns into a Test on its own when neither of the above exists;
25+
- either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.
26+
27+
A parked `chainsaw-test.yaml.disabled` is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.
28+
29+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor.
2230

2331
## What Chainsaw buys over the BATS suite
2432

@@ -30,7 +38,7 @@ Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`,
3038

3139
## Test anatomy
3240

33-
Each suite is a directory with a `chainsaw-test.yaml` (one or more `Test` docs) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
41+
Each suite is a directory with one or more `Test` docs (in one of the layouts above, usually a `chainsaw-test.yaml`) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
3442

3543
Declarative suites assert on `status.conditions` and concrete fields. Inherently imperative suites keep their logic in `script` steps:
3644

‎hack/e2e-chainsaw/_lib/leak-guard.sh‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -175,9 +175,33 @@ lg_allowlisted() {
175175
awk -v s="$1" '$1 !~ /^#/ && $1 == s { print $2; exit }' "$2"
176176
}
177177

178+
# lg_suite_has_test ROOT SUITE — true when ROOT/SUITE is a directory in or
179+
# below which Chainsaw would find a Test: a chainsaw-test.yaml, a
180+
# chainsaw-test.yml, or a numbered step file such as 01-install.yaml. This is
181+
# the rule hack/select-e2e.sh enumerates the suites with (the find and sed that
182+
# build all_apps there), spelled the same way on purpose: a layout one side
183+
# accepts and the other refuses would run a suite the selector never selects,
184+
# or make a suite that runs impossible to allowlist. Change both together. A
185+
# parked chainsaw-test.yaml.disabled matches none of the names. The type test
186+
# is the selector's too: a step-file name counts unless it is a directory, and
187+
# a chainsaw-test.y*ml counts whatever its type, so Chainsaw reports a
188+
# directory by that name instead of this validation stopping the run first.
189+
lg_suite_has_test() {
190+
local root=$1 suite=$2 path
191+
[ -d "$root/$suite" ] || return 1
192+
while IFS= read -r path; do
193+
case ${path##*/} in
194+
chainsaw-test.yaml|chainsaw-test.yml) return 0 ;;
195+
esac
196+
[[ ${path##*/} =~ ^[0-9]+-[^/]*\.ya?ml$ ]] && return 0
197+
done < <(find "$root/$suite" -mindepth 1 \( ! -type d -o -name chainsaw-test.yaml -o -name chainsaw-test.yml \) \( -name chainsaw-test.yaml -o -name chainsaw-test.yml -o -name '[0-9]*-*.yaml' -o -name '[0-9]*-*.yml' \) 2>/dev/null)
198+
return 1
199+
}
200+
178201
# lg_allowlist_validate ALLOWLIST ROOT — every entry must name an existing suite
179202
# directory under ROOT and carry an issue ref, so a renamed or removed suite
180-
# cannot keep an exemption nobody reads.
203+
# cannot keep an exemption nobody reads. A suite directory is one Chainsaw
204+
# would find a Test in, whatever its layout (lg_suite_has_test).
181205
lg_allowlist_validate() {
182206
local list=$1 root=$2 bad=0 suite ref rest
183207
# `|| [ -n "$suite" ]` keeps a last line with no newline, which `read`
@@ -189,8 +213,8 @@ lg_allowlist_validate() {
189213
echo "leak-guard: allowlist line for '$suite' must be '<suite-dir> <issue-ref>'" >&2
190214
bad=1
191215
fi
192-
if [ ! -f "$root/$suite/chainsaw-test.yaml" ] && [ ! -f "$root/$suite/chainsaw-test.yml" ]; then
193-
echo "leak-guard: allowlist names '$suite', which is not a suite directory under $root" >&2
216+
if ! lg_suite_has_test "$root" "$suite"; then
217+
echo "leak-guard: allowlist names '$suite', which is not a suite directory under $root (no chainsaw-test.yaml, chainsaw-test.yml or numbered step file in or below it)" >&2
194218
bad=1
195219
fi
196220
done <"$list"

‎hack/select-e2e.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,8 @@ inert_config_pattern='^(examples/|\.github/|\.claude/|\.gemini/|img/|hack/testda
178178
# and `-type f` would be too narrow. A chainsaw-test.y*ml counts whatever its
179179
# type: Chainsaw stats it and fails loudly on a directory, and leaving that
180180
# suite out would hide the failure instead.
181+
# The leak guard's allowlist accepts a suite by the same rule, in
182+
# lg_suite_has_test (hack/e2e-chainsaw/_lib/leak-guard.sh). Change both together.
181183
#
182184
# Captured before the sed/sort rather than piped straight into them: a pipeline
183185
# carries its LAST command's status, so `$(find ... | sed | sort)` would report

0 commit comments

Comments
 (0)