Skip to content

Commit ae36143

Browse files
themoriartimyasnikovdaniil
authored andcommitted
fix(e2e): validate the leak allowlist against every suite layout
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the selectors already see step-file suites and a Test nested below the suite directory. A suite in one of those layouts could not be allowlisted at all: the run stopped before the first suite. Validate the allowlist with the rule hack/select-e2e.sh enumerates suites with, spelled the same way on both sides and cross-referenced so they move together. Describe the three layouts once in the e2e-chainsaw README, point the agent doc at it, and list kubernetes-proxmox as the parked suite. Closes #4659 Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM (cherry picked from commit 782e0ab) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
1 parent 53a1d2d commit ae36143

4 files changed

Lines changed: 71 additions & 6 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Guidance for writing, changing, and reviewing Cozystack's end-to-end (E2E) tests and the CI that runs them. Read this **before** touching anything under `hack/e2e-chainsaw/`, the bootstrap/OpenAPI BATS (`hack/e2e-*.bats`), `hack/*.sh` test helpers, either E2E CI workflow (`.github/workflows/pull-requests.yaml` for same-repo pull requests, `.github/workflows/e2e-fork.yaml` for forks), or `packages/core/testing/`. §10 covers how the two split the work and which invariants hold them together.
44

5-
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, each with a `chainsaw-test.yaml`; §7 lists the other layouts Chainsaw accepts). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
5+
The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per app, usually with a `chainsaw-test.yaml`; the README there describes the layouts a suite can take, and §7 how they are selected). Cluster bootstrap (`hack/e2e-install-cozystack.bats`, `hack/e2e-prepare-cluster.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS.
66

77
## The core principle
88

‎hack/chainsaw-leak-guard_test.bats‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,42 @@ baseline_of_platform() {
434434
rm -rf "$FIX"
435435
}
436436

437+
@test "an allowlist entry is accepted for every suite layout the selectors see" {
438+
# hack/select-e2e.sh enumerates a suite as a directory in or below which
439+
# Chainsaw finds a Test: chainsaw-test.yaml, chainsaw-test.yml, numbered step
440+
# files, or a Test nested in a subdirectory. The allowlist has to accept the
441+
# same set, or a suite that runs cannot be exempted and the run stops before
442+
# the first suite.
443+
new_fixture
444+
mkdir -p "$FIX/root/plain" "$FIX/root/yml" "$FIX/root/steps" "$FIX/root/nested/sub"
445+
: >"$FIX/root/plain/chainsaw-test.yaml"
446+
: >"$FIX/root/yml/chainsaw-test.yml"
447+
: >"$FIX/root/steps/01-install.yaml"
448+
: >"$FIX/root/nested/sub/chainsaw-test.yaml"
449+
printf 'plain cozystack/cozystack#1\nyml cozystack/cozystack#1\nsteps cozystack/cozystack#1\nnested cozystack/cozystack#1\n' >"$FIX/allow"
450+
lg lg_allowlist_validate "$FIX/allow" "$FIX/root"
451+
rm -rf "$FIX"
452+
}
453+
454+
@test "a parked, empty or step-file-less suite directory cannot be allowlisted" {
455+
# A chainsaw-test.yaml.disabled is registered nowhere and runs nowhere, so an
456+
# exemption for it would exempt nothing today and a real suite the day the
457+
# suffix goes. A file that only looks like a step file (no digits before the
458+
# dash) is not one to Chainsaw either.
459+
new_fixture
460+
mkdir -p "$FIX/root/parked" "$FIX/root/empty" "$FIX/root/notsteps"
461+
: >"$FIX/root/parked/chainsaw-test.yaml.disabled"
462+
: >"$FIX/root/notsteps/install-01.yaml"
463+
for suite in parked empty notsteps; do
464+
printf '%s cozystack/cozystack#1\n' "$suite" >"$FIX/allow"
465+
if lg lg_allowlist_validate "$FIX/allow" "$FIX/root" 2>/dev/null; then
466+
echo "an allowlist entry for '$suite' was accepted" >&2
467+
return 1
468+
fi
469+
done
470+
rm -rf "$FIX"
471+
}
472+
437473
@test "an allowlist whose last line has no newline still has that line validated" {
438474
new_fixture
439475
mkdir -p "$FIX/root/kafka"

‎hack/e2e-chainsaw/README.md‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,15 @@ hack/e2e-chainsaw/
1818
└── ...
1919
```
2020

21-
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`.
21+
A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Chainsaw finds a Test. Three layouts do that, and the selector (`hack/select-e2e.sh`), the runner (`hack/e2e-chainsaw-run.sh`) and the leak-guard allowlist all recognise the same three:
22+
23+
- `chainsaw-test.yaml`, the usual one, or `chainsaw-test.yml` when the `.yaml` is absent;
24+
- numbered step files such as `01-install.yaml`, which Chainsaw turns into a Test on its own when neither of the above exists;
25+
- either of those in a subdirectory of the suite, since Chainsaw walks every subdirectory of the directory it is handed.
26+
27+
A parked `chainsaw-test.yaml.disabled` is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted.
28+
29+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor.
2230

2331
## What Chainsaw buys over the BATS suite
2432

@@ -30,7 +38,7 @@ Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`,
3038

3139
## Test anatomy
3240

33-
Each suite is a directory with a `chainsaw-test.yaml` (one or more `Test` docs) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
41+
Each suite is a directory with one or more `Test` docs (in one of the layouts above, usually a `chainsaw-test.yaml`) plus the manifests it applies. A step has `try` (operations: `apply`, `assert`, `error`, `script`, `delete`, …), optional `catch` (diagnostics on failure), and optional `finally` (teardown regardless of outcome).
3442

3543
Declarative suites assert on `status.conditions` and concrete fields. Inherently imperative suites keep their logic in `script` steps:
3644

‎hack/e2e-chainsaw/_lib/leak-guard.sh‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -175,9 +175,30 @@ lg_allowlisted() {
175175
awk -v s="$1" '$1 !~ /^#/ && $1 == s { print $2; exit }' "$2"
176176
}
177177

178+
# lg_suite_has_test ROOT SUITE — true when ROOT/SUITE is a directory in or
179+
# below which Chainsaw would find a Test: a chainsaw-test.yaml, a
180+
# chainsaw-test.yml, or a numbered step file such as 01-install.yaml. This is
181+
# the rule hack/select-e2e.sh enumerates the suites with (the find and sed that
182+
# build all_apps there), spelled the same way on purpose: a layout one side
183+
# accepts and the other refuses would run a suite the selector never selects,
184+
# or make a suite that runs impossible to allowlist. Change both together. A
185+
# parked chainsaw-test.yaml.disabled matches none of the names.
186+
lg_suite_has_test() {
187+
local root=$1 suite=$2 path
188+
[ -d "$root/$suite" ] || return 1
189+
while IFS= read -r path; do
190+
case ${path##*/} in
191+
chainsaw-test.yaml|chainsaw-test.yml) return 0 ;;
192+
esac
193+
[[ ${path##*/} =~ ^[0-9]+-[^/]*\.ya?ml$ ]] && return 0
194+
done < <(find "$root/$suite" -mindepth 1 ! -type d \( -name chainsaw-test.yaml -o -name chainsaw-test.yml -o -name '[0-9]*-*.yaml' -o -name '[0-9]*-*.yml' \) 2>/dev/null)
195+
return 1
196+
}
197+
178198
# lg_allowlist_validate ALLOWLIST ROOT — every entry must name an existing suite
179199
# directory under ROOT and carry an issue ref, so a renamed or removed suite
180-
# cannot keep an exemption nobody reads.
200+
# cannot keep an exemption nobody reads. A suite directory is one Chainsaw
201+
# would find a Test in, whatever its layout (lg_suite_has_test).
181202
lg_allowlist_validate() {
182203
local list=$1 root=$2 bad=0 suite ref rest
183204
# `|| [ -n "$suite" ]` keeps a last line with no newline, which `read`
@@ -189,8 +210,8 @@ lg_allowlist_validate() {
189210
echo "leak-guard: allowlist line for '$suite' must be '<suite-dir> <issue-ref>'" >&2
190211
bad=1
191212
fi
192-
if [ ! -f "$root/$suite/chainsaw-test.yaml" ] && [ ! -f "$root/$suite/chainsaw-test.yml" ]; then
193-
echo "leak-guard: allowlist names '$suite', which is not a suite directory under $root" >&2
213+
if ! lg_suite_has_test "$root" "$suite"; then
214+
echo "leak-guard: allowlist names '$suite', which is not a suite directory under $root (no chainsaw-test.yaml, chainsaw-test.yml or numbered step file in or below it)" >&2
194215
bad=1
195216
fi
196217
done <"$list"

0 commit comments

Comments
 (0)