Skip to content

Commit 9ac67d7

Browse files
authored
test(tests): preserve Bats failure reports during fixture cleanup (#3848)
## What this PR does Test-level `EXIT` cleanup overrides the Bats TAP handler, so a failed assertion can disappear from test results. This moves cleanup in eight unit files after their assertions. Failed tests keep scratch files for inspection. Prerequisite for [the Bats runner change](#3849). Existing subshell traps in the OpenAPI and CAPK tests remain supported. Closes #3623 ### Validation `make unit-tests test-controllers -j4 --output-sync=target` passed. The converted files passed under Bats 1.14.0 and `dash`. A deliberately failed assertion produces `not ok` after this change, and the trap audit rejects an added test-level handler. ### Downstream repositories Test cleanup and its documentation only. No downstream consumer changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ```
2 parents 15f57be + 10727c0 commit 9ac67d7

9 files changed

Lines changed: 163 additions & 100 deletions

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
6666

6767
- Chainsaw deletes the resources it `apply`-ed during its cleanup phase (bounded by the `delete`/`cleanup` timeouts in `hack/e2e-chainsaw/.chainsaw.yaml`). Do not hand-roll teardown for resources Chainsaw created.
6868
- A self-contained `trap '… ' EXIT` **inside a single `script` step** — to kill a port-forward or remove a temp dir — is fine, because it runs in a contained subprocess with its variables in scope. See `hack/e2e-chainsaw/bucket/chainsaw-test.yaml`. What is banned is test-level trap-based cleanup of the BATS kind. The same carve-out holds inside a BATS `@test` when the trap sits in an explicit subshell — `( … trap "kill $pid" EXIT … )` — because a subshell trap does not replace the one the `bats` binary installs, so a failure inside it still prints its `not ok`. `hack/e2e-test-openapi.bats` relies on this to kill a backgrounded `kubectl proxy`; moving that cleanup to the end of the body would leak a process holding a fixed port rather than fix anything.
69-
- The ban extends to every BATS file under `hack/`, subdirectories and the `e2e-` prefixed ones included, for a second reason worth knowing before you debug one: an `EXIT` trap inside an `@test` body replaces the one the `bats` binary installs for its own bookkeeping, and a test that then **fails** prints no TAP line at all. It does not appear as `not ok`; it disappears, and the run ends with `# bats warning: Executed N instead of expected M tests` and a non-zero exit. Verified with Bats 1.14.0. Anyone reading the tail of the output, or grepping it for `not ok`, sees a green suite — and the CI runner `hack/cozytest.sh`, which is not the `bats` binary, reports the same failure correctly, so the two disagree exactly when it matters. Clean up at the end of the test body instead: both runners set `-e`, so the cleanup is unreachable on failure and the scratch directory is left behind for inspection, which is what you want from a failed test anyway. When a suite reports zero failures, confirm it also reports how many tests it ran: an exit code answers "did anything fail", never "did anything run". `hack/bats-no-exit-trap.bats` enforces this across every `hack/**/*.bats`, subdirectories included: a file that carries no `# EXIT-TRAP DEBT: N` comment must contain no EXIT-trap line at all, and a file that carries one must install exactly N — so a trap appearing or disappearing fails until the file's own number is corrected. Note what that does and does not buy: it is a ratchet on the number's *accuracy*, not on the debt itself, because adding a trap and raising `N` in the same change is green. Nothing mechanical stops the count growing — review does, which is the point of the number living in the file being reviewed. A trap inside an explicit subshell is exempt from the ban but still counted, so a declaration is not by itself an admission of debt; `hack/e2e-test-openapi.bats` is the current example and says so in its own header. The scan reads `.bats` files only, so a handler reaching a test body from a sourced `.sh` is outside it — `hack/e2e-chainsaw/_lib/run-kubernetes.sh` installs two, each benign for its own reason rather than by design: the one in `cozy_capture_tenant_talos` because that function is declared with `(` and runs in a subshell, and the one in `run_kubernetes_test` because no `@test` calls it despite being declared with `{`. Seven `hack/*.bats` source that library. Treat the guard as a ratchet over a common spelling, not as proof that a file installs no handler. A handful of files still declare a debt while their conversion waits on the branches that own them; when one of those reports zero failures, reconcile its `1..N` plan against its `ok` count before believing the run.
69+
- The ban extends to every BATS file under `hack/`, subdirectories and the `e2e-` prefixed ones included, for a second reason worth knowing before you debug one: an `EXIT` trap inside an `@test` body replaces the one the `bats` binary installs for its own bookkeeping, and a test that then **fails** prints no TAP line at all. It does not appear as `not ok`; it disappears, and the run ends with `# bats warning: Executed N instead of expected M tests` and a non-zero exit. Verified with Bats 1.14.0. Anyone reading the tail of the output, or grepping it for `not ok`, sees a green suite — and the CI runner `hack/cozytest.sh`, which is not the `bats` binary, reports the same failure correctly, so the two disagree exactly when it matters. Clean up at the last reachable point after an assertion whose failure aborts the test body; the scratch directory then survives for inspection on failure, which is what a failed test wants anyway. A failure in an errexit-exempt `if` or AND-OR context can continue into cleanup and let cleanup supply the test status, so preserve the verdict explicitly in that shape. When a suite reports zero failures, confirm it also reports how many tests it ran: an exit code answers "did anything fail", never "did anything run". `hack/bats-no-exit-trap.bats` enforces this across every `hack/**/*.bats`, subdirectories included: a file that carries no `# EXIT-TRAP DEBT: N` comment must contain no EXIT-trap line at all, and a file that carries one must install exactly N — so a trap appearing or disappearing fails until the file's own number is corrected. Note what that does and does not buy: it is a ratchet on the number's *accuracy*, not on the debt itself, because adding a trap and raising `N` in the same change is green. Nothing mechanical stops the count growing — review does, which is the point of the number living in the file being reviewed. A trap inside an explicit subshell is exempt from the ban but still counted, so a declaration is not by itself an admission of debt; `hack/e2e-test-openapi.bats` and `hack/capk-provider_test.bats` document their exempt subshell traps in their headers. No file declares conversion debt after the test-level handlers are removed; the remaining declarations count those exempt traps. The scan reads `.bats` files only, so a handler reaching a test body from a sourced `.sh` is outside it — `hack/e2e-chainsaw/_lib/run-kubernetes.sh` installs two, each benign for its own reason rather than by design: the one in `cozy_capture_tenant_talos` because that function is declared with `(` and runs in a subshell, and the one in `run_kubernetes_test` because no `@test` calls it despite being declared with `{`. Several `hack/*.bats` files source that library. Treat the guard as a ratchet over a common spelling, not as proof that a file installs no handler. Whenever a suite reports zero failures, reconcile its `1..N` plan against its `ok` count before believing the run.
7070

7171
### 4. Do not mask cleanup or teardown failures
7272

‎hack/build-matrix_test.bats‎

Lines changed: 31 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
#!/usr/bin/env bats
2-
# EXIT-TRAP DEBT: 14 -- see hack/bats-no-exit-trap.bats; lower it as the traps go, delete it at zero.
32
# Unit tests for hack/build-matrix.sh — the CI build-matrix selector.
43
#
54
# Run via hack/cozytest.sh from the repo root (make bats-unit-tests); the
65
# relative `hack/build-matrix.sh` calls below resolve against that cwd. A bats
76
# setup() hook would be dead here — cozytest never invokes it — so the
87
# repo-root cwd is supplied by the runner rather than a setup() cd.
8+
#
9+
# Test-level EXIT traps replace Bats' own handler and hide failing TAP results.
10+
# Cleanup follows aborting assertions; see docs/agents/e2e-testing.md.
911

1012
@test "no argument emits the full matrix" {
1113
out=$(hack/build-matrix.sh)
@@ -38,17 +40,19 @@
3840
}
3941

4042
@test "talos-only diff selects nothing (handled by the dedicated leg)" {
41-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
43+
tmp=$(mktemp)
4244
echo "packages/core/talos/images/matchbox/Dockerfile" > "$tmp"
4345
out=$(hack/build-matrix.sh "$tmp")
4446
[ "$out" = '[]' ]
47+
rm -f "$tmp"
4548
}
4649

4750
@test "installer-only diff selects nothing (handled by finalize)" {
48-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
51+
tmp=$(mktemp)
4952
echo "packages/core/installer/images/cozystack-operator/Dockerfile" > "$tmp"
5053
out=$(hack/build-matrix.sh "$tmp")
5154
[ "$out" = '[]' ]
55+
rm -f "$tmp"
5256
}
5357

5458
@test "FULL sentinel emits the full matrix" {
@@ -57,10 +61,11 @@
5761
}
5862

5963
@test "single-package diff selects only that unit" {
60-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
64+
tmp=$(mktemp)
6165
echo "packages/apps/mariadb/values.yaml" > "$tmp"
6266
out=$(hack/build-matrix.sh "$tmp")
6367
[ "$out" = '["packages/apps/mariadb"]' ]
68+
rm -f "$tmp"
6469
}
6570

6671
@test "kubeovn source patch selects the kubeovn image build" {
@@ -72,84 +77,95 @@
7277
}
7378

7479
@test "two-package diff selects both units" {
75-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
80+
tmp=$(mktemp)
7681
printf 'packages/apps/mariadb/values.yaml\npackages/system/dashboard/values.yaml\n' > "$tmp"
7782
out=$(hack/build-matrix.sh "$tmp")
7883
echo "$out" | grep -q '"packages/apps/mariadb"'
7984
echo "$out" | grep -q '"packages/system/dashboard"'
8085
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -eq 2 ]
86+
rm -f "$tmp"
8187
}
8288

8389
@test "docs-only diff selects nothing" {
84-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
90+
tmp=$(mktemp)
8591
echo "docs/agents/overview.md" > "$tmp"
8692
out=$(hack/build-matrix.sh "$tmp")
8793
[ "$out" = '[]' ]
94+
rm -f "$tmp"
8895
}
8996

9097
@test "a package with no image target selects nothing" {
91-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
98+
tmp=$(mktemp)
9299
# postgres ships no in-repo image build, so it is not a build unit.
93100
echo "packages/apps/postgres/values.yaml" > "$tmp"
94101
out=$(hack/build-matrix.sh "$tmp")
95102
[ "$out" = '[]' ]
103+
rm -f "$tmp"
96104
}
97105

98106
@test "seaweedfs change fans out to objectstorage-controller" {
99-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
107+
tmp=$(mktemp)
100108
echo "packages/system/seaweedfs/values.yaml" > "$tmp"
101109
out=$(hack/build-matrix.sh "$tmp")
102110
[ "$out" = '["packages/system/objectstorage-controller"]' ]
111+
rm -f "$tmp"
103112
}
104113

105114
@test "seaweedfs change does not duplicate an already-selected objectstorage-controller" {
106-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
115+
tmp=$(mktemp)
107116
printf 'packages/system/seaweedfs/values.yaml\npackages/system/objectstorage-controller/values.yaml\n' > "$tmp"
108117
out=$(hack/build-matrix.sh "$tmp")
109118
[ "$out" = '["packages/system/objectstorage-controller"]' ]
119+
rm -f "$tmp"
110120
}
111121

112122
@test "cozy-lib change forces the full matrix" {
113-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
123+
tmp=$(mktemp)
114124
echo "packages/library/cozy-lib/templates/_helpers.tpl" > "$tmp"
115125
out=$(hack/build-matrix.sh "$tmp")
116126
echo "$out" | grep -q '"packages/core/platform"'
117127
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
128+
rm -f "$tmp"
118129
}
119130

120131
@test "common-envs.mk change forces the full matrix" {
121-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
132+
tmp=$(mktemp)
122133
echo "hack/common-envs.mk" > "$tmp"
123134
out=$(hack/build-matrix.sh "$tmp")
124135
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
136+
rm -f "$tmp"
125137
}
126138

127139
@test "go.mod change forces the full matrix" {
128-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
140+
tmp=$(mktemp)
129141
echo "go.mod" > "$tmp"
130142
out=$(hack/build-matrix.sh "$tmp")
131143
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
144+
rm -f "$tmp"
132145
}
133146

134147
@test "build workflow change forces the full matrix" {
135-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
148+
tmp=$(mktemp)
136149
echo ".github/workflows/pull-requests.yaml" > "$tmp"
137150
out=$(hack/build-matrix.sh "$tmp")
138151
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
152+
rm -f "$tmp"
139153
}
140154

141155
@test "root Go source change (api/) forces the full matrix" {
142-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
156+
tmp=$(mktemp)
143157
echo "api/apps/v1alpha1/kubernetes/types.go" > "$tmp"
144158
out=$(hack/build-matrix.sh "$tmp")
145159
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
160+
rm -f "$tmp"
146161
}
147162

148163
@test "root Go source change (pkg/) forces the full matrix" {
149-
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
164+
tmp=$(mktemp)
150165
echo "pkg/cluster/reconciler.go" > "$tmp"
151166
out=$(hack/build-matrix.sh "$tmp")
152167
[ "$(echo "$out" | tr ',' '\n' | wc -l)" -gt 20 ]
168+
rm -f "$tmp"
153169
}
154170

155171
@test "emitted JSON is parseable and matches make build's unit list" {

‎hack/capture-dataplane.bats‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
#!/usr/bin/env bats
2-
# EXIT-TRAP DEBT: 1 -- see hack/bats-no-exit-trap.bats; lower it as the traps go, delete it at zero.
32
# -----------------------------------------------------------------------------
43
# Unit tests for the pure decision/parsing helpers in
54
# hack/e2e-capture-dataplane.sh -- specifically the LoadBalancer-datapath
@@ -38,6 +37,9 @@
3837
# unreachable, so their stubs answer `nc -z` with a failure on purpose. A stub
3938
# that stops short leaves the test green against every implementation.
4039
#
40+
# Test-level EXIT traps replace Bats' own handler and hide failing TAP results.
41+
# Cleanup follows aborting assertions; see docs/agents/e2e-testing.md.
42+
#
4143
# Title syntax constraints (inherited from cozytest.sh's awk parser):
4244
# - Titles delimited by ASCII double quotes; embedded quotes truncate.
4345
# - Only [A-Za-z0-9] from the title survives into the function name, so keep
@@ -205,7 +207,6 @@ E2E_CAPTURE_DATAPLANE_LIB=1
205207

206208
@test "runtime checks LoadBalancers when there are no affected pods" {
207209
tmp=$(mktemp -d)
208-
trap 'rm -rf "$tmp"' EXIT
209210
calls="$tmp/kubectl.calls"
210211
mkdir -p "$tmp/bin"
211212
cat > "$tmp/bin/kubectl" <<'EOF'
@@ -221,6 +222,7 @@ EOF
221222
awk '$0 ~ /^get svc -A / { found = 1 } END { exit !found }' "$calls"
222223
awk 'index($0, "checking LoadBalancers independently") { found = 1 } END { exit !found }' "$tmp/stdout"
223224
awk 'index($0, "no Service type=LoadBalancer") { found = 1 } END { exit !found }' "$tmp/stdout"
225+
rm -rf "$tmp"
224226
}
225227

226228
@test "lb_filter_services keeps only LoadBalancer rows that have an ingress IP" {

0 commit comments

Comments
 (0)