Skip to content

Commit 5137d98

Browse files
committed
test(bats): refuse a negated assertion across every suite
A negated assertion is the natural way to spell "this must not appear", and an absence is what nothing else in a test covers, so the errexit exemption lands where it is least likely to be noticed and invisible in a green run. Switching runners does not lift it: the ERR trap the bats binary reports through carries the same exemption, so neither runner sees a negated assertion that has any statement below it. Add a suite that scans every hack/**/*.bats, subdirectories included, and fails on a statement whose first token is `!` -- naming the file, the line the statement opens on, and the form to use instead. Statement boundaries are the POSIX separators, each measured against the runner before being admitted: `;` `&` `&&` `||` `{` `(`, the `)` closing a case pattern, and the words `then` `do` `else`. A keyword may legally be preceded only by whitespace or by `;` -- every other separator in front of one is a shell syntax error, checked with `sh -n` rather than recalled -- so the scan admits exactly those two and that part of the class is closed rather than patched a spelling at a time. Both spellings are vacuous under the runner, so missing either is a silent miss, the one direction this scan must not err in. For the same reason the scan takes a `!` with no space before its subshell, `!(cmd)`, which is vacuous on dash and bash alike, and reads through a `time` or `time -p` prefix, which is vacuous under bats and under cozytest where /bin/sh is bash. A scan that wants `!` and then whitespace as the first token misses both. On a dash /bin/sh `time` is not a keyword and `time ! cmd` fails, so there it is an over-count. Five over-counts remain, all erring loud, each named in the header: they are enumerated rather than removed, because a removed error leaves no trace while a named one tells the next reader what they are looking at. Inverted conditions are untouched because errexit is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. Each claim the suite makes is pinned against removing the thing it claims, and the separator class is measured branch by branch rather than only at its newest entries. The recursion is exercised through the audit function on a fixture in a subdirectory rather than through a second copy of its find expression, because a copy passes while the function loses its recursion -- and an audit that returns empty reads as a clean tree. The condition-strip carries fixtures that put the negation after an `&&` inside the condition, where only the strip can save it. The continuation fold is pinned by a fixture whose negation sits on the continued line, so folded and unfolded name different lines. Both keyword spellings, the `time` prefix and `!(cmd)` have fixtures of their own. The subshell over-count carries a fixture for both of its halves -- that the scan reports the shape, and that the shape really does fail. Both halves of the premise are pinned, not just the half about the banned form. A test asserts the runner still passes a bare negated assertion; a second asserts the prescribed `if cmd; then ...; false; fi` does fail, under both runners, and fails AT the form rather than somewhere before it. Every converted absence rests on that second half, and its failure mode is asymmetric: were `false` in a `then` body to stop propagating, the first test would still pass while every conversion went silently green. The rule joins the E2E conventions doc as its own section, and the inline list in AGENTS.md that an agent may read instead of the doc -- it belongs beside the EXIT-trap ban for the same reason, both leaving a green suite that checks nothing. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 40dac09 commit 5137d98

3 files changed

Lines changed: 449 additions & 1 deletion

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ This file provides structured guidance for AI coding assistants and agents worki
3232

3333
- **E2E tests and E2E CI** (e.g., "write/fix an e2e test", "stabilize a flaky test", "add a bats test", "change the e2e workflow", "why did e2e fail")
3434
- Read: [`e2e-testing.md`](./docs/agents/e2e-testing.md)
35-
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), fail-fast on HelmRelease readiness
35+
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it cannot fail — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness
3636

3737
- **General questions about contributing**
3838
- Read: [`contributing.md`](./docs/agents/contributing.md)

‎docs/agents/e2e-testing.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,12 @@ Consequences to keep in mind when touching either workflow:
170170

171171
**Security prerequisite (repo setting).** The split is only safe if fork runs require maintainer approval before they execute. The repository must keep **Settings → Actions → "Require approval for all outside collaborators"** enabled, so a fork's `pull_request` run — which produces the artifacts `e2e-fork.yaml` consumes — cannot run untrusted code (or feed the privileged run) without a maintainer's go-ahead.
172172

173+
### 11. Never negate a BATS assertion with `!`
174+
175+
An assertion spelled `! cmd` inside an `@test` body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so the failing status is dropped where it is produced. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.
176+
177+
The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare `! true`, a negated `[ … ]`, one inside a `for` body, and one following `;`, `&&` or `||` all pass. Write the absence so that it fails — `if cmd; then echo "FAIL: <what must not have happened>"; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument.
178+
173179
## Chainsaw v0.2.15 gotchas
174180

175181
The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026); none of the traps below are fixed upstream yet, so the workarounds stay until a bump is possible. Each one has already cost a debugging session — this is the "no one gets it right the first time" list, worth a read before writing a new assert.
@@ -194,6 +200,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
194200
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`.
195201
8. If it touches parent-HR behavior, add the `status.history` remediation guard (`hack/e2e-chainsaw/_lib/remediation-guard.sh`).
196202
9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
203+
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`.
197204

198205
## The container lane: what the srv nodes are, and what that costs
199206

0 commit comments

Comments
 (0)