Skip to content

Commit 1e7ba63

Browse files
committed
docs(agents): define testing and authoring conventions
Require tests for behavior changes without inventing failures for prose or unchanged upstream updates. Keep new image build filenames explicit and preserve the existing lookup policy. Correct the assertion guidance: Bats can fail on a final negation, while cozytest appends a successful return. Separate numeric validation from text coercion and account for null parent maps in Helm values. Validation: markdownlint (MD013 excluded) and git diff --check pass. Fixes: 5137d98 ("test(bats): refuse a negated assertion across every suite") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 3b5ede1 commit 1e7ba63

3 files changed

Lines changed: 43 additions & 6 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ This file provides structured guidance for AI coding assistants and agents worki
3636

3737
- **E2E tests and E2E CI** (e.g., "write/fix an e2e test", "stabilize a flaky test", "add a bats test", "change the e2e workflow", "why did e2e fail")
3838
- Read: [`e2e-testing.md`](./docs/agents/e2e-testing.md)
39-
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it cannot fail — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness
39+
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it can be silently ignored — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness
4040

4141
- **General questions about contributing**
4242
- Read: [`contributing.md`](./docs/agents/contributing.md)

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -181,9 +181,9 @@ Consequences to keep in mind when touching either workflow:
181181

182182
### 11. Never negate a BATS assertion with `!`
183183

184-
An assertion spelled `! cmd` inside an `@test` body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so the failing status is dropped where it is produced. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.
184+
An assertion spelled `! cmd` inside an `@test` body can silently pass. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so a failing inverted status does not stop the test body. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.
185185

186-
The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare `! true`, a negated `[ … ]`, one inside a `for` body, and one following `;`, `&&` or `||` all pass. Write the absence so that it fails — `if cmd; then echo "FAIL: <what must not have happened>"; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument.
186+
The exemption applies to any inverted command, not just pipelines: `! true`, negated `[ … ]`, negation inside a `for` body, and negation following `;`, `&&` or `||` do not trigger errexit. A later successful statement can hide their failure. Write the absence so that it fails — `if cmd; then echo "FAIL: <what must not have happened>"; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument.
187187

188188
## Chainsaw v0.2.15 gotchas
189189

@@ -209,7 +209,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
209209
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`.
210210
8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see.
211211
9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
212-
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`.
212+
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and can be silently ignored; use `if cmd; then echo "FAIL: …"; false; fi`.
213213

214214
## The container lane: what the srv nodes are, and what that costs
215215

‎docs/agents/overview.md‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m
1616

1717
## Code Layout
1818

19-
```
19+
```text
2020
.
2121
├── packages/ # Main directory for cozystack packages
2222
│ ├── core/ # Core platform logic charts (installer, platform)
@@ -50,7 +50,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m
5050

5151
Every package is a Helm chart following the umbrella chart pattern:
5252

53-
```
53+
```text
5454
packages/<category>/<package-name>/
5555
├── Chart.yaml # Chart definition and parameter docs
5656
├── Makefile # Development workflow targets
@@ -110,21 +110,54 @@ A chart also declares the limits on its own resource name there, with `## @name
110110
- **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md).
111111
- **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers.
112112

113+
### Test first, failure path first
114+
115+
These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules.
116+
117+
1. Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it.
118+
2. Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract.
119+
3. Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases.
120+
4. A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI.
121+
5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked `FIXME(#N)`. The PR that fixes the bug flips the test to pin the fixed behaviour.
122+
123+
Do not negate a BATS assertion with `!`: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; `hack/cozytest.sh` appends a successful return and misses it there too. See [§11 of `e2e-testing.md`](./e2e-testing.md#11-never-negate-a-bats-assertion-with-).
124+
125+
### Linters
126+
127+
Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline.
128+
129+
Markdown follows these markdownlint rules: every code fence names a language (`text` when nothing else fits), table separators have spaces around the pipes (`| --- |`), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as [Prose Formatting](../../AGENTS.md#prose-formatting) explains.
130+
113131
## Conventions
114132

115133
### Helm Charts
134+
116135
- Follow **umbrella chart** pattern for system components
117136
- Include upstream charts in `charts/` directory (vendored, not referenced)
118137
- Override configuration in root `values.yaml`
119138
- Use `values.schema.json` for input validation and dashboard UI rendering
139+
- An unquoted numeric value (`name: 123`) reaches a template as a number: `len` on it aborts the render with `len of type float64` (`int64` for a value given with `--set`) before any `fail` message is reached, and `if not $v` treats `0` as unset. For fields validated as text that permit numeric inputs, reject missing or null values with `kindIs "invalid"`, then coerce with `printf "%v"` and check emptiness with `eq $v ""`. Coercing null first produces the non-empty string `<nil>`.
140+
- For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic `printf "%v"` can emit exponent notation for a float (`4200000000` becomes `4.2e+09`), so it is not integer normalization.
141+
- For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent.
120142

121143
**No `lookup` that copies another object into the render.** A `lookup` reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one.
122144

123145
A `lookup` is fine when a repeated render gives the same output and carries nothing from another object into it: checking that an object exists, keeping a generated Secret's value, keeping an immutable field of the chart's own live object, or a guard that `fail`s on an unsafe change. Keep its empty branch rendering, because offline `helm template` and helm-unittest without a `kubernetesProvider` mock see only that branch; test the other branch with such a mock.
124146

125147
To use a fact about the cluster instead, read it from values: `_cluster` and `_namespace` in the `cozystack-values` Secret, or `valuesFrom` on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with `.Capabilities.APIVersions.Has`, since a `lookup` on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object.
126148

149+
### Container images
150+
151+
For a new container image built from source maintained in this repository, name the build definition `Containerfile` and pass it to `docker buildx build` with `--file` in the package `Makefile`. Keep existing `Dockerfile`s and vendored build definitions unchanged.
152+
153+
### Examples and fixtures
154+
155+
Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), IPv6 from RFC 3849 (`2001:db8::/32`), domains from RFC 2606 and RFC 6761 (`example.com`, `.example`, `.test`, `.invalid`), ASNs from RFC 5398 (`64496`–`64511`, `65536`–`65551`), MAC addresses from RFC 7042 (`00:00:5E:00:53:00`–`00:00:5E:00:53:FF`). Where no reserved range exists, use an obviously synthetic placeholder such as `<NODE_IP>`.
156+
157+
When a test depends on a protocol-defined address's behavior, use that value, such as `127.0.0.1` or `localhost` for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints.
158+
127159
### Go Code
160+
128161
- Follow standard **Go conventions** and idioms
129162
- Use **controller-runtime** patterns for Kubernetes controllers
130163
- Prefer **kubebuilder** for API definitions and controllers
@@ -170,20 +203,24 @@ Before working on these areas, read the relevant doc first:
170203
## Things Agents Should Not Do
171204

172205
### Never Edit These
206+
173207
- Do not modify files in `/vendor/` (Go dependencies)
174208
- Do not edit generated files: `zz_generated.*.go`
175209
- Do not change `go.mod`/`go.sum` manually (use `go get`)
176210
- Do not edit upstream charts in `packages/*/charts/` directly (use patches)
177211
- Do not modify image digests in `values.yaml` (generated by build)
178212

179213
### Version Control
214+
180215
- Do not commit built artifacts from `_out`
181216
- Do not commit test artifacts or temporary files
182217

183218
### Git Operations
219+
184220
- Do not force push to main/master
185221
- Do not update git config
186222
- Do not perform destructive operations without explicit request
187223

188224
### Core Components
225+
189226
- Do not modify `packages/core/platform/` without understanding migration impact

0 commit comments

Comments
 (0)