|
| 1 | +--- |
| 2 | +name: test-audit |
| 3 | +description: Invoke whenever writing, changing, reviewing, or sweeping tests anywhere in the OpenShell workspace (Rust crates, e2e/rust, the conformance suite, tests/ansible tmachine suites, Python, TypeScript SDK, Go SDK). Authoring gate for new tests plus an audit workflow for low-value, implementation-coupled, mock-only, or duplicative tests and the test-only production seams they demand. Trigger keywords - test audit, audit tests, junk tests, low-value test, prune tests, test quality, test coverage gap, mock-only coverage. |
| 4 | +metadata: |
| 5 | + internal: true |
| 6 | +--- |
| 7 | + |
| 8 | +# Test Audit |
| 9 | + |
| 10 | +Three modes, one value bar. Authoring mode gates every new or changed test at |
| 11 | +write time. Audit mode runs focused sweeps of tests that re-assert source, |
| 12 | +duplicate stronger proof, couple behavior to implementation, rely on mocks |
| 13 | +where a real dependency is cheap and available, or keep test-only production |
| 14 | +seams alive. Campaign mode prunes one whole subsystem's test surface (every |
| 15 | +test file a crate or SDK owns) — treat it as several coherent audit passes, |
| 16 | +not one giant edit. |
| 17 | + |
| 18 | +This skill covers test *quality*, not test *coverage gaps*. If you're looking |
| 19 | +for missing coverage rather than junk coverage, that's a `create-spike` |
| 20 | +investigation, not this skill. The two are complementary: an audit here can |
| 21 | +surface that a mocked-only test is standing in for coverage that should exist |
| 22 | +against a real dependency — file that as a coverage gap, don't just delete the |
| 23 | +weak test and leave the gap unrecorded. |
| 24 | + |
| 25 | +## Why this exists in OpenShell specifically |
| 26 | + |
| 27 | +OpenShell spans five test surfaces that don't share a harness or a failure |
| 28 | +mode: |
| 29 | + |
| 30 | +- **Rust unit tests** (`crates/*/src/**`, inline `#[cfg(test)] mod tests`) — |
| 31 | + fast, run against fakes/stubs (e.g. `openshell-driver-podman`'s |
| 32 | + `test_utils::spawn_podman_stub`, a hyper server over a Unix socket standing |
| 33 | + in for the real daemon). |
| 34 | +- **`e2e/rust/tests/`** — a real gateway + real driver (Docker, Podman, |
| 35 | + Kubernetes, VM), feature-gated per driver (`e2e-docker`, `e2e-podman`, |
| 36 | + `e2e-kubernetes*`), driven by shell scripts (`e2e-*.sh`) that stand up the |
| 37 | + actual stack. |
| 38 | +- **`crates/openshell-conformance/src/scenarios/`** — driver-agnostic |
| 39 | + behavioral scenarios run against whichever driver-backed gateway `mise run |
| 40 | + e2e:*` points at. The right home for a `ComputeDriver`-contract guarantee |
| 41 | + that every driver must prove, instead of five copies of the same assertion. |
| 42 | +- **`tests/ansible/` + `tests/suites/drivers/*` (the Nix `tmachine` harness)** |
| 43 | + — VM-backed driver-specific suites wired through `tests/config.nix`. Easy to |
| 44 | + believe a fixture is orphaned when it's actually consumed only here — a |
| 45 | + prior pass on #3663 wrongly called three Podman userns fixtures "orphaned" |
| 46 | + because their only consumer was an Ansible playbook, invisible to a |
| 47 | + `.rs`/`.yml` grep. Read `tests/config.nix` and the actual playbooks before |
| 48 | + concluding a fixture, script, or env var is unused. |
| 49 | +- **Python (`python/`), TypeScript SDK (`sdk/typescript/`), Go SDK |
| 50 | + (`sdk/go/`)** — each with its own test runner and its own conventions; see |
| 51 | + Discovery below for the exact commands. |
| 52 | + |
| 53 | +A test that passes against a mock but was never run against the real |
| 54 | +dependency is a hypothesis, not proof. #3663 found exactly this for the |
| 55 | +Podman driver: resource-limit admission and daemon-failure handling were |
| 56 | +fully unit-tested against a fake client with zero real-daemon coverage — a |
| 57 | +gap invisible from the outside, since the tests are green and clippy has |
| 58 | +nothing to say about them. Mock-only coverage of a real-dependency contract is |
| 59 | +a first-class junk pattern here (see below), not an afterthought. |
| 60 | + |
| 61 | +## Authoring gate |
| 62 | + |
| 63 | +Before adding any test, answer four questions; a missing answer means do not |
| 64 | +add it yet: |
| 65 | + |
| 66 | +1. What observable behavior, invariant, or independent contract does it |
| 67 | + protect? If the feature crosses a real boundary (process, privilege/identity, |
| 68 | + declaration-to-enforcement, our-code-to-external-tool — see |
| 69 | + [Integration boundaries](#integration-boundaries-test-what-crosses-not-just-whats-inside)), |
| 70 | + name the boundary explicitly; "the function returns the right value" is not |
| 71 | + the same claim as "the value survives the handoff to whatever consumes it." |
| 72 | +2. What credible regression makes it fail? |
| 73 | +3. Why does existing coverage not already catch that failure? Each contract |
| 74 | + has one primary test owner at the strongest boundary — usually the |
| 75 | + `crates/openshell-conformance` scenario if the contract applies to more |
| 76 | + than one driver, or the specific `e2e/rust/tests/<driver>_*.rs` file if it's |
| 77 | + genuinely driver-specific (a Podman API error shape, a rootless-only |
| 78 | + Podman flag). Another layer needs its own distinct risk, such as a real |
| 79 | + daemon a mock can't fail the way the real one does. Prefer extending a |
| 80 | + table-driven case or shared fixture over a near-duplicate test. |
| 81 | +4. Does it need a production seam (export, flag, wrapper, injection hook) that |
| 82 | + no production caller needs? If yes, move the test to the real boundary |
| 83 | + instead. |
| 84 | + |
| 85 | +Then check the test against every [junk pattern](#junk-patterns); a match |
| 86 | +fails the gate unless the [retention bar](#retention-bar) names the contract |
| 87 | +it independently guards. A test that would break under behavior-preserving |
| 88 | +refactoring is asserting implementation, not behavior; rewrite it at the |
| 89 | +owning boundary before landing it. |
| 90 | + |
| 91 | +Bug regression tests must fail on the pre-fix code for the intended reason and |
| 92 | +pass after the owner-boundary repair. A regression test that never |
| 93 | +demonstrably failed proves the mock, not the fix. One regression at the owner |
| 94 | +boundary covers the bug; do not replay the same scenario at every layer it |
| 95 | +crosses (unit test, e2e test, and conformance scenario all asserting the same |
| 96 | +Podman error string is three maintenance burdens for one bug). |
| 97 | + |
| 98 | +## Unit tests: verify the contract, not the implementation |
| 99 | + |
| 100 | +The most common defect in agent-authored unit tests looks like a passing |
| 101 | +test: it runs, asserts, and stays green. The flaw is where the expected value |
| 102 | +came from. Ask of every assertion: **where did this expected value come |
| 103 | +from?** |
| 104 | + |
| 105 | +- Traced or run from the implementation itself — circular. Proves the code |
| 106 | + matches itself, not that it's correct. Rederive from an independent source |
| 107 | + or reject the test. |
| 108 | +- Derived from a spec, protocol, real dependency, or an independently |
| 109 | + computed value — valid, since it can actually fail when the code is wrong. |
| 110 | + |
| 111 | +Example (#3663): `parse_cpu_to_microseconds`/`parse_memory_to_bytes` convert |
| 112 | +`"500m"`/`"512Mi"` to cgroup values. Hardcoding `50_000`/`536_870_912` |
| 113 | +because "that's what the function returns" proves nothing. |
| 114 | +`podman_resource_limits.rs` instead checks a real `podman run |
| 115 | +--cpus=0.5 --memory=512m` container's actual `cpu.max`/`memory.max` — an |
| 116 | +independent yardstick that would catch a regression in the math. |
| 117 | + |
| 118 | +When no independent ground truth exists, derive the expected value by a |
| 119 | +different calculation path than the implementation, not by re-running it. |
| 120 | +Same rule applies to e2e/conformance assertions computed by calling the same |
| 121 | +code path a second time. |
| 122 | + |
| 123 | +## Integration boundaries: test what crosses, not just what's inside |
| 124 | + |
| 125 | +A test can pass the check above and still miss the bug, because the bug lives |
| 126 | +in the handoff between components, not inside either one. Mocking every |
| 127 | +collaborator proves each side behaves correctly given its assumptions about |
| 128 | +the other — not that the assumption is true. Name the boundaries a feature |
| 129 | +crosses and make sure at least one test crosses each for real: |
| 130 | + |
| 131 | +- **Process** — does a `Result<(), Err>` actually surface as the right exit |
| 132 | + code and error text through `main()`? Only proven for `PodmanApiError::Connection` |
| 133 | + once `podman_preflight.rs` spawned the real binary and read its exit code. |
| 134 | +- **Privilege/identity** — does data written by one identity stay readable by |
| 135 | + the identity that reads it later? The rootful playbook bug: a reference file |
| 136 | + was captured as `root` but always read back by the unprivileged `tmachine` |
| 137 | + test runner. Every task looked correct in isolation; the bug was the handoff. |
| 138 | +- **Declaration-to-enforcement** — does a value the API echoes back actually |
| 139 | + get enforced underneath? `sandbox_templates.rs` proves the API stores |
| 140 | + `--cpu`/`--memory`; `podman_resource_limits.rs` proves cgroups enforce it. |
| 141 | + Different claims — one test passing says nothing about the other. |
| 142 | +- **Our-code-to-external-tool** — does our belief about a real tool's |
| 143 | + behavior match the tool, not our assumption about it? Checked repeatedly |
| 144 | + against real Podman (man page, `podman run` output) rather than trusting |
| 145 | + driver-side comments. |
| 146 | + |
| 147 | +A hundred green unit tests that all mock the far side of every seam will |
| 148 | +never catch a boundary bug. Ask: "if I mocked every collaborator, which |
| 149 | +handoffs would I never exercise?" Those are worth a real dependency instead of |
| 150 | +a mock. |
| 151 | + |
| 152 | +## Junk patterns |
| 153 | + |
| 154 | +The shared checklist for both modes: the authoring gate rejects a new test |
| 155 | +that matches one, and audits hunt for existing tests that do. |
| 156 | + |
| 157 | +- assertion-free coverage probes; |
| 158 | +- self-comparisons and identity copiers; |
| 159 | +- copied fixtures, inventories, manifests, or export lists; |
| 160 | +- exact source, import, or string greps; |
| 161 | +- private predicate or call-shape tests duplicated at real boundaries; |
| 162 | +- duplicate invocations of the same contract across unit, e2e, and |
| 163 | + conformance layers with no distinct failure mode at each layer; |
| 164 | +- provider-local (per-driver) replays of a shared conformance-level contract; |
| 165 | +- tests whose only purpose is preserving test-only exports, globals, or |
| 166 | + wrappers; |
| 167 | +- dead production code whose only callers are tests; |
| 168 | +- expected values produced by, traced from, or hand-simulated on the same |
| 169 | + implementation under test instead of an independent source — see |
| 170 | + [Unit tests: verify the contract, not the implementation](#unit-tests-verify-the-contract-not-the-implementation) |
| 171 | + for the diagnostic question and a worked example; |
| 172 | +- mocks that implement the asserted behavior, or one identical mock standing |
| 173 | + in for different APIs; |
| 174 | +- **a mock/stub standing in for a real dependency that is actually available |
| 175 | + and cheap to run in CI** (a `spawn_podman_stub`-style fake covering behavior |
| 176 | + a real `podman run` could prove directly) — flag this even if the test |
| 177 | + itself is well-formed, because the gap it's hiding is coverage-shaped, not |
| 178 | + junk-shaped; |
| 179 | +- fixtures that supply the receipt, admission, or callback ordering the owner |
| 180 | + should produce, or persistence asserted against a store the path never |
| 181 | + writes; |
| 182 | +- capability tests that restate declared flags instead of exercising the |
| 183 | + delivery or acknowledgement the flag promises; |
| 184 | +- negative controls that pass for an unrelated reason, such as a denial from a |
| 185 | + different guard, a rejection the production path never reaches, or (Ansible |
| 186 | + suites specifically) an assertion that would pass trivially because the |
| 187 | + environment it's asserting against was never actually reached; |
| 188 | +- names or fixtures that promise more than the input exercises, such as a |
| 189 | + "verifies rootful parity" test that only runs under rootless because a |
| 190 | + service-user assertion silently skipped it. |
| 191 | + |
| 192 | +## Value bar |
| 193 | + |
| 194 | +Tests justify their maintenance cost by protecting behavior, a credible |
| 195 | +regression, or an independently meaningful contract. In an audit, an existing |
| 196 | +test that must change for behavior-preserving source reorganization is |
| 197 | +suspect, not automatically deletable; the authoring gate still rejects new |
| 198 | +ones. |
| 199 | + |
| 200 | +Before judging a candidate, read the complete test and its production owner, |
| 201 | +entry point, callers, callees, sibling implementations across drivers, |
| 202 | +overlapping tests at other layers, CI routing (which `mise run` task and which |
| 203 | +`.github/workflows/*.yml` job actually executes it — a test can be well-formed |
| 204 | +and simply never run; check `required-features` and workflow env vars, not |
| 205 | +just the test file), and relevant history. Read root and scoped `AGENTS.md` |
| 206 | +files first. When the test claims dependency-backed behavior (a Podman API |
| 207 | +shape, a cgroup value, a systemd unit's behavior), verify it against the real |
| 208 | +dependency or its documentation directly — don't take the mock's word for it. |
| 209 | + |
| 210 | +## Discovery |
| 211 | + |
| 212 | +Keep discovery read-only and report evidence before editing. For broad scope, |
| 213 | +run parallel discovery lanes when available, one per surface: |
| 214 | + |
| 215 | +- Rust unit tests: `cargo nextest run --config-file .config/nextest.toml` (or |
| 216 | + `mise run test:rust`) per crate under audit; |
| 217 | +- `e2e/rust/tests/`: `cargo check --manifest-path e2e/rust/Cargo.toml |
| 218 | + --all-targets --features <e2e-driver-feature>` to enumerate what's gated |
| 219 | + behind which feature, then cross-reference against |
| 220 | + `.github/workflows/branch-e2e.yml` and `tasks/test.toml` to confirm the |
| 221 | + feature actually gets exercised in CI (it may not — see the Podman `E2E_FEATURES` |
| 222 | + empty-string wiring bug found in #3663, where a real, well-written test was |
| 223 | + silently skipped by an env var default); |
| 224 | +- conformance suite: `crates/openshell-conformance/src/scenarios/`; |
| 225 | +- `tests/ansible/` + `tests/suites/drivers/*`: read `tests/config.nix`'s |
| 226 | + `testsuites` list to find every playbook a suite actually runs, not just the |
| 227 | + ones with an obvious name; |
| 228 | +- Python: `mise run test:python` (`uv run pytest python/`); |
| 229 | +- TypeScript SDK: `mise run sdk:ts:test`; |
| 230 | +- Go SDK: `mise run go:test` / `go:test:integration`; |
| 231 | +- a cross-cutting pattern sweep (grep for the junk patterns above across |
| 232 | + whichever surfaces are in scope). |
| 233 | + |
| 234 | +Outside campaign mode, prefer a few high-confidence candidates over a large |
| 235 | +speculative inventory. Hunt for the [junk patterns](#junk-patterns). |
| 236 | + |
| 237 | +## Retention bar |
| 238 | + |
| 239 | +Keep a test when it independently enforces a public API, gRPC/proto contract, |
| 240 | +`ComputeDriver` guarantee, gateway config schema, policy/OPA contract, |
| 241 | +security boundary, sandbox lifecycle invariant, SDK (Go/TypeScript/Python) |
| 242 | +contract, or CI/release artifact contract. Also keep: |
| 243 | + |
| 244 | +- call ordering when order is observable behavior; |
| 245 | +- regressions with a credible failure mode; |
| 246 | +- source inspection when it is the cheapest independent guard: it fails when |
| 247 | + the contract changes (a config key, a wire byte, a CLI flag) and survives an |
| 248 | + identifier-only refactor; |
| 249 | +- a retained test that fails on the baseline: treat it as a possible product |
| 250 | + bug, reproduce it, and repair the owner rather than deleting it. |
| 251 | + |
| 252 | +Static or slow is not a deletion reason — a `tests/ansible` VM suite is slow |
| 253 | +by nature and may still be the only real-daemon proof that exists. A test that |
| 254 | +resembles implementation may still be the independent contract; prove |
| 255 | +otherwise before removing it. |
| 256 | + |
| 257 | +## Candidate evidence |
| 258 | + |
| 259 | +Record every field below before editing. A missing field means the candidate |
| 260 | +is not ready for deletion: |
| 261 | + |
| 262 | +- exact test name and location (crate/file, or `e2e/rust/tests/<file>.rs`, or |
| 263 | + `tests/ansible/playbooks/...`); |
| 264 | +- what failure it can actually detect; |
| 265 | +- non-test callers of the covered production or support seam; |
| 266 | +- stronger remaining owner-boundary proof, or why no proof is needed; |
| 267 | +- whether it currently runs in CI at all (check the workflow, not just the |
| 268 | + feature gate — see Discovery above) and what happens if you delete it |
| 269 | + versus if you fix its wiring instead; |
| 270 | +- relevant history and the reason the test or seam exists; |
| 271 | +- production or test-support deletion unlocked; |
| 272 | +- risk and the focused validation command. |
| 273 | + |
| 274 | +## Edit shape |
| 275 | + |
| 276 | +Choose one coherent owner-boundary batch, scoped to one crate, one SDK, or one |
| 277 | +driver's `e2e/rust/tests/` files at a time. Delete obsolete test-only exports, |
| 278 | +globals, wrappers, and dead production paths instead of preserving aliases. |
| 279 | +Move retained regressions to their canonical owners (conformance suite if the |
| 280 | +contract is driver-agnostic; the driver-specific file if it's genuinely |
| 281 | +driver-local). Consolidate repeated per-driver assertions of the same |
| 282 | +`ComputeDriver` contract into one conformance scenario. |
| 283 | + |
| 284 | +Prefer net-negative production LOC. Do not add replacement tests that restate |
| 285 | +the same implementation, and do not convert uncertain candidates into cleanup |
| 286 | +to increase deletion counts. If an audit surfaces mock-only coverage of a |
| 287 | +real-dependency contract, that is a coverage-gap finding, not a deletion — say |
| 288 | +so explicitly in the handoff rather than silently leaving it. |
| 289 | + |
| 290 | +## Validation |
| 291 | + |
| 292 | +Follow the language's own pre-commit gate, not just the test itself — a |
| 293 | +passing test in isolation can still fail `cargo fmt --all -- --check` or |
| 294 | +`cargo clippy`. |
| 295 | + |
| 296 | +1. Run the smallest owner and sibling tests directly (`cargo nextest run -p |
| 297 | + <crate> <filter>`, `cargo test --manifest-path e2e/rust/Cargo.toml --test |
| 298 | + <name> --features <feature>`, `uv run pytest python/<path>`, etc.) before |
| 299 | + running anything broader. |
| 300 | +2. For e2e tests that need a real driver and are runnable locally (Podman is |
| 301 | + commonly available; Docker/Kubernetes/VM often are not in a given |
| 302 | + environment), actually build and run them rather than trusting a compile |
| 303 | + check — `cargo build -p openshell-driver-podman` then `cargo test |
| 304 | + --manifest-path e2e/rust/Cargo.toml --test <name> --features e2e-podman` |
| 305 | + is cheap and catches wiring bugs a `cargo check` won't. |
| 306 | +3. For removed source greps or plan assertions, run the executable script or |
| 307 | + dry-run that owns the real contract. |
| 308 | +4. Run `cargo fmt --all -- --check` (Rust) and the equivalent formatter for |
| 309 | + whatever language changed, then `git diff --check`. |
| 310 | +5. Run `mise run pre-commit` once per coherent batch, not after every single |
| 311 | + file edit — it recompiles/clippies the full Rust workspace and is |
| 312 | + expensive; batch changes and run it before committing, not iteratively. |
| 313 | +6. Inspect `git diff --numstat`; report production/tooling separately from |
| 314 | + tests and test support. |
| 315 | + |
| 316 | +## Check for overlap before filing an issue or opening a PR |
| 317 | + |
| 318 | +Before filing a new issue for a finding: |
| 319 | + |
| 320 | +- Search issues with `--state all` (not just open) and more than one phrasing |
| 321 | + — component name, file path, symptom, and subsystem name can each surface |
| 322 | + different matching issues. |
| 323 | +- If something adjacent but not identical exists, say so in the new issue |
| 324 | + rather than ignoring it or treating it as a duplicate. |
| 325 | + |
| 326 | +Before opening a PR, or starting implementation: |
| 327 | + |
| 328 | +- Check open PRs for overlapping files or behavior, not just titles |
| 329 | + (`gh pr list --search`, then `gh pr diff <number>` on anything adjacent) — a |
| 330 | + PR's title can be unrelated while its diff covers the same ground. |
| 331 | +- If an in-flight PR already does part of the work, cross-reference it and |
| 332 | + narrow scope instead of duplicating it. |
| 333 | +- Re-check if meaningful time passed since filing — overlap can appear later. |
| 334 | + |
| 335 | +## Landing and continuation |
| 336 | + |
| 337 | +Commit, push, open a PR, or land only when authorized — this repo requires |
| 338 | +explicit permission before `git push` (see the repository's own CLAUDE.md |
| 339 | +push-workflow rules) and only creates commits when asked. Use |
| 340 | +`create-github-pr` for PR mechanics once authorized. Scope one coherent PR per |
| 341 | +audited surface; after landing, rerun read-only discovery for the next |
| 342 | +high-confidence batch rather than queuing speculative follow-ups. |
| 343 | + |
| 344 | +A new or substantially changed test-audit finding that reveals a real product |
| 345 | +gap (not a junk test, but missing coverage the audit surfaced) belongs in its |
| 346 | +own issue via `create-github-issue` or `create-spike` — do not fold it into |
| 347 | +the audit PR. Check for overlap first, per the section above, before filing |
| 348 | +it. |
| 349 | + |
| 350 | +## Handoff |
| 351 | + |
| 352 | +Report: |
| 353 | + |
| 354 | +- root cause and removed low-value categories, by surface (unit / e2e / |
| 355 | + conformance / tmachine / SDK); |
| 356 | +- production owner simplifications; |
| 357 | +- retained false positives and why they remain valuable; |
| 358 | +- any mock-only-coverage findings surfaced but not fixed here (coverage gaps, |
| 359 | + not junk — flag for a separate issue); |
| 360 | +- focused and full proof actually run, and what could not be run locally |
| 361 | + (e.g. drivers without local hardware/daemon access) versus what was |
| 362 | + verified end-to-end; |
| 363 | +- production versus test LOC; |
| 364 | +- PR and merge state; |
| 365 | +- named follow-ups. |
0 commit comments