Skip to content

Commit 71436db

Browse files
committed
chore(agents): add test-audit skill for contributor test-quality gating
OpenShell had no internal skill for auditing test quality across its five distinct test surfaces (Rust unit tests, e2e/rust, the conformance suite, the tmachine/Ansible harness, and the Python/TypeScript/Go SDKs). Modeled on openclaw's test-audit skill, adapted to this repo's actual tooling (mise tasks, cargo nextest, uv, driver feature gates) instead of Vitest, with two OpenShell-specific additions grounded in lessons from NVIDIA#3663/NVIDIA#3690: a section on deriving expected values from an independent source rather than tracing the implementation itself, and a section on testing the handoff between components (process, privilege, and declaration-to-enforcement boundaries) rather than only what's inside each one. Wired into CONTRIBUTING.md's contributor skills table per sync-agent-infra's maintenance map; no other cross-references needed updating. Signed-off-by: politerealism <[email protected]>
1 parent ca18712 commit 71436db

2 files changed

Lines changed: 366 additions & 0 deletions

File tree

‎.agents/skills/test-audit/SKILL.md‎

Lines changed: 365 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,365 @@
1+
---
2+
name: test-audit
3+
description: Invoke whenever writing, changing, reviewing, or sweeping tests anywhere in the OpenShell workspace (Rust crates, e2e/rust, the conformance suite, tests/ansible tmachine suites, Python, TypeScript SDK, Go SDK). Authoring gate for new tests plus an audit workflow for low-value, implementation-coupled, mock-only, or duplicative tests and the test-only production seams they demand. Trigger keywords - test audit, audit tests, junk tests, low-value test, prune tests, test quality, test coverage gap, mock-only coverage.
4+
metadata:
5+
internal: true
6+
---
7+
8+
# Test Audit
9+
10+
Three modes, one value bar. Authoring mode gates every new or changed test at
11+
write time. Audit mode runs focused sweeps of tests that re-assert source,
12+
duplicate stronger proof, couple behavior to implementation, rely on mocks
13+
where a real dependency is cheap and available, or keep test-only production
14+
seams alive. Campaign mode prunes one whole subsystem's test surface (every
15+
test file a crate or SDK owns) — treat it as several coherent audit passes,
16+
not one giant edit.
17+
18+
This skill covers test *quality*, not test *coverage gaps*. If you're looking
19+
for missing coverage rather than junk coverage, that's a `create-spike`
20+
investigation, not this skill. The two are complementary: an audit here can
21+
surface that a mocked-only test is standing in for coverage that should exist
22+
against a real dependency — file that as a coverage gap, don't just delete the
23+
weak test and leave the gap unrecorded.
24+
25+
## Why this exists in OpenShell specifically
26+
27+
OpenShell spans five test surfaces that don't share a harness or a failure
28+
mode:
29+
30+
- **Rust unit tests** (`crates/*/src/**`, inline `#[cfg(test)] mod tests`) —
31+
fast, run against fakes/stubs (e.g. `openshell-driver-podman`'s
32+
`test_utils::spawn_podman_stub`, a hyper server over a Unix socket standing
33+
in for the real daemon).
34+
- **`e2e/rust/tests/`** — a real gateway + real driver (Docker, Podman,
35+
Kubernetes, VM), feature-gated per driver (`e2e-docker`, `e2e-podman`,
36+
`e2e-kubernetes*`), driven by shell scripts (`e2e-*.sh`) that stand up the
37+
actual stack.
38+
- **`crates/openshell-conformance/src/scenarios/`** — driver-agnostic
39+
behavioral scenarios run against whichever driver-backed gateway `mise run
40+
e2e:*` points at. The right home for a `ComputeDriver`-contract guarantee
41+
that every driver must prove, instead of five copies of the same assertion.
42+
- **`tests/ansible/` + `tests/suites/drivers/*` (the Nix `tmachine` harness)**
43+
— VM-backed driver-specific suites wired through `tests/config.nix`. Easy to
44+
believe a fixture is orphaned when it's actually consumed only here — a
45+
prior pass on #3663 wrongly called three Podman userns fixtures "orphaned"
46+
because their only consumer was an Ansible playbook, invisible to a
47+
`.rs`/`.yml` grep. Read `tests/config.nix` and the actual playbooks before
48+
concluding a fixture, script, or env var is unused.
49+
- **Python (`python/`), TypeScript SDK (`sdk/typescript/`), Go SDK
50+
(`sdk/go/`)** — each with its own test runner and its own conventions; see
51+
Discovery below for the exact commands.
52+
53+
A test that passes against a mock but was never run against the real
54+
dependency is a hypothesis, not proof. #3663 found exactly this for the
55+
Podman driver: resource-limit admission and daemon-failure handling were
56+
fully unit-tested against a fake client with zero real-daemon coverage — a
57+
gap invisible from the outside, since the tests are green and clippy has
58+
nothing to say about them. Mock-only coverage of a real-dependency contract is
59+
a first-class junk pattern here (see below), not an afterthought.
60+
61+
## Authoring gate
62+
63+
Before adding any test, answer four questions; a missing answer means do not
64+
add it yet:
65+
66+
1. What observable behavior, invariant, or independent contract does it
67+
protect? If the feature crosses a real boundary (process, privilege/identity,
68+
declaration-to-enforcement, our-code-to-external-tool — see
69+
[Integration boundaries](#integration-boundaries-test-what-crosses-not-just-whats-inside)),
70+
name the boundary explicitly; "the function returns the right value" is not
71+
the same claim as "the value survives the handoff to whatever consumes it."
72+
2. What credible regression makes it fail?
73+
3. Why does existing coverage not already catch that failure? Each contract
74+
has one primary test owner at the strongest boundary — usually the
75+
`crates/openshell-conformance` scenario if the contract applies to more
76+
than one driver, or the specific `e2e/rust/tests/<driver>_*.rs` file if it's
77+
genuinely driver-specific (a Podman API error shape, a rootless-only
78+
Podman flag). Another layer needs its own distinct risk, such as a real
79+
daemon a mock can't fail the way the real one does. Prefer extending a
80+
table-driven case or shared fixture over a near-duplicate test.
81+
4. Does it need a production seam (export, flag, wrapper, injection hook) that
82+
no production caller needs? If yes, move the test to the real boundary
83+
instead.
84+
85+
Then check the test against every [junk pattern](#junk-patterns); a match
86+
fails the gate unless the [retention bar](#retention-bar) names the contract
87+
it independently guards. A test that would break under behavior-preserving
88+
refactoring is asserting implementation, not behavior; rewrite it at the
89+
owning boundary before landing it.
90+
91+
Bug regression tests must fail on the pre-fix code for the intended reason and
92+
pass after the owner-boundary repair. A regression test that never
93+
demonstrably failed proves the mock, not the fix. One regression at the owner
94+
boundary covers the bug; do not replay the same scenario at every layer it
95+
crosses (unit test, e2e test, and conformance scenario all asserting the same
96+
Podman error string is three maintenance burdens for one bug).
97+
98+
## Unit tests: verify the contract, not the implementation
99+
100+
The most common defect in agent-authored unit tests looks like a passing
101+
test: it runs, asserts, and stays green. The flaw is where the expected value
102+
came from. Ask of every assertion: **where did this expected value come
103+
from?**
104+
105+
- Traced or run from the implementation itself — circular. Proves the code
106+
matches itself, not that it's correct. Rederive from an independent source
107+
or reject the test.
108+
- Derived from a spec, protocol, real dependency, or an independently
109+
computed value — valid, since it can actually fail when the code is wrong.
110+
111+
Example (#3663): `parse_cpu_to_microseconds`/`parse_memory_to_bytes` convert
112+
`"500m"`/`"512Mi"` to cgroup values. Hardcoding `50_000`/`536_870_912`
113+
because "that's what the function returns" proves nothing.
114+
`podman_resource_limits.rs` instead checks a real `podman run
115+
--cpus=0.5 --memory=512m` container's actual `cpu.max`/`memory.max` — an
116+
independent yardstick that would catch a regression in the math.
117+
118+
When no independent ground truth exists, derive the expected value by a
119+
different calculation path than the implementation, not by re-running it.
120+
Same rule applies to e2e/conformance assertions computed by calling the same
121+
code path a second time.
122+
123+
## Integration boundaries: test what crosses, not just what's inside
124+
125+
A test can pass the check above and still miss the bug, because the bug lives
126+
in the handoff between components, not inside either one. Mocking every
127+
collaborator proves each side behaves correctly given its assumptions about
128+
the other — not that the assumption is true. Name the boundaries a feature
129+
crosses and make sure at least one test crosses each for real:
130+
131+
- **Process** — does a `Result<(), Err>` actually surface as the right exit
132+
code and error text through `main()`? Only proven for `PodmanApiError::Connection`
133+
once `podman_preflight.rs` spawned the real binary and read its exit code.
134+
- **Privilege/identity** — does data written by one identity stay readable by
135+
the identity that reads it later? The rootful playbook bug: a reference file
136+
was captured as `root` but always read back by the unprivileged `tmachine`
137+
test runner. Every task looked correct in isolation; the bug was the handoff.
138+
- **Declaration-to-enforcement** — does a value the API echoes back actually
139+
get enforced underneath? `sandbox_templates.rs` proves the API stores
140+
`--cpu`/`--memory`; `podman_resource_limits.rs` proves cgroups enforce it.
141+
Different claims — one test passing says nothing about the other.
142+
- **Our-code-to-external-tool** — does our belief about a real tool's
143+
behavior match the tool, not our assumption about it? Checked repeatedly
144+
against real Podman (man page, `podman run` output) rather than trusting
145+
driver-side comments.
146+
147+
A hundred green unit tests that all mock the far side of every seam will
148+
never catch a boundary bug. Ask: "if I mocked every collaborator, which
149+
handoffs would I never exercise?" Those are worth a real dependency instead of
150+
a mock.
151+
152+
## Junk patterns
153+
154+
The shared checklist for both modes: the authoring gate rejects a new test
155+
that matches one, and audits hunt for existing tests that do.
156+
157+
- assertion-free coverage probes;
158+
- self-comparisons and identity copiers;
159+
- copied fixtures, inventories, manifests, or export lists;
160+
- exact source, import, or string greps;
161+
- private predicate or call-shape tests duplicated at real boundaries;
162+
- duplicate invocations of the same contract across unit, e2e, and
163+
conformance layers with no distinct failure mode at each layer;
164+
- provider-local (per-driver) replays of a shared conformance-level contract;
165+
- tests whose only purpose is preserving test-only exports, globals, or
166+
wrappers;
167+
- dead production code whose only callers are tests;
168+
- expected values produced by, traced from, or hand-simulated on the same
169+
implementation under test instead of an independent source — see
170+
[Unit tests: verify the contract, not the implementation](#unit-tests-verify-the-contract-not-the-implementation)
171+
for the diagnostic question and a worked example;
172+
- mocks that implement the asserted behavior, or one identical mock standing
173+
in for different APIs;
174+
- **a mock/stub standing in for a real dependency that is actually available
175+
and cheap to run in CI** (a `spawn_podman_stub`-style fake covering behavior
176+
a real `podman run` could prove directly) — flag this even if the test
177+
itself is well-formed, because the gap it's hiding is coverage-shaped, not
178+
junk-shaped;
179+
- fixtures that supply the receipt, admission, or callback ordering the owner
180+
should produce, or persistence asserted against a store the path never
181+
writes;
182+
- capability tests that restate declared flags instead of exercising the
183+
delivery or acknowledgement the flag promises;
184+
- negative controls that pass for an unrelated reason, such as a denial from a
185+
different guard, a rejection the production path never reaches, or (Ansible
186+
suites specifically) an assertion that would pass trivially because the
187+
environment it's asserting against was never actually reached;
188+
- names or fixtures that promise more than the input exercises, such as a
189+
"verifies rootful parity" test that only runs under rootless because a
190+
service-user assertion silently skipped it.
191+
192+
## Value bar
193+
194+
Tests justify their maintenance cost by protecting behavior, a credible
195+
regression, or an independently meaningful contract. In an audit, an existing
196+
test that must change for behavior-preserving source reorganization is
197+
suspect, not automatically deletable; the authoring gate still rejects new
198+
ones.
199+
200+
Before judging a candidate, read the complete test and its production owner,
201+
entry point, callers, callees, sibling implementations across drivers,
202+
overlapping tests at other layers, CI routing (which `mise run` task and which
203+
`.github/workflows/*.yml` job actually executes it — a test can be well-formed
204+
and simply never run; check `required-features` and workflow env vars, not
205+
just the test file), and relevant history. Read root and scoped `AGENTS.md`
206+
files first. When the test claims dependency-backed behavior (a Podman API
207+
shape, a cgroup value, a systemd unit's behavior), verify it against the real
208+
dependency or its documentation directly — don't take the mock's word for it.
209+
210+
## Discovery
211+
212+
Keep discovery read-only and report evidence before editing. For broad scope,
213+
run parallel discovery lanes when available, one per surface:
214+
215+
- Rust unit tests: `cargo nextest run --config-file .config/nextest.toml` (or
216+
`mise run test:rust`) per crate under audit;
217+
- `e2e/rust/tests/`: `cargo check --manifest-path e2e/rust/Cargo.toml
218+
--all-targets --features <e2e-driver-feature>` to enumerate what's gated
219+
behind which feature, then cross-reference against
220+
`.github/workflows/branch-e2e.yml` and `tasks/test.toml` to confirm the
221+
feature actually gets exercised in CI (it may not — see the Podman `E2E_FEATURES`
222+
empty-string wiring bug found in #3663, where a real, well-written test was
223+
silently skipped by an env var default);
224+
- conformance suite: `crates/openshell-conformance/src/scenarios/`;
225+
- `tests/ansible/` + `tests/suites/drivers/*`: read `tests/config.nix`'s
226+
`testsuites` list to find every playbook a suite actually runs, not just the
227+
ones with an obvious name;
228+
- Python: `mise run test:python` (`uv run pytest python/`);
229+
- TypeScript SDK: `mise run sdk:ts:test`;
230+
- Go SDK: `mise run go:test` / `go:test:integration`;
231+
- a cross-cutting pattern sweep (grep for the junk patterns above across
232+
whichever surfaces are in scope).
233+
234+
Outside campaign mode, prefer a few high-confidence candidates over a large
235+
speculative inventory. Hunt for the [junk patterns](#junk-patterns).
236+
237+
## Retention bar
238+
239+
Keep a test when it independently enforces a public API, gRPC/proto contract,
240+
`ComputeDriver` guarantee, gateway config schema, policy/OPA contract,
241+
security boundary, sandbox lifecycle invariant, SDK (Go/TypeScript/Python)
242+
contract, or CI/release artifact contract. Also keep:
243+
244+
- call ordering when order is observable behavior;
245+
- regressions with a credible failure mode;
246+
- source inspection when it is the cheapest independent guard: it fails when
247+
the contract changes (a config key, a wire byte, a CLI flag) and survives an
248+
identifier-only refactor;
249+
- a retained test that fails on the baseline: treat it as a possible product
250+
bug, reproduce it, and repair the owner rather than deleting it.
251+
252+
Static or slow is not a deletion reason — a `tests/ansible` VM suite is slow
253+
by nature and may still be the only real-daemon proof that exists. A test that
254+
resembles implementation may still be the independent contract; prove
255+
otherwise before removing it.
256+
257+
## Candidate evidence
258+
259+
Record every field below before editing. A missing field means the candidate
260+
is not ready for deletion:
261+
262+
- exact test name and location (crate/file, or `e2e/rust/tests/<file>.rs`, or
263+
`tests/ansible/playbooks/...`);
264+
- what failure it can actually detect;
265+
- non-test callers of the covered production or support seam;
266+
- stronger remaining owner-boundary proof, or why no proof is needed;
267+
- whether it currently runs in CI at all (check the workflow, not just the
268+
feature gate — see Discovery above) and what happens if you delete it
269+
versus if you fix its wiring instead;
270+
- relevant history and the reason the test or seam exists;
271+
- production or test-support deletion unlocked;
272+
- risk and the focused validation command.
273+
274+
## Edit shape
275+
276+
Choose one coherent owner-boundary batch, scoped to one crate, one SDK, or one
277+
driver's `e2e/rust/tests/` files at a time. Delete obsolete test-only exports,
278+
globals, wrappers, and dead production paths instead of preserving aliases.
279+
Move retained regressions to their canonical owners (conformance suite if the
280+
contract is driver-agnostic; the driver-specific file if it's genuinely
281+
driver-local). Consolidate repeated per-driver assertions of the same
282+
`ComputeDriver` contract into one conformance scenario.
283+
284+
Prefer net-negative production LOC. Do not add replacement tests that restate
285+
the same implementation, and do not convert uncertain candidates into cleanup
286+
to increase deletion counts. If an audit surfaces mock-only coverage of a
287+
real-dependency contract, that is a coverage-gap finding, not a deletion — say
288+
so explicitly in the handoff rather than silently leaving it.
289+
290+
## Validation
291+
292+
Follow the language's own pre-commit gate, not just the test itself — a
293+
passing test in isolation can still fail `cargo fmt --all -- --check` or
294+
`cargo clippy`.
295+
296+
1. Run the smallest owner and sibling tests directly (`cargo nextest run -p
297+
<crate> <filter>`, `cargo test --manifest-path e2e/rust/Cargo.toml --test
298+
<name> --features <feature>`, `uv run pytest python/<path>`, etc.) before
299+
running anything broader.
300+
2. For e2e tests that need a real driver and are runnable locally (Podman is
301+
commonly available; Docker/Kubernetes/VM often are not in a given
302+
environment), actually build and run them rather than trusting a compile
303+
check — `cargo build -p openshell-driver-podman` then `cargo test
304+
--manifest-path e2e/rust/Cargo.toml --test <name> --features e2e-podman`
305+
is cheap and catches wiring bugs a `cargo check` won't.
306+
3. For removed source greps or plan assertions, run the executable script or
307+
dry-run that owns the real contract.
308+
4. Run `cargo fmt --all -- --check` (Rust) and the equivalent formatter for
309+
whatever language changed, then `git diff --check`.
310+
5. Run `mise run pre-commit` once per coherent batch, not after every single
311+
file edit — it recompiles/clippies the full Rust workspace and is
312+
expensive; batch changes and run it before committing, not iteratively.
313+
6. Inspect `git diff --numstat`; report production/tooling separately from
314+
tests and test support.
315+
316+
## Check for overlap before filing an issue or opening a PR
317+
318+
Before filing a new issue for a finding:
319+
320+
- Search issues with `--state all` (not just open) and more than one phrasing
321+
— component name, file path, symptom, and subsystem name can each surface
322+
different matching issues.
323+
- If something adjacent but not identical exists, say so in the new issue
324+
rather than ignoring it or treating it as a duplicate.
325+
326+
Before opening a PR, or starting implementation:
327+
328+
- Check open PRs for overlapping files or behavior, not just titles
329+
(`gh pr list --search`, then `gh pr diff <number>` on anything adjacent) — a
330+
PR's title can be unrelated while its diff covers the same ground.
331+
- If an in-flight PR already does part of the work, cross-reference it and
332+
narrow scope instead of duplicating it.
333+
- Re-check if meaningful time passed since filing — overlap can appear later.
334+
335+
## Landing and continuation
336+
337+
Commit, push, open a PR, or land only when authorized — this repo requires
338+
explicit permission before `git push` (see the repository's own CLAUDE.md
339+
push-workflow rules) and only creates commits when asked. Use
340+
`create-github-pr` for PR mechanics once authorized. Scope one coherent PR per
341+
audited surface; after landing, rerun read-only discovery for the next
342+
high-confidence batch rather than queuing speculative follow-ups.
343+
344+
A new or substantially changed test-audit finding that reveals a real product
345+
gap (not a junk test, but missing coverage the audit surfaced) belongs in its
346+
own issue via `create-github-issue` or `create-spike` — do not fold it into
347+
the audit PR. Check for overlap first, per the section above, before filing
348+
it.
349+
350+
## Handoff
351+
352+
Report:
353+
354+
- root cause and removed low-value categories, by surface (unit / e2e /
355+
conformance / tmachine / SDK);
356+
- production owner simplifications;
357+
- retained false positives and why they remain valuable;
358+
- any mock-only-coverage findings surfaced but not fixed here (coverage gaps,
359+
not junk — flag for a separate issue);
360+
- focused and full proof actually run, and what could not be run locally
361+
(e.g. drivers without local hardware/daemon access) versus what was
362+
verified end-to-end;
363+
- production versus test LOC;
364+
- PR and merge state;
365+
- named follow-ups.

0 commit comments

Comments
 (0)