Skip to content

Commit 423927f

Browse files
authored
chore(mutation): shrink to report-only — drop the ratchet, baseline and graduation (#1457, #1781) (#1828)
* chore(mutation): shrink the lane to report-only (#1457, #1781 wave 2) The mutation harness's two real catches (#1474, #1475) both came from humans reading the weekly score report. The ratchet half never operated: the baseline was committed exactly twice (8cce0ef, 60400d0), both times with `stableRuns: 0, gating: false`, and was never updated after the very fixes it triggered — the weekly job computed a new baseline and then `git checkout --`d it, uploading a proposal nobody applied in 3+ weeks. A gate nobody arms is harness weight; the report is the part that paid. Deletes ratchet.ts + ratchet.test.ts, mutation-baselines/, and every baseline/graduation/gating path in run.ts (`--update`, `mutation:baseline`). run.ts now exits non-zero only on a harness failure, never on a score. The report renders the per-kernel table (kernel, score, killed, survived, total, timeouts) plus the surviving mutants a strengthening PR works from. Kernel scoping stays: stryker.config.json and KERNEL_MODULES are untouched. * fix(mutation): restore denominator coverage and publish the table before judging the shard set Review of #1828: - `report.test.ts` re-asserts that Ignored/CompileError/RuntimeError leave the denominator — the one behaviour `ratchet.test.ts` covered and nothing replaced. A `tally()` edit that counted tool noise would have deflated every published score with a green `mutation:test`. - `assertShardsCoverModules` now runs after `emit()`, so an incomplete shard set still publishes the kernels that completed instead of only an error string. This makes the workflow comments' claim about the job summary true rather than re-wording them down. * chore(mutation): trigger the affected lane on exactly the paths that can select mutants The PR lane returns an empty matrix unless the diff touches the harness, so the kernel-source and `**/*.test.ts` triggers only bought a 1-4 min no-op job on ~96% of PRs. `on.pull_request.paths` is now exactly `LANE_TOOLING` plus the workflow file, asserted in both directions by workflow.test.ts against the exported constant — a missing path would let a harness change merge unproven, an extra one starts a job that can only answer `[]`. Also drops the workflow header's contradictory scope paragraph: it claimed the lane selects on kernel sources and any test reaching one, which has not been true since the ratchet went. * fix(mutation): score and publish a short shard set before failing on the count The expected-count check ran inside readShardedReports, before anything was summarized, so on the weekly's real `--expect-shards 10` one dead shard threw away the nine that had reported — the earlier reorder only moved the zero-mutants check. The merge now returns the shard count, and both verdicts run after emit() with the same exit code and `score` stage. Regression uses the weekly argument shape (`--expect-shards 10`, one shard present) and asserts the reporting kernel's row reaches stdout while the run still fails.
1 parent 9d6154e commit 423927f

21 files changed

Lines changed: 402 additions & 880 deletions

‎.github/workflows/mutation-affected.yml‎

Lines changed: 23 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,43 +2,33 @@ name: Mutation Affected
22

33
# PR-side half of the decision-kernel mutation lane (issue #1415).
44
#
5-
# Graduation, not a flag day: this job runs on every PR that touches a kernel
6-
# module, but `scripts/mutation/run.ts` only exits non-zero once the committed
7-
# baseline reports `gating: true` — earned by two consecutive stable weekly
8-
# sweeps (mutation-weekly.yml). Until then it is a report with the same numbers,
9-
# so the gate's first failing day is not also its first running day.
5+
# Advisory only — the lane reports and never gates (#1457). `scripts/mutation/run.ts`
6+
# exits non-zero on a harness failure, never on a score.
107
#
11-
# Cost control: before graduation an affected run is a report nobody acts on, so
12-
# `select` returns an empty matrix and no mutants run — except when the diff
13-
# touches the lane's own tooling, the one case where a pre-graduation run buys
14-
# something (the gate has to be proven before it can bite).
8+
# Cost control: the kernel report that pays is the weekly sweep, so this lane
9+
# spends mutants on one thing only — proving the harness still runs end to end
10+
# when the harness itself changes. `select` therefore returns an empty matrix
11+
# unless the diff touches the lane's own sources. Selecting on derived kernel
12+
# ownership instead would run the full ten-shard sweep on 24 of the last 40
13+
# merged PRs.
1514
#
16-
# Scope is the AFFECTED modules only, and affectedness is DERIVED from the import
17-
# graph (scripts/mutation/ownership.ts): a kernel source, or any test that reaches
18-
# one. Reaching a kernel is a superset of killing its mutants, so the `select` job
19-
# frequently returns several modules — sharded like the weekly sweep so the PR's
20-
# wall clock is one module, not their sum. The weekly run stays the full sweep.
15+
# On a harness diff the matrix is the canary (`LANE_CANARY`, the registry's
16+
# cheapest real sweep — lane sources own no kernel, so derivation alone would
17+
# select nothing) plus any kernel that same diff reaches through the import
18+
# graph (scripts/mutation/ownership.ts): a kernel source, or a test that reaches
19+
# one. Selected modules are sharded like the weekly sweep, so the PR's wall
20+
# clock is one module rather than their sum.
2121

2222
on:
2323
pull_request:
2424
paths:
25-
# Kernel sources, every src/package test (ownership is derived, so any
26-
# test may own a kernel — `select` decides, not this filter), and the
27-
# lane's own tooling. scripts/mutation/workflow.test.ts asserts this
28-
# covers the registry.
29-
- 'packages/kernel/src/errors.ts'
30-
- 'src/daemon/ref-frame.ts'
31-
- 'src/commands/interaction/runtime/settle.ts'
32-
- 'src/utils/scroll-edge-state.ts'
33-
- 'packages/selectors/src/**'
34-
- 'packages/ad-script/src/internal/target-annotation-serde.ts'
35-
- 'src/snapshot/snapshot-occlusion.ts'
36-
- 'src/**/*.test.ts'
37-
- 'packages/*/src/**/*.test.ts'
25+
# Exactly `LANE_TOOLING` in scripts/mutation/run.ts plus this file — the
26+
# only diffs that can produce a non-empty matrix, asserted both ways by
27+
# scripts/mutation/workflow.test.ts. A kernel or test path here would only
28+
# start a job that selects nothing.
3829
- 'scripts/mutation/**'
3930
- 'scripts/lib/**'
4031
- 'stryker.config.json'
41-
- 'mutation-baselines/**'
4232
- '.github/workflows/mutation-affected.yml'
4333

4434
permissions:
@@ -64,7 +54,7 @@ jobs:
6454
- name: Setup toolchain
6555
uses: ./.github/actions/setup-node-pnpm
6656

67-
- name: Ratchet self-test
57+
- name: Harness self-test
6858
uses: ./.github/actions/run-gate
6959
with: { gate: mutation-model }
7060

@@ -138,7 +128,7 @@ jobs:
138128
.tmp/mutation/lane-envelope.json
139129
if-no-files-found: warn
140130

141-
ratchet:
131+
report:
142132
name: Affected decision-kernel mutants
143133
needs: [select, mutants]
144134
if: always() && needs.select.result == 'success'
@@ -162,7 +152,8 @@ jobs:
162152

163153
# No shards means no affected kernel: run.ts reports "nothing to mutate"
164154
# and still writes the envelope, so the lane is never silently absent.
165-
- name: Ratchet the affected modules
155+
# Either invocation writes the score table to $GITHUB_STEP_SUMMARY.
156+
- name: Score the affected modules
166157
run: |
167158
if [ -d .tmp/mutation/shards ]; then
168159
expected=$(echo '${{ needs.select.outputs.modules }}' | jq length)
@@ -177,7 +168,7 @@ jobs:
177168
if: failure()
178169
run: |
179170
pnpm gate mutation --affected --fail-envelope \
180-
"affected ratchet failed before producing a verdict (run ${{ github.run_id }})" || true
171+
"affected report failed before producing a score table (run ${{ github.run_id }})" || true
181172
182173
- name: Upload mutation report
183174
if: always()

‎.github/workflows/mutation-weekly.yml‎

Lines changed: 16 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,11 @@ name: Mutation Weekly
88
# Sharded one job per module: the whole sweep is ~2,150 mutants, and the selector
99
# module alone is ~1,280 of them, so a single job would sit near the 30-minute
1010
# acceptance budget on an ubuntu runner. The shards' JSON reports are merged into
11-
# one verdict by the ratchet job (`--report-dir`), so the ratchet still sees a
12-
# full sweep.
11+
# one score table by the report job (`--report-dir`).
1312
#
14-
# The lane reports; it does not commit. The proposed baseline rides in the
15-
# artifact and applying it is a reviewed `pnpm mutation:baseline` commit — a score
16-
# can never lower itself. Gating (after two consecutive stable weekly runs) is
17-
# enforced on PRs by mutation-affected.yml.
13+
# The lane reports and never gates (#1457): the per-kernel table lands in the job
14+
# summary and the artifact, and a low score is an input for a human-authored
15+
# test-strengthening PR (#1474, #1475 were written that way).
1816

1917
on:
2018
schedule:
@@ -90,8 +88,8 @@ jobs:
9088
.tmp/mutation/lane-envelope.json
9189
if-no-files-found: warn
9290

93-
ratchet:
94-
name: Mutation ratchet
91+
report:
92+
name: Mutation score report
9593
runs-on: ubuntu-latest
9694
needs: shard
9795
if: always()
@@ -103,7 +101,7 @@ jobs:
103101
- name: Setup toolchain
104102
uses: ./.github/actions/setup-node-pnpm
105103

106-
- name: Ratchet self-test
104+
- name: Harness self-test
107105
uses: ./.github/actions/run-gate
108106
with: { gate: mutation-model }
109107

@@ -113,9 +111,13 @@ jobs:
113111
pattern: mutation-shard-*
114112
path: .tmp/mutation/shards
115113

116-
# run.ts writes the markdown verdict to $GITHUB_STEP_SUMMARY when the
114+
# run.ts writes the per-kernel score table to $GITHUB_STEP_SUMMARY when the
117115
# runner exports it, so the summary and the artifact carry the same numbers.
118-
- name: Ratchet the merged sweep and propose the next baseline
116+
# It publishes the table before judging the shard set, so even the one
117+
# failure it can raise — an incomplete shard set — still reports the kernels
118+
# that completed. A red lane here always means the sweep did not happen,
119+
# never a low score.
120+
- name: Score the merged sweep
119121
uses: ./.github/actions/run-gate
120122
with:
121123
gate: mutation-check
@@ -124,20 +126,14 @@ jobs:
124126
.tmp/mutation/shards
125127
--expect-shards
126128
10
127-
--update
128129
129-
- name: Stage the proposed baseline artifact
130-
run: |
131-
cp mutation-baselines/decision-kernels.json .tmp/mutation/proposed-baseline.json
132-
git checkout -- mutation-baselines/decision-kernels.json
133-
134-
# The self-test and the artifact download both run before the ratchet, so a
135-
# failure there would otherwise leave the aggregate lane with no envelope.
130+
# The self-test and the artifact download both run before the scoring step,
131+
# so a failure there would otherwise leave the aggregate lane with no envelope.
136132
- name: Record a failed lane envelope
137133
if: failure()
138134
run: |
139135
pnpm gate mutation --fail-envelope \
140-
"weekly ratchet job failed before producing a verdict (run ${{ github.run_id }})" || true
136+
"weekly report job failed before producing a score table (run ${{ github.run_id }})" || true
141137
142138
# Freshness/drift telemetry (#1430): the envelope states commit, Stryker
143139
# version, config hash, duration and result, so a lane going dark or a tool
@@ -167,5 +163,4 @@ jobs:
167163
path: |
168164
.tmp/mutation/shards
169165
.tmp/mutation/lane-envelope.json
170-
.tmp/mutation/proposed-baseline.json
171166
if-no-files-found: warn

‎.oxfmtrc.json‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
"node_modules/**",
1111
"**/*.md",
1212
"scripts/maestro-conformance/corpus/**",
13-
"fallow-baselines/**",
14-
"mutation-baselines/**"
13+
"fallow-baselines/**"
1514
]
1615
}

‎docs/agents/testing.md‎

Lines changed: 32 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about
4444
| Daemon RPC wire surface (the declarations listed in `test/wire-compat/surface.ts` — JSON-RPC envelope, request/response/error/artifact/progress framing, `/health` payload, HTTP auth headers) | `pnpm exec vitest run --project unit-core test/wire-compat` holds the ledger to its source and prints the digest to paste; `pnpm check:daemon-wire-compat` compares it against the last released tag and requires a `DAEMON_RPC_PROTOCOL_VERSION` bump or a `compatibleChanges` ack for the drift. Read ADR 0006 to decide which; `test/wire-compat/README.md` walks both |
4545
| Anything in `src/`, `test/` | `pnpm format` (`skills/` is Markdown-only guidance: oxfmt ignores `**/*.md`, and the affected-check selector classifies it docs-only) |
4646
| Workspace package source (`packages/*/src/**`) | Root format/lint/typecheck plus layering (R11 package-boundaries); Vitest resolves affected tests through the module graph; package manifests/tsconfigs fail open to the full set |
47-
| A decision kernel or its tests (`packages/kernel/src/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `packages/selectors/src/`) | `pnpm mutation:affected --base origin/main` (minutes; GitHub runs it per PR — see the mutation ratchet section) |
47+
| A decision kernel or its tests (`packages/kernel/src/errors.ts`, `src/daemon/ref-frame.ts`, `src/commands/interaction/runtime/settle.ts`, `src/utils/scroll-edge-state.ts`, `packages/selectors/src/`) | `pnpm mutation:run --modules <kernel>` (minutes; optional — the lane reports, it never gates, see the mutation section) |
4848

4949
Two traps worth naming:
5050

@@ -243,7 +243,7 @@ The three facts the manifest cannot derive live together in `scripts/gate/declar
243243
coverage wrapper, one reporting-only `test:*` script, and the Android replay owner hidden inside a
244244
third-party action's `script:` input.
245245

246-
## Mutation ratchet over decision kernels
246+
## Mutation report over decision kernels
247247

248248
Mutation score is the mechanical answer to "is this test load-bearing or decorative". A full-suite
249249
sweep is unaffordable, so the scope is an enumerated list of pure decision kernels — modules where a
@@ -252,46 +252,47 @@ surviving mutant means a silently wrong agent-facing decision. The registry
252252
are asserted against it, and PR-affected selection maps changed files through it. Modules that spawn
253253
subprocesses or wait real time stay out by construction.
254254

255+
Mutation runs **report-only** on the seven decision kernels — a weekly full sweep plus a per-PR
256+
affected sweep. It never gates: `scripts/mutation/run.ts` exits non-zero on a harness failure (a
257+
missing report, an incomplete shard set, a bad argument) and never on a score. A low score is an
258+
input for a human-authored test-strengthening PR, which is exactly how #1474 and #1475 were written.
259+
The ratchet, baseline file and graduation rule this lane used to carry were deleted in #1457: in
260+
three weeks nobody applied a baseline, so the gate half never operated while the report half was
261+
paying.
262+
255263
```sh
256-
pnpm mutation:test # ratchet self-test (fast, no Stryker)
264+
pnpm mutation:test # harness self-test (fast, no Stryker)
257265
pnpm mutation:run --modules selectors # one module locally (~7 min for selectors)
258-
pnpm mutation:check # ratchet an existing .tmp/mutation/mutation.json
259-
pnpm mutation:baseline # full sweep, then record it (reviewed commit)
266+
pnpm mutation:check # score an existing .tmp/mutation/mutation.json
260267
```
261268

262269
- **Weekly full sweep** (`.github/workflows/mutation-weekly.yml`) runs `shardMatrix()` from the
263270
registry: one job per module, except modules that declare a `shards` count and are sliced with
264-
`--shard i/n` (selectors is ~1,280 mutants, well past the 30-minute budget in one job). The ratchet
265-
merges the shard reports (`--report-dir`) for one verdict and requires the full set
266-
(`--expect-shards`), so a dead shard fails the lane instead of scoring its module as 0. Results are
267-
reported as a job summary plus an artifact. It never commits: the proposed baseline rides in the
268-
artifact, and applying it is a reviewed `pnpm mutation:baseline` commit, so a score cannot lower
269-
itself.
270-
- **PR lane** (`.github/workflows/mutation-affected.yml`) derives the affected shard matrix
271-
(`--list-affected`) and merges the shards into one verdict. Before graduation the matrix is empty —
272-
a report nobody acts on is not worth the runner minutes — unless the diff touches the lane's own
273-
tooling, the one pre-graduation run that buys something: the gate has to be proven before it bites.
274-
Lane sources own no kernel, so that exception adds `LANE_CANARY` (`kernel-errors`, the registry's
275-
cheapest real sweep) to whatever the diff derives; otherwise it would select zero mutants and prove
276-
nothing. `scripts/mutation/selection.test.ts` drives both halves of the rule through the real CLI
277-
against a throwaway worktree commit.
278-
- **Ratchet**: scores may only rise. `mutation-baselines/decision-kernels.json` records the
279-
high-water score per module plus the Stryker version and config content hash that produced it, so a
280-
score change caused by a tool/config change is reported as provenance drift, never as a
281-
test-strength regression.
282-
- **Graduation, not a flag day**: gating is off until two consecutive comparable weekly sweeps pass
283-
(`stableRuns`/`requiredStableRuns` in the baseline); the PR job starts selecting modules — and
284-
failing on them — once the committed baseline says `gating: true`. A regression or provenance drift
285-
resets the counter.
271+
`--shard i/n` (selectors is ~1,280 mutants, well past the 30-minute budget in one job). The report
272+
job merges the shard reports (`--report-dir`) into one per-kernel table — kernel, score, killed,
273+
survived, total, timeouts, plus the surviving mutants — and requires the full set
274+
(`--expect-shards`), so a dead shard fails the lane instead of publishing its module as 0%. The
275+
table lands in the job summary and the artifact.
276+
- **PR lane** (`.github/workflows/mutation-affected.yml`) exists to prove the harness still runs end
277+
to end when the harness changes, so its matrix (`--list-affected`) is empty unless the diff touches
278+
the lane's own sources: the weekly sweep is the kernel report, and selecting on derived kernel
279+
ownership would run the full ten-shard sweep on 24 of the last 40 merged PRs for a report nobody
280+
gates on. The workflow triggers on exactly those sources (`LANE_TOOLING` in `run.ts`, asserted both
281+
ways by `workflow.test.ts`), so a PR that could only select `[]` never starts the job. Lane sources
282+
own no kernel, so a harness diff adds `LANE_CANARY` (`kernel-errors`, the registry's cheapest real
283+
sweep) to whatever kernels that same diff derives; otherwise it would select zero mutants and prove
284+
nothing. It reports the same table. `scripts/mutation/selection.test.ts` drives both halves of the
285+
rule through the real CLI against a throwaway worktree commit.
286+
- **Provenance**: every report and lane envelope carries the Stryker version and the config content
287+
hash that produced it, so scores measured across a tool or config change are not read as
288+
test-strength change.
286289
- **Test scope** is derived from Vitest's module graph (`vitest related` over the mutated files), the
287290
same delegation `pnpm check:affected` uses; see `scripts/mutation/test-scope.ts` for the three
288291
groups it drops and why dropping them cannot hide a surviving mutant.
289292
- **Test ownership is derived, never listed** (`scripts/mutation/ownership.ts`): a test owns every
290293
kernel its imports reach, so `src/__tests__/daemon-error.test.ts` selects `kernel-errors` through
291-
`src/daemon.ts` without naming it. A listed set of test files would silently omit exactly those
292-
indirect tests and rot as tests are added — weakening one would skip the ratchet. Reaching a kernel
293-
is a superset of killing its mutants, so the PR lane over-selects on purpose and shards the
294-
selected modules; a false positive costs runner minutes, a false negative costs the gate. Non-kernel
294+
`src/daemon.ts` without naming it. Reaching a kernel is a superset of killing its mutants, so the
295+
derivation over-selects on purpose; it applies to the modules a lane-tooling diff selects. Non-kernel
295296
*sources* are not owned: they can only move a score through those tests, and the weekly sweep
296297
re-measures the whole surface.
297298
- **Lane envelope** (`scripts/lib/lane-envelope.ts`, issue #1430): every run writes

‎mutation-baselines/decision-kernels.json‎

Lines changed: 0 additions & 71 deletions
This file was deleted.

‎package.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,6 @@
119119
"size": "node scripts/size-report.mjs",
120120
"perf": "node --experimental-strip-types scripts/perf/run.ts",
121121
"mutation:run": "node --experimental-strip-types scripts/mutation/run.ts",
122-
"mutation:baseline": "node --experimental-strip-types scripts/mutation/run.ts --update",
123122
"mutation:check": "node --experimental-strip-types scripts/mutation/run.ts --no-run",
124123
"mutation:affected": "node --experimental-strip-types scripts/mutation/run.ts --affected",
125124
"mutation:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/mutation/*.test.ts",

0 commit comments

Comments
 (0)