Skip to content

Commit 2ac62f0

Browse files
committed
ci(coverage): fail a PR whose touched functions score CRAP 30 or more
The PR step ran the verdict with `continue-on-error`, so a function over the threshold only annotated the diff. It now gates the job: the score itself, or a report that cannot answer the question, is red. The `main` step keeps `continue-on-error`, because that report is the whole-repo backlog `main` carries on purpose, and both keep needing `run-coverage` to have succeeded — without a coverage map there is no verdict to report, only the coverage step's own red. CONTRIBUTING.md gains how to read a failure, since a PR now stops on two different reasons: debt a touched function owes, and a stale coverage run that has to be re-run. The end-to-end cases drive the real entrypoint against a real git repo and real fallow, which is the only way to prove rename-aware attribution and the map cross-check agree with the model. Closes #3426
1 parent e45cb65 commit 2ac62f0

4 files changed

Lines changed: 370 additions & 23 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -335,21 +335,22 @@ jobs:
335335
--base
336336
${{ github.event.pull_request.base.sha }}
337337
338-
# Report-only: scores the functions this PR touches with the coverage map the coverage step
339-
# wrote (fallow-crap.json), as a job summary and warning annotations. It never fails the job:
340-
# a tooling error marks only this step. `!cancelled()` keeps it running after a failed
341-
# changed-line gate.
342-
- name: Report coverage-based CRAP
338+
# Blocking: scores the functions this PR touches with the coverage map the coverage step wrote
339+
# (fallow-crap.json) and fails the job when one is at or above the threshold, or when the map
340+
# cannot be trusted to answer. `!cancelled()` keeps it running after a failed changed-line
341+
# gate; it still needs `run-coverage` to have succeeded, because without a coverage map there
342+
# is no verdict to report — only the coverage step's own red.
343+
- name: Check coverage-based CRAP of touched functions
343344
if: ${{ !cancelled() && steps.run-coverage.outcome == 'success' && github.event_name == 'pull_request' }}
344-
continue-on-error: true
345345
uses: ./.github/actions/run-gate
346346
with:
347347
gate: coverage-crap
348348
args: |
349349
--base
350350
${{ github.event.pull_request.base.sha }}
351351
352-
# On main, the same report covers every function: the backlog of untested complex code.
352+
# Report-only on main: the same scoring over every function is the backlog of untested
353+
# complex code, which `main` carries on purpose, so it stays advisory.
353354
- name: Report coverage-based CRAP for the whole repository
354355
if: ${{ !cancelled() && steps.run-coverage.outcome == 'success' && github.event_name == 'push' }}
355356
continue-on-error: true

‎CONTRIBUTING.md‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -232,8 +232,9 @@ Useful direct entry points:
232232
- `pnpm perf --platform ios` or `pnpm perf --platform android` — device performance harness
233233
- `pnpm check:fallow --base origin/main` — changed-code quality gate
234234
- `pnpm check:coverage-crap --base origin/main` — CRAP from measured coverage for the functions a
235-
change touches (run `pnpm test:coverage` first; `--all` scans every function). Both report scores
236-
at or above 30. `check:fallow` leaves CRAP off because it has no coverage.
235+
change touches; it fails when one scores 30 or more, so run `pnpm test:coverage` first. `--all`
236+
reports every function and never fails: that is `main`'s backlog. Both list the file-wide
237+
`fallow-crap.json` exemptions they honoured. `check:fallow` leaves CRAP off without coverage.
237238
- `pnpm fallow:all` — full-tree audit, including grandfathered baseline findings
238239
- `pnpm fallow:baseline` — intentionally regenerate both reviewed Fallow baselines
239240

@@ -242,6 +243,17 @@ mutation/fuzz lanes, contention policy, and test-speed rules. For real devices,
242243
[`docs/agents/device-verification.md`](docs/agents/device-verification.md); a fixture-backed test does
243244
not prove that a native path was active.
244245

246+
### Reading a coverage-CRAP failure
247+
248+
`check:coverage-crap --base <ref>` scores the functions whose lines your diff adds, so it also
249+
fails when it cannot score them. It prints `no trustworthy verdict` and names the files when
250+
`coverage/coverage-istanbul.json` is missing, malformed, or records fewer files than the same run's
251+
`coverage/lcov.info` enumerated; when a file you touched was dropped as corrupt or never recorded;
252+
or when a touched function postdates the map and would be scored from complexity alone. Every one of
253+
those is a stale coverage run, not a debt you owe: re-run `pnpm test:coverage` on the commit and
254+
re-check. What does count as debt is editing a function that already scores 30 or more — the score
255+
does not have to be new. Moving a file without editing its lines touches none of its functions.
256+
245257
## Test app and Maestro compatibility
246258

247259
The Expo fixture app owns its setup, simulator/device, Metro, replay, and Maestro instructions in

‎docs/agents/testing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ pnpm gate --help
2222
```
2323

2424
`check:affected --run` reports coverage obligations without instrumenting; when related tests are
25-
selected, it runs one capped `vitest related` command. Run local coverage to investigate a specific
26-
coverage question or CI failure.
25+
selected, it runs one capped `vitest related` command. Run local coverage, then
26+
`check:coverage-crap` gates the touched functions (CONTRIBUTING.md).
2727

2828
Two selection traps recur:
2929

0 commit comments

Comments
 (0)