Skip to content

Commit f149a2a

Browse files
cliffhallclaude
andcommitted
ci(coverage): enforce the TypeScript per-file coverage gate in CI and local:gate
Adds a parallel `coverage` job to typescript.yml (one leg per TypeScript server, beside `validate`, as the Inspector's CI does) and a `coverage` stage to `local:gate:stages` right after `validate`. The workflow guard derives the stage from the new CI job, and its stage list now pins `coverage` by name. Documents the stage in docs/quality-gate.md, the TypeScript half of the coverage rule in AGENTS.md, how to diagnose a red coverage stage in the pre-push-gate skill, and the in-process harness and coverage gate in the testing skill. The docs describe the state after the four per-server coverage PRs for #4854 land, which add the 90/90/90/90 per-file thresholds. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: cliffhall <[email protected]>
1 parent 004608f commit f149a2a

8 files changed

Lines changed: 208 additions & 109 deletions

File tree

‎.claude/skills/pre-push-gate/SKILL.md‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,40 @@ the server's own `tsc`. An error in a test file only shows in `typecheck`.
154154
Re-run just that suite to iterate: `npm test -w src/<server>`. Read the failure
155155
before changing a timeout: an assertion that races is a bug in the test.
156156

157+
### `coverage`
158+
159+
A TypeScript file is below 90% on lines, statements, functions or branches.
160+
Vitest names each file and dimension:
161+
162+
```
163+
ERROR: Coverage for branches (71.42%) does not meet global threshold (90%) for roots-utils.ts
164+
```
165+
166+
("global threshold" is Vitest's wording; with `perFile: true` it is checked
167+
per file.) The root `npm run coverage` carries on into the remaining
168+
workspaces after a failure, so **the failing report is not the last thing
169+
printed**: find the `npm ERR!` block, which names the workspace
170+
(`in workspace: @modelcontextprotocol/server-<name>`), and read the `ERROR:`
171+
lines just above it.
172+
173+
1. Re-run that server alone: `npm run coverage -w src/<server>`.
174+
2. Read its per-file table. The `Uncovered Line #s` column is where the
175+
missing tests go (a long list is truncated to `...`, and a long file name
176+
to `...-name.ts`; open `coverage/index.html` in the workspace, or narrow
177+
the run, for the full list). A branch shortfall with no uncovered lines is
178+
the untaken side of a condition, `??`, `?.` or default parameter.
179+
3. **Write the tests.** Drive what a client sees through a `Client` over the
180+
in-memory transport; call a helper directly only for its logic (`testing`
181+
skill).
182+
4. Only for code that genuinely cannot run, annotate it at the source with
183+
its reason, `/* v8 ignore next -- <reason> */` (or `start`/`stop` around a
184+
block). A branch that is merely awkward to reach is not unreachable.
185+
186+
**Never lower a threshold, drop `perFile`, or widen the coverage `exclude`**
187+
to get green; the gate is the rule in `AGENTS.md` **Always test new or
188+
modified code**. A failing _test_ in this stage is an ordinary test failure:
189+
fix it as under `test`, above.
190+
157191
### `validate:py`
158192

159193
The summary at the end names the failing server, and the step that failed is
@@ -221,8 +255,7 @@ The details of the lease are in
221255

222256
## Not part of the gate
223257

224-
`npm run skills:eval` (real model calls), `npm run coverage` (a report; no
225-
threshold is enforced yet) and `npm run pack:verify` (installs each package's
226-
publish artifact and boots it; needs the network, and belongs to the release
227-
flow) are separate commands. Run the eval suite when you
228-
add a skill or change a description.
258+
`npm run skills:eval` (real model calls) and `npm run pack:verify` (installs
259+
each package's publish artifact and boots it; needs the network, and belongs to
260+
the release flow) are separate commands. Run the eval suite when you add a
261+
skill or change a description.

‎.claude/skills/pre-push-gate/evals/evals.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@
2323
"prompt": "verify:no-test-retries is failing in my gate run but I never added a test retry. What is it matching?",
2424
"expect": "pre-push-gate"
2525
},
26+
{
27+
"prompt": "The coverage stage of my gate run went red, but the last report it printed was all green. How do I find what failed and fix it?",
28+
"expect": "pre-push-gate"
29+
},
2630
{
2731
"prompt": "What is 17 multiplied by 23?",
2832
"expect": null

‎.claude/skills/testing/SKILL.md‎

Lines changed: 90 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -7,19 +7,19 @@ disable-model-invocation: false
77
# Testing
88

99
The **rules** are in [`AGENTS.md`](../../../AGENTS.md) under **Always test new
10-
or modified code**: new behavior comes with tests, test at the protocol level
11-
where you can, and keep expected error output off the console. This skill
12-
explains which harness each server has **today**, how to write a test with it,
13-
and which command runs what.
10+
or modified code**: new behavior comes with tests, every TypeScript file clears
11+
the per-file coverage gate, test at the protocol level where you can, and keep
12+
expected error output off the console. This skill explains the harness each
13+
server has, how to write a test with it, which command runs what, and how to
14+
clear the per-file coverage gate.
1415

1516
Driving a running server by hand, with the Inspector or an LLM client, is
1617
`/client-smoke`. It is evidence for a PR, not a substitute for a test.
1718

18-
⚠️ **The harnesses are uneven, and this page describes them as they are.** Only
19-
`everything` can be driven in-process today. A shared in-process harness for
20-
every server, and a per-file coverage threshold, are the work of #4854
21-
(TypeScript) and #4855 (Python); neither exists yet, and this skill changes
22-
when they land. Do not write a test that assumes them.
19+
⚠️ **The TypeScript and Python harnesses differ.** All four TypeScript servers
20+
are tested in-process, a real `Client` linked to the server over an in-memory
21+
transport (#4854). The Python servers' tests call their functions directly,
22+
and those servers have no coverage gate yet (#4855).
2323

2424
## Where a test goes
2525

@@ -35,18 +35,14 @@ anywhere else in a workspace is never run and reports nothing. Shared test
3535
helpers go in `__tests__/` without the `.test` suffix, as
3636
`src/everything/__tests__/helpers.ts` does.
3737

38-
## What each server's tests do today
38+
## What each server's tests do
3939

4040
| Server | How its tests reach the code |
4141
| --- | --- |
42-
| `everything` | Call a `register…` function with a mocked `McpServer`, capture the handler it registered, and invoke the handler directly. `__tests__/helpers.ts` types the captured handlers |
43-
| `filesystem` | Unit tests of `lib.ts` and the path helpers, plus an SDK `Client` over `StdioClientTransport` that spawns the built `dist/index.js` |
44-
| `sequentialthinking` | Unit tests of `lib.ts`, plus an SDK `Client` over stdio against `dist/index.js` |
45-
| `memory` | Direct calls on the exported `KnowledgeGraphManager` and `register…` functions, with a mocked server |
42+
| `everything`, `filesystem`, `memory`, `sequentialthinking` | An SDK `Client` connected to the server's `createServer()` over `InMemoryTransport`, in the vitest process; unit tests of the helper modules beside them; at most a thin spawn-based smoke of the built `dist/index.js` |
4643
| `fetch`, `git`, `time` | Direct calls on the functions in `server.py`, with `unittest.mock` |
4744

48-
No test in the repo uses an in-memory transport yet, and no Python test opens
49-
a `ClientSession`.
45+
No Python test opens a `ClientSession` yet.
5046

5147
## Choosing a harness for a new test
5248

@@ -59,16 +55,27 @@ Ask what the test must prove.
5955
error result, a capability, a notification): drive the server through a
6056
client. A handler called directly skips the SDK's input validation, output
6157
schema validation and error mapping, which is where the wire-level bugs in
62-
this repo have been. The regression tests in `filesystem` and
63-
`sequentialthinking` that go through a `Client` exist for exactly that
64-
reason.
58+
this repo have been.
6559

66-
Which client harness is available depends on the server.
60+
Which client harness is available depends on the language.
6761

68-
### `everything`: in-process, over an in-memory transport
62+
### TypeScript: in-process, over an in-memory transport
6963

70-
`src/everything/server/index.ts` exports `createServer()`, so a test can link a
71-
real `Client` to a real server with no process and no build:
64+
Each TypeScript server exports a `createServer(…)` factory and starts its
65+
transport only from a guarded `main()`, so importing it starts nothing and a
66+
test can link a real `Client` to a real server with no process and no build:
67+
68+
| Server | Factory | Takes |
69+
| --- | --- | --- |
70+
| `everything` | `server/index.ts` | nothing; returns `{ server, cleanup }` |
71+
| `filesystem` | `server.ts` | the allowed directories |
72+
| `memory` | `index.ts` | the graph file's path |
73+
| `sequentialthinking` | `index.ts` | nothing |
74+
75+
Each server's `__tests__/` already has a shared helper that does the
76+
connecting (and, where the server needs one, the temporary directory or file).
77+
**Use it** rather than writing the linking by hand again. What it does, shown
78+
for `everything`:
7279

7380
```ts
7481
import { describe, it, expect, beforeEach, afterEach } from "vitest";
@@ -112,61 +119,36 @@ describe("echo, over the protocol", () => {
112119
});
113120
```
114121

115-
Call `cleanup()` in `afterEach`: `createServer()` starts timers (the roots sync
116-
after `initialize`, the task store) that otherwise outlive the test. A tool
117-
registered by `registerConditionalTools` appears only when the `Client`
118-
declares the matching capability (`roots`, `sampling`, `elicitation`) in its
119-
constructor options.
120-
121-
### `filesystem`, `memory`, `sequentialthinking`: a client over stdio, against the build
122-
123-
These three build their server at module scope and connect stdio when
124-
`index.ts` is imported, so there is no factory to call and the in-memory
125-
recipe above does not apply to them. A protocol-level test spawns the built
126-
server instead:
127-
128-
```ts
129-
const distIndexPath = path.join(packageRoot, "dist", "index.js");
130-
const transport = new StdioClientTransport({
131-
command: process.execPath,
132-
args: [distIndexPath],
133-
stderr: "pipe",
134-
});
135-
const client = new Client({ name: "test-client", version: "0.0.0" });
136-
try {
137-
await client.connect(transport);
138-
// … assert …
139-
} finally {
140-
await client.close(); // this is what stops the child process
141-
}
142-
```
143-
144-
Closing in `finally` (or in `afterEach`, as the `filesystem` tests do) matters:
145-
a failed assertion that skips `close()` leaves the spawned server running.
146-
147-
`src/sequentialthinking/__tests__/input-schema.test.ts` is the reference.
148-
`stderr: "pipe"` keeps the server's startup banner off the console.
149-
150-
Give the server what it needs to start. `filesystem` takes its allowed
151-
directories as arguments (`args: [distIndexPath, testDir]`, with `testDir` a
152-
`realpath`-resolved temporary directory, as
153-
`src/filesystem/__tests__/structured-content.test.ts` does); with none, and a
154-
client that offers no Roots, every file operation is refused. Environment
155-
variables such as `MEMORY_FILE_PATH` go in the transport's `env` option: the
156-
child inherits only the SDK's default safelist (`PATH`, `HOME`, `USER` and the
157-
like), not the other variables of the test process.
158-
159-
⚠️ **These tests run the last build, not your edit.** `npm test` does not build.
160-
A test that spawns `dist/index.js` passes or fails on whatever `tsc` last
161-
wrote, so after changing a server's source, build before trusting the result,
162-
or run `npm run validate -w src/<server>`, which builds before it tests.
163-
164-
⚠️ **A missing `dist/` is handled two ways.** The `sequentialthinking` tests
165-
wrap themselves in `skipIf(!existsSync(distIndexPath))`, so with no build they
166-
are reported as **skipped**, and the run stays green having tested nothing over
167-
the wire. The `filesystem` stdio tests and `memory`'s dist-layout test have no
168-
such guard and **fail**. A skipped test in this repo usually means "not
169-
built".
122+
Close the client (and, for `everything`, call `cleanup()`) in `afterEach`:
123+
`everything`'s `createServer()` starts timers (the roots sync after
124+
`initialize`, the task store) that otherwise outlive the test. A tool
125+
`everything` registers in `registerConditionalTools` appears only when the
126+
`Client` declares the matching capability (`roots`, `sampling`,
127+
`elicitation`) in its constructor options.
128+
129+
Assert on what the client receives (tool, resource and prompt lists, call
130+
results, errors, notifications), never on SDK internals or a handler's context
131+
object: the SDK v2 migration (#4856) must be able to update these tests with
132+
import and type changes alone.
133+
134+
### TypeScript: the spawn smoke of the built binary
135+
136+
What only the published entry point can show (that `dist/index.js` starts,
137+
reads its arguments and serves over stdio) is checked by a thin test that
138+
spawns it with `StdioClientTransport`. Everything else is tested in-process.
139+
Two hazards apply to that kind of test, and only to it:
140+
141+
- ⚠️ **It runs the last build, not your edit.** `npm test` does not build. Build
142+
first, or run `npm run validate -w src/<server>`, which builds before it
143+
tests.
144+
- ⚠️ **A spawn test guarded with `skipIf(!existsSync(distIndexPath))` is
145+
reported as skipped with no build**, and the run stays green having tested
146+
nothing over stdio. A skipped test in this repo usually means "not built".
147+
148+
Close the client in `finally` or `afterEach`: that is what stops the child
149+
process. Pipe its stderr (`stderr: "pipe"`), and pass environment variables
150+
in the transport's `env` option, since the child inherits only the SDK's
151+
default safelist (`PATH`, `HOME`, `USER` and the like).
170152

171153
### Python: `ClientSession` over stdio
172154

@@ -214,6 +196,7 @@ than adding a dependency for one test.
214196
| One TypeScript server | `npm test -w src/<server>` |
215197
| One file, or one test | `npm test -w src/<server> -- __tests__/<file>.test.ts -t "<name>"` |
216198
| Every TypeScript server | `npm test` at the root |
199+
| One TypeScript server's coverage gate | `npm run coverage -w src/<server>` (see below) |
217200
| One TypeScript server's whole chain | `npm run validate -w src/<server>` (format check, lint, typecheck, build, test) |
218201
| One Python server | `uv run pytest` in `src/<server>` |
219202
| One Python test | `uv run pytest <path to the test file>::<name>` in `src/<server>` (for example `tests/test_server.py::test_git_checkout_existing_branch`, or `test/time_server_test.py::…` in `time`) |
@@ -226,26 +209,43 @@ files in `filesystem`, `memory` and `sequentialthinking`: their build
226209

227210
## `test` versus `coverage`
228211

229-
`test` is `vitest run`. `coverage` is `vitest run --coverage`: the same suite,
230-
plus a v8 coverage report over the workspace's `**/*.ts` (tests and `dist/`
231-
excluded), written to the terminal and to the ignored `coverage/` directory.
212+
`test` is `vitest run`, the fast loop, and part of `validate`. `coverage` is
213+
`vitest run --coverage`: the same suite, instrumented, with a v8 report over
214+
the workspace's `**/*.ts` (tests and `dist/` excluded) written to the terminal
215+
and to the ignored `coverage/` directory (`coverage/index.html` has the
216+
line-by-line view).
232217

233218
```sh
234219
npm run coverage -w src/<server> # one server
235220
npm run coverage # all four
236221
```
237222

238-
Today the report is **informational**. No vitest config sets a threshold, so
239-
`coverage` fails only when a test fails, and neither `validate` nor CI runs it.
240-
The Python servers have no coverage tooling at all: `pytest-cov` is not a dev
241-
dependency of any of them. Read the report to find what a change left
242-
untested; do not describe it as a gate.
223+
**`coverage` is a gate.** Each server's `vitest.config.ts` sets
224+
`coverage.thresholds` to 90 on lines, statements, functions and branches with
225+
`perFile: true`, so the command fails when any one file is below 90 on any
226+
one of them, even if the totals are above. CI runs it as its own job
227+
(`typescript.yml` → **Coverage \<server\>**) and `npm run local:gate` as its own
228+
stage. `test` and `validate` never measure coverage, so a change can pass
229+
them and still fail here: run `coverage` on the server you changed before you
230+
push.
231+
232+
Clearing a red file:
233+
234+
- The `Uncovered Line #s` column is the to-do list. A branch shortfall with no
235+
uncovered line is the untaken side of a condition, `??`, `?.` or default
236+
parameter: write the test that takes it.
237+
- Only code that **cannot** run gets an ignore, at the source and with its
238+
reason: `/* v8 ignore next -- <reason> */`, or `/* v8 ignore start -- <reason> */`
239+
… `/* v8 ignore stop */` around a block. "Hard to test" is not a reason.
240+
- Never lower a threshold, drop `perFile`, or add a file to the coverage
241+
`exclude` to get green.
243242

244243
⚠️ **A spawned server is invisible to the report.** v8 coverage measures the
245-
vitest process, not a child it starts, so code reached only through a stdio
246-
test reads as uncovered: `sequentialthinking`'s `index.ts` reports 0% while its
247-
stdio tests exercise it. A low number on an entry file is not, by itself,
248-
missing tests.
244+
vitest process, not a child it starts, so code reached only through the spawn
245+
smoke reads as uncovered. That is why behavior is tested in-process: a file
246+
whose only test spawns it cannot clear the gate.
247+
248+
The Python servers have no coverage tooling yet (#4855).
249249

250250
## Error output
251251

‎.github/workflows/typescript.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,39 @@ jobs:
8989
working-directory: src/${{ matrix.package }}
9090
run: npm run validate
9191

92+
# The per-file coverage gate (#4854): one leg per TypeScript server, beside
93+
# `validate` rather than inside it, as the Inspector's CI does (inspector#2159).
94+
# Each server's `coverage` (`vitest run --coverage`) fails when any file is
95+
# below 90% on lines, statements, functions or branches; the thresholds live
96+
# in the server's `vitest.config.ts`. It re-runs the suite instrumented, so it
97+
# is kept out of the fast `test`/`validate` loop and given its own runner.
98+
# `npm run local:gate` runs the same check through the root `npm run coverage`.
99+
coverage:
100+
timeout-minutes: 15
101+
needs: [detect-packages]
102+
strategy:
103+
fail-fast: false
104+
matrix:
105+
package: ${{ fromJson(needs.detect-packages.outputs.packages) }}
106+
name: Coverage ${{ matrix.package }}
107+
runs-on: ubuntu-latest
108+
steps:
109+
- uses: actions/checkout@v6
110+
111+
- uses: actions/setup-node@v7
112+
with:
113+
node-version: 22
114+
cache: npm
115+
116+
# The same install as the `validate` leg: this package plus the root. Its
117+
# `prepare` builds the package, so a test that launches `dist/` finds it.
118+
- name: Install dependencies
119+
run: npm ci --workspace src/${{ matrix.package }} --include-workspace-root
120+
121+
- name: Enforce the per-file coverage gate
122+
working-directory: src/${{ matrix.package }}
123+
run: npm run coverage
124+
92125
# The boot smoke (#4871): every server started the way a client starts it —
93126
# the built `dist/index.js` for the TypeScript servers, the console script
94127
# through `uv run` for the Python ones — over each transport it implements,

0 commit comments

Comments
 (0)