Skip to content

Commit 571e2b3

Browse files
authored
Merge pull request #4976 from modelcontextprotocol/v2/chore/4855-py-coverage-gate
chore(python): per-file 90% coverage gate in CI and local:gate (#4855)
2 parents fc478d1 + 38f9e6d commit 571e2b3

42 files changed

Lines changed: 5599 additions & 421 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/pre-push-gate/SKILL.md‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ unquoted `#` or `:` in a `description`.
7575
`verify:skills:cli` is the authoritative validator. It fetches the pinned
7676
Claude Code CLI with `npx` when the installed one is a different version, so it
7777
**fails offline**. (`validate:py`'s `sync` step is the other stage that can:
78-
`uv sync --locked` downloads when a server's environment is missing or stale.)
78+
`uv sync --locked` downloads when a server's environment is missing or stale.
79+
`coverage:py` opens with the same sync, a no-op once `validate:py` has run.)
7980

8081
### `verify:typecheck-coverage`
8182

@@ -205,6 +206,38 @@ the last `[validate:py] <server>: <step>` line above it. Re-run one server with
205206

206207
`uv: command not found` means `uv` is not installed. Never substitute `pip`.
207208

209+
### `coverage:py`
210+
211+
The Python per-file coverage gate: every file each server measures must reach
212+
**90% on lines and 90% on branches**. The summary names the server, the step,
213+
and for a shortfall each file and dimension below 90:
214+
215+
```
216+
FAIL time — per-file check (2 below 90%)
217+
src/time/src/mcp_server_time/server.py: lines 84.2% (75/89) is below 90%
218+
src/time/src/mcp_server_time/server.py: branches 78.5% (11/14) is below 90%
219+
```
220+
221+
Re-run one server with `npm run coverage:py -- <server>`. The per-file table
222+
above the summary gives both figures for every file, and coverage.py's own
223+
report above that (`term-missing`) lists the uncovered lines and partial
224+
branches (`120->124` is the jump from line 120 to 124 never taken).
225+
226+
| Step | Fix |
227+
| --- | --- |
228+
| `sync` | As for `validate:py`: run `uv lock` in that server and commit the lockfile |
229+
| `pytest --cov` | A test failed, so no verdict was reached; fix it as for `validate:py`'s `pytest`. `unrecognized arguments: --cov` means `pytest-cov` is missing from the server's dev dependencies |
230+
| `per-file check`, "below 90%" | Write tests for the lines and branches the report lists. Drive the server in-process through a `ClientSession` where you can (`/testing`) |
231+
| `per-file check`, "was not written" or "measured no files" | The run produced no usable `coverage.json`: check the server's `[tool.coverage.run]` (`branch = true`, `source` naming its package) |
232+
| `per-file check`, "not measured with branch coverage" or "no covered_branches/num_branches" | `branch = true` is missing from `[tool.coverage.run]`. The check refuses such a report rather than counting every file's zero branches as 100% |
233+
234+
**Never lower the gate**, and never drop a file from `source` to clear it.
235+
Code that genuinely cannot run (a guard the types already rule out, a branch
236+
the SDK's input validation makes unreachable) is marked at the source with
237+
`# pragma: no cover # <reason>`; every pragma carries its reason
238+
(`AGENTS.md` **Always test new or modified code**). A pragma on a line that is
239+
merely awkward to test is not justified; write the test.
240+
208241
### `smoke`
209242

210243
`smoke — FAIL <server> over <transport>` followed by the reason and the tail

‎.claude/skills/project-structure/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,13 +64,13 @@ in-process harness.
6464

6565
```
6666
src/<server>/
67-
├── pyproject.toml dependencies, the console script; pytest config in `fetch` and `git`
67+
├── pyproject.toml dependencies, the console script, pytest and coverage config
6868
├── uv.lock, .python-version
6969
├── src/mcp_server_<name>/
7070
│ ├── __init__.py main(): the command-line flags, then asyncio.run(serve(...))
7171
│ ├── __main__.py python -m mcp_server_<name>
7272
│ └── server.py everything else
73-
└── tests/ test/ in `time`
73+
└── tests/ pytest suite, driving serve() in-process
7474
```
7575

7676
`server.py` holds the Pydantic input models, the functions that do the work,

‎.claude/skills/testing/SKILL.md‎

Lines changed: 88 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -16,18 +16,18 @@ clear the per-file coverage gate.
1616
Driving a running server by hand, with the Inspector or an LLM client, is
1717
`/client-smoke`. It is evidence for a PR, not a substitute for a test.
1818

19-
⚠️ **The TypeScript and Python harnesses differ.** All four TypeScript servers
20-
are tested in-process, a real `Client` linked to the server over an in-memory
21-
transport (#4854). The Python servers' tests call their functions directly,
22-
and those servers have no coverage gate yet (#4855).
19+
**Every server is tested in-process, and every file is gated.** The four
20+
TypeScript servers link a real `Client` to the server over an in-memory
21+
transport (#4854); the three Python servers link a `ClientSession` to the real
22+
`serve()` over in-memory streams (#4855). Both languages hold each file to a
23+
per-file coverage floor, in CI and in `npm run local:gate`.
2324

2425
## Where a test goes
2526

2627
| Server | Directory | File name | Runner |
2728
| --- | --- | --- | --- |
2829
| `everything`, `filesystem`, `memory`, `sequentialthinking` | `src/<server>/__tests__/` | `<subject>.test.ts` | vitest, `globals: true` |
29-
| `fetch`, `git` | `src/<server>/tests/` | `test_<subject>.py` | pytest |
30-
| `time` | `src/time/test/` | `<subject>_test.py` | pytest |
30+
| `fetch`, `git`, `time` | `src/<server>/tests/` | `test_<subject>.py` | pytest, with `pytest-asyncio` (`asyncio_mode = "auto"`) |
3131
| Root tooling | `scripts/`, beside the script | `<name>.test.mjs` | `node --test` |
3232

3333
Each vitest config includes only `**/__tests__/**/*.test.ts`, so a test file
@@ -41,9 +41,9 @@ connection harness does (`__tests__/harness.ts` in `everything`,
4141
| Server | How its tests reach the code |
4242
| --- | --- |
4343
| `everything`, `filesystem`, `memory`, `sequentialthinking` | An SDK `Client` connected to the server's `createServer()` over `InMemoryTransport`, in the vitest process, through the server's own harness, plus unit tests of helper modules such as `lib.ts`. `everything` drives its stdio, SSE and Streamable HTTP transports in-process too; the other three have one thin spawn smoke of the built `dist/index.js` |
44-
| `fetch`, `git`, `time` | Direct calls on the functions in `server.py`, with `unittest.mock` |
44+
| `fetch`, `git`, `time` | A `ClientSession` linked in-process to the real `serve()` over in-memory streams (below), plus direct calls on the pure functions in `server.py` |
4545

46-
No Python test opens a `ClientSession` yet.
46+
No TypeScript test uses an in-memory transport yet.
4747

4848
## Choosing a harness for a new test
4949

@@ -162,44 +162,71 @@ in the transport's `env` option (`MEMORY_FILE_PATH`, say), since the child
162162
inherits only the SDK's default safelist (`PATH`, `HOME`, `USER` and the
163163
like).
164164

165-
### Python: `ClientSession` over stdio
165+
### Python: `ClientSession` over in-memory streams, in-process
166166

167-
Each Python server constructs its `Server` inside `serve()`, so it cannot be
168-
handed to an in-memory session. A protocol-level test spawns the module and
169-
talks to it through the SDK's stdio client:
167+
Each Python server builds its `Server` inside `serve()` and serves it over the
168+
`stdio_server()` it imports into `server.py`. A test replaces that one name
169+
with a context manager that yields the server side of a pair of in-memory
170+
streams, runs the real `serve()` in a task group, and opens a `ClientSession`
171+
on the client side. No process, no build, and the SDK's input validation and
172+
error mapping run exactly as they do for a real client:
170173

171174
```python
172-
import sys
175+
from contextlib import asynccontextmanager
176+
from typing import Any, AsyncIterator
173177

174178
import anyio
175-
from mcp import ClientSession, StdioServerParameters
176-
from mcp.client.stdio import stdio_client
177-
178-
179-
def test_get_current_time_over_stdio() -> None:
180-
async def _run() -> None:
181-
params = StdioServerParameters(
182-
command=sys.executable, args=["-m", "mcp_server_time"]
183-
)
184-
async with stdio_client(params) as (read, write):
185-
async with ClientSession(read, write) as session:
186-
await session.initialize()
187-
result = await session.call_tool(
188-
"get_current_time", {"timezone": "UTC"}
189-
)
190-
assert result.isError is False
191-
192-
anyio.run(_run)
193-
```
179+
import pytest
180+
from mcp import ClientSession
181+
from mcp.shared.memory import create_client_server_memory_streams
182+
183+
import mcp_server_time.server as server_module
184+
185+
186+
@asynccontextmanager
187+
async def connect(**serve_kwargs: Any) -> AsyncIterator[ClientSession]:
188+
async with create_client_server_memory_streams() as (client, server):
194189

195-
`sys.executable` is the server's own virtual environment under `uv run pytest`,
196-
and the install there is editable, so this runs your edit with no build step.
190+
@asynccontextmanager
191+
async def fake_stdio_server() -> AsyncIterator[Any]:
192+
yield server
193+
194+
with pytest.MonkeyPatch.context() as mp:
195+
mp.setattr(server_module, "stdio_server", fake_stdio_server)
196+
async with anyio.create_task_group() as tg:
197+
tg.start_soon(lambda: server_module.serve(**serve_kwargs))
198+
async with ClientSession(*client) as session:
199+
await session.initialize()
200+
yield session
201+
tg.cancel_scope.cancel()
202+
203+
204+
async def test_get_current_time() -> None:
205+
async with connect() as session:
206+
result = await session.call_tool("get_current_time", {"timezone": "UTC"})
207+
assert result.isError is False
208+
```
197209

198-
How an async test is written differs by server, because the dev dependencies
199-
do: `fetch` has `pytest-asyncio` with `asyncio_mode = "auto"`, so an
200-
`async def test_…` just works. `git` and `time` do not have it; wrap the
201-
coroutine in `anyio.run(...)` as above (`git`'s `serve()` test does), rather
202-
than adding a dependency for one test.
210+
- **Patch the name `server.py` looks up** (`mcp_server_<name>.server.stdio_server`),
211+
not `mcp.server.stdio.stdio_server`: `server.py` imported the function, so
212+
the module attribute is what `serve()` calls.
213+
- **Cancel the task group after the session closes.** `serve()` runs until its
214+
streams end, so without the cancel the test never returns.
215+
- **Assert on the wire shape where you can**, for example
216+
`result.model_dump(by_alias=True, mode="json")`, rather than on Python
217+
attribute names: the SDK v2 port (#4851) renames them, and a wire-level
218+
assertion survives it.
219+
- **No network in a test.** `fetch` replaces `httpx.AsyncClient` with one on an
220+
`httpx.MockTransport`; `git` works against a temporary repository its
221+
fixture creates and closes before removing.
222+
223+
`pytest-asyncio` with `asyncio_mode = "auto"` is a dev dependency of all three,
224+
so an `async def test_…` runs as it stands.
225+
226+
A subprocess test over stdio (`StdioServerParameters(command=sys.executable,
227+
args=["-m", "mcp_server_<name>"])` with `mcp.client.stdio.stdio_client`) is
228+
kept to a thin smoke of the entry point. coverage.py does not measure a child
229+
process, so code reached only that way reads as uncovered.
203230

204231
## Tests that pin a known bug
205232

@@ -236,8 +263,10 @@ In Python the same text follows `#`. A bug with no issue yet reads
236263
| One TypeScript server's coverage gate | `npm run coverage -w src/<server>` (see below) |
237264
| One TypeScript server's whole chain | `npm run validate -w src/<server>` (format check, lint, typecheck, build, test) |
238265
| One Python server | `uv run pytest` in `src/<server>` |
239-
| One Python test | `uv run pytest <path to the test file>::<name>` in `src/<server>` (for example `tests/test_server.py::test_git_checkout_existing_branch`, or `test/time_server_test.py::…` in `time`) |
266+
| One Python test | `uv run pytest <path to the test file>::<name>` in `src/<server>` (for example `tests/test_server.py::test_git_checkout_existing_branch`) |
240267
| One Python server's whole chain | `npm run validate:py -- <server>` |
268+
| One Python server's coverage gate | `npm run coverage:py -- <server>` at the root |
269+
| Every Python server's coverage gate | `npm run coverage:py` at the root |
241270
| Root tooling | `npm run test:scripts` |
242271

243272
`npm run typecheck -w src/<server>` is the only step that typechecks the test
@@ -277,13 +306,30 @@ Clearing a red file:
277306
- Never lower a threshold, drop `perFile`, or add a file to the coverage
278307
`exclude` to get green.
279308

309+
**The Python servers' coverage is a gate.** `npm run coverage:py -- <server>`
310+
runs, in `src/<server>`:
311+
312+
```sh
313+
uv run --frozen pytest --cov --cov-report=term-missing --cov-report=json
314+
```
315+
316+
which prints coverage.py's report (the missing lines and partial branches)
317+
and writes `coverage.json` (ignored by git). `scripts/lib/py-coverage.mjs` then
318+
checks **every file** in it: at least 90% of its lines and 90% of its branches,
319+
each file on its own. What is measured is the server's `[tool.coverage.run]`
320+
(`branch = true`, `source` set to its package). `local:gate` and CI's
321+
**Coverage \<server\>** job both run it. It is kept out of the plain
322+
`uv run pytest` loop, which stays fast and uninstrumented.
323+
324+
A line or branch that genuinely cannot run is marked
325+
`# pragma: no cover # <reason>`, with the reason on the same line; never
326+
lower the gate. Diagnosing a red run is `/pre-push-gate`.
327+
280328
⚠️ **A spawned server is invisible to the report.** v8 coverage measures the
281329
vitest process, not a child it starts, so code reached only through a spawn
282330
smoke reads as uncovered. That is why behavior is tested in-process: a file
283331
whose only test spawns it cannot clear the gate.
284332

285-
The Python servers have no coverage tooling yet (#4855).
286-
287333
## Error output
288334

289335
A test that expects an error keeps it off the console. Spy on the logger for

‎.github/workflows/python.yml‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
node-version: 22
3939

4040
- name: Run node --test
41-
run: node --test scripts/validate-py.test.mjs
41+
run: node --test scripts/validate-py.test.mjs scripts/coverage-py.test.mjs scripts/lib/py-coverage.test.mjs
4242

4343
test:
4444
timeout-minutes: 15
@@ -68,6 +68,37 @@ jobs:
6868
- name: Validate
6969
run: node scripts/validate-py.mjs ${{ matrix.package }}
7070

71+
# The per-file coverage gate (#4855), in parallel with `test`: every file of
72+
# each server at >= 90% on lines and on branches. The same script
73+
# `npm run coverage:py` runs locally, one server per leg. fail-fast is off so
74+
# one server's shortfall does not cancel the other legs' verdicts.
75+
coverage:
76+
timeout-minutes: 15
77+
needs: [detect-packages]
78+
strategy:
79+
fail-fast: false
80+
matrix:
81+
package: ${{ fromJson(needs.detect-packages.outputs.packages) }}
82+
name: Coverage ${{ matrix.package }}
83+
runs-on: ubuntu-latest
84+
steps:
85+
- uses: actions/checkout@v6
86+
87+
- name: Install uv
88+
uses: astral-sh/setup-uv@v7
89+
90+
- name: Set up Python
91+
uses: actions/setup-python@v7
92+
with:
93+
python-version-file: "src/${{ matrix.package }}/.python-version"
94+
95+
- uses: actions/setup-node@v7
96+
with:
97+
node-version: 22
98+
99+
- name: Coverage
100+
run: npm run coverage:py -- ${{ matrix.package }}
101+
71102
build:
72103
timeout-minutes: 10
73104
needs: [detect-packages, test]

‎AGENTS.md‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ servers/
5252
│ └── time/ Py mcp-server-time PyPI Time and timezone conversion
5353
├── .claude/skills/ On-demand procedures (see the Skills index above)
5454
├── .changeset/ Pending changesets for the TypeScript servers, and the changesets config
55-
├── scripts/ The pre-push gate (gate-lease, smoke-servers, validate-py), its guards (verify-*),
55+
├── scripts/ The pre-push gate (gate-lease, smoke-servers, validate-py, coverage-py), its guards (verify-*),
5656
│ the skills tooling, and release tooling (npm-publish-guard, prepare-python-release,
5757
│ pack-and-verify, release-manifest)
5858
├── docs/ Design documents; quality-gate.md is the gate's reference (stages, CI vs local, the lease);
@@ -110,6 +110,7 @@ uv run pytest
110110
uv run --frozen pyright
111111
uv run ruff check .
112112
uv build
113+
npm run coverage:py -- <server> # at the root: the per-file coverage gate
113114

114115
# Both languages: the pre-push gate (see Before pushing)
115116
npm run local:gate
@@ -123,7 +124,8 @@ It runs every check CI runs, for both languages, in one command:
123124
workspace's format check, lint, typecheck, build and tests), `coverage` (each
124125
TypeScript workspace's per-file coverage gate), `validate:py`
125126
(each Python server's locked sync, `ruff check`, `ruff format --check`,
126-
pyright, pytest and build), `verify:skills:cli`, and `smoke` (every server
127+
pyright, pytest and build), `coverage:py` (each Python server's per-file
128+
coverage gate), `verify:skills:cli`, and `smoke` (every server
127129
booted over each transport it implements). The stage-by-stage reference, and
128130
what CI runs where, is [`docs/quality-gate.md`](./docs/quality-gate.md);
129131
diagnosing a red stage is the `pre-push-gate` skill.
@@ -452,25 +454,30 @@ here.
452454

453455
- **New or changed behavior comes with tests.** TypeScript servers use
454456
**vitest** (with `@vitest/coverage-v8`), with tests in the server's
455-
`__tests__/` directory. Python servers use **pytest** (`pytest-asyncio` where
456-
the server is async), with tests in the server's `tests/` directory (`test/` in
457-
`time`).
457+
`__tests__/` directory. Python servers use **pytest** (with `pytest-asyncio`),
458+
with tests in the server's `tests/` directory.
458459
- **Coverage is gated per file.** The rule for each language:
459460
- **TypeScript**: every file in a server clears **≥ 90 on all four
460461
dimensions** (lines, statements, functions and branches). The thresholds
461462
are set in each server's `vitest.config.ts` (`perFile: true`). A PR that
462463
drops any file below 90 on any dimension fails CI (`typescript.yml` →
463464
**Coverage \<server\>**) and `npm run local:gate`.
465+
- **Python**: every file clears **≥ 90 on lines and on branches**, the two
466+
dimensions coverage.py measures (it has no native function dimension). A
467+
PR that drops any Python file below 90 on either fails CI (`python.yml` →
468+
**Coverage \<server\>**) and `npm run local:gate`.
464469
- **A genuinely unreachable branch is annotated at the source, never waved
465470
through by lowering the gate.** Every ignore carries its reason:
466-
`/* v8 ignore next -- <reason> */` in TypeScript. Reach for one only when the
471+
`/* v8 ignore next -- <reason> */` in TypeScript,
472+
`# pragma: no cover # <reason>` in Python. Reach for one only when the
467473
code cannot be exercised (a defensive guard for a value the types already
468474
rule out, say), never because a test is hard to write.
469-
- **`coverage` is its own command, separate from `test` and `validate`.**
475+
- **Coverage is its own command, separate from the fast test loop.**
470476
`npm run coverage -w src/<server>` (or the root `npm run coverage` for every
471-
TypeScript server) runs the suite instrumented and enforces the gate;
472-
`test` and `validate` stay fast and do not. CI and `npm run local:gate` run
473-
it as a stage of its own.
477+
TypeScript server) and `npm run coverage:py` (`-- <server>` for one) run the
478+
suites instrumented and enforce the gate; `test`, `validate` and
479+
`uv run pytest` stay fast and do not. CI and `npm run local:gate` run each
480+
as a stage of its own.
474481
- **Test at the protocol level where you can**: drive the server through an MCP
475482
client over an in-memory transport and assert on what comes back, rather than
476483
only calling internal functions.

‎CONTRIBUTING.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,9 +151,10 @@ The rules for maintainers and the agents working for them are in
151151
pushing, run `npm run format` at the repository root and then
152152
**`npm run local:gate`**, which runs every check CI runs for both languages:
153153
the repo-wide guards, each TypeScript server's `validate` (format check, lint
154-
where a warning fails like an error, typecheck, build, tests), each Python
155-
server's `validate:py` chain, the skills validator, and a boot smoke of every
156-
server. [`docs/quality-gate.md`](./docs/quality-gate.md) describes each stage.
154+
where a warning fails like an error, typecheck, build, tests) and per-file
155+
coverage gate (`coverage`), each Python
156+
server's `validate:py` chain and per-file coverage gate (`coverage:py`), the
157+
skills validator, and a boot smoke of every server. [`docs/quality-gate.md`](./docs/quality-gate.md) describes each stage.
157158
While iterating, `npm run validate -w src/<server>` checks a single TypeScript
158159
server and `npm run validate:py -- <server>` a single Python one; neither
159160
replaces the gate.

0 commit comments

Comments
 (0)