You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f149a2a
Browse filesBrowse the repository at this point in the historyBrowse files
ci(coverage): enforce the TypeScript per-file coverage gate in CI and local:gate
Adds a parallel `coverage` job to typescript.yml (one leg per TypeScript
server, beside `validate`, as the Inspector's CI does) and a `coverage` stage
to `local:gate:stages` right after `validate`. The workflow guard derives the
stage from the new CI job, and its stage list now pins `coverage` by name.
Documents the stage in docs/quality-gate.md, the TypeScript half of the
coverage rule in AGENTS.md, how to diagnose a red coverage stage in the
pre-push-gate skill, and the in-process harness and coverage gate in the
testing skill. The docs describe the state after the four per-server
coverage PRs for #4854 land, which add the 90/90/90/90 per-file thresholds.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: cliffhall <[email protected]>
Copy file name to clipboardExpand all lines: .claude/skills/testing/SKILL.md
+90-90Lines changed: 90 additions & 90 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,19 +7,19 @@ disable-model-invocation: false
7
7
# Testing
8
8
9
9
The **rules** are in [`AGENTS.md`](../../../AGENTS.md) under **Always test new
10
-
or modified code**: new behavior comes with tests, test at the protocol level
11
-
where you can, and keep expected error output off the console. This skill
12
-
explains which harness each server has **today**, how to write a test with it,
13
-
and which command runs what.
10
+
or modified code**: new behavior comes with tests, every TypeScript file clears
11
+
the per-file coverage gate, test at the protocol level where you can, and keep
12
+
expected error output off the console. This skill explains the harness each
13
+
server has, how to write a test with it, which command runs what, and how to
14
+
clear the per-file coverage gate.
14
15
15
16
Driving a running server by hand, with the Inspector or an LLM client, is
16
17
`/client-smoke`. It is evidence for a PR, not a substitute for a test.
17
18
18
-
⚠️ **The harnesses are uneven, and this page describes them as they are.** Only
19
-
`everything` can be driven in-process today. A shared in-process harness for
20
-
every server, and a per-file coverage threshold, are the work of #4854
21
-
(TypeScript) and #4855 (Python); neither exists yet, and this skill changes
22
-
when they land. Do not write a test that assumes them.
19
+
⚠️ **The TypeScript and Python harnesses differ.** All four TypeScript servers
20
+
are tested in-process, a real `Client` linked to the server over an in-memory
21
+
transport (#4854). The Python servers' tests call their functions directly,
22
+
and those servers have no coverage gate yet (#4855).
23
23
24
24
## Where a test goes
25
25
@@ -35,18 +35,14 @@ anywhere else in a workspace is never run and reports nothing. Shared test
35
35
helpers go in `__tests__/` without the `.test` suffix, as
36
36
`src/everything/__tests__/helpers.ts` does.
37
37
38
-
## What each server's tests do today
38
+
## What each server's tests do
39
39
40
40
| Server | How its tests reach the code |
41
41
| --- | --- |
42
-
|`everything`| Call a `register…` function with a mocked `McpServer`, capture the handler it registered, and invoke the handler directly. `__tests__/helpers.ts` types the captured handlers |
43
-
|`filesystem`| Unit tests of `lib.ts` and the path helpers, plus an SDK `Client` over `StdioClientTransport` that spawns the built `dist/index.js`|
44
-
|`sequentialthinking`| Unit tests of `lib.ts`, plus an SDK `Client` over stdio against `dist/index.js`|
45
-
|`memory`| Direct calls on the exported `KnowledgeGraphManager` and `register…` functions, with a mocked server |
42
+
|`everything`, `filesystem`, `memory`, `sequentialthinking`| An SDK `Client` connected to the server's `createServer()` over `InMemoryTransport`, in the vitest process; unit tests of the helper modules beside them; at most a thin spawn-based smoke of the built `dist/index.js`|
46
43
|`fetch`, `git`, `time`| Direct calls on the functions in `server.py`, with `unittest.mock`|
47
44
48
-
No test in the repo uses an in-memory transport yet, and no Python test opens
49
-
a `ClientSession`.
45
+
No Python test opens a `ClientSession` yet.
50
46
51
47
## Choosing a harness for a new test
52
48
@@ -59,16 +55,27 @@ Ask what the test must prove.
59
55
error result, a capability, a notification): drive the server through a
60
56
client. A handler called directly skips the SDK's input validation, output
61
57
schema validation and error mapping, which is where the wire-level bugs in
62
-
this repo have been. The regression tests in `filesystem` and
63
-
`sequentialthinking` that go through a `Client` exist for exactly that
64
-
reason.
58
+
this repo have been.
65
59
66
-
Which client harness is available depends on the server.
60
+
Which client harness is available depends on the language.
67
61
68
-
### `everything`: in-process, over an in-memory transport
62
+
### TypeScript: in-process, over an in-memory transport
69
63
70
-
`src/everything/server/index.ts` exports `createServer()`, so a test can link a
71
-
real `Client` to a real server with no process and no build:
64
+
Each TypeScript server exports a `createServer(…)` factory and starts its
65
+
transport only from a guarded `main()`, so importing it starts nothing and a
66
+
test can link a real `Client` to a real server with no process and no build:
awaitclient.close(); // this is what stops the child process
141
-
}
142
-
```
143
-
144
-
Closing in `finally` (or in `afterEach`, as the `filesystem` tests do) matters:
145
-
a failed assertion that skips `close()` leaves the spawned server running.
146
-
147
-
`src/sequentialthinking/__tests__/input-schema.test.ts` is the reference.
148
-
`stderr: "pipe"` keeps the server's startup banner off the console.
149
-
150
-
Give the server what it needs to start. `filesystem` takes its allowed
151
-
directories as arguments (`args: [distIndexPath, testDir]`, with `testDir` a
152
-
`realpath`-resolved temporary directory, as
153
-
`src/filesystem/__tests__/structured-content.test.ts` does); with none, and a
154
-
client that offers no Roots, every file operation is refused. Environment
155
-
variables such as `MEMORY_FILE_PATH` go in the transport's `env` option: the
156
-
child inherits only the SDK's default safelist (`PATH`, `HOME`, `USER` and the
157
-
like), not the other variables of the test process.
158
-
159
-
⚠️ **These tests run the last build, not your edit.**`npm test` does not build.
160
-
A test that spawns `dist/index.js` passes or fails on whatever `tsc` last
161
-
wrote, so after changing a server's source, build before trusting the result,
162
-
or run `npm run validate -w src/<server>`, which builds before it tests.
163
-
164
-
⚠️ **A missing `dist/` is handled two ways.** The `sequentialthinking` tests
165
-
wrap themselves in `skipIf(!existsSync(distIndexPath))`, so with no build they
166
-
are reported as **skipped**, and the run stays green having tested nothing over
167
-
the wire. The `filesystem` stdio tests and `memory`'s dist-layout test have no
168
-
such guard and **fail**. A skipped test in this repo usually means "not
169
-
built".
122
+
Close the client (and, for `everything`, call `cleanup()`) in `afterEach`:
123
+
`everything`'s `createServer()` starts timers (the roots sync after
124
+
`initialize`, the task store) that otherwise outlive the test. A tool
125
+
`everything` registers in `registerConditionalTools` appears only when the
126
+
`Client` declares the matching capability (`roots`, `sampling`,
127
+
`elicitation`) in its constructor options.
128
+
129
+
Assert on what the client receives (tool, resource and prompt lists, call
130
+
results, errors, notifications), never on SDK internals or a handler's context
131
+
object: the SDK v2 migration (#4856) must be able to update these tests with
132
+
import and type changes alone.
133
+
134
+
### TypeScript: the spawn smoke of the built binary
135
+
136
+
What only the published entry point can show (that `dist/index.js` starts,
137
+
reads its arguments and serves over stdio) is checked by a thin test that
138
+
spawns it with `StdioClientTransport`. Everything else is tested in-process.
139
+
Two hazards apply to that kind of test, and only to it:
140
+
141
+
- ⚠️ **It runs the last build, not your edit.**`npm test` does not build. Build
142
+
first, or run `npm run validate -w src/<server>`, which builds before it
143
+
tests.
144
+
- ⚠️ **A spawn test guarded with `skipIf(!existsSync(distIndexPath))` is
145
+
reported as skipped with no build**, and the run stays green having tested
146
+
nothing over stdio. A skipped test in this repo usually means "not built".
147
+
148
+
Close the client in `finally` or `afterEach`: that is what stops the child
149
+
process. Pipe its stderr (`stderr: "pipe"`), and pass environment variables
150
+
in the transport's `env` option, since the child inherits only the SDK's
151
+
default safelist (`PATH`, `HOME`, `USER` and the like).
170
152
171
153
### Python: `ClientSession` over stdio
172
154
@@ -214,6 +196,7 @@ than adding a dependency for one test.
214
196
| One TypeScript server |`npm test -w src/<server>`|
215
197
| One file, or one test |`npm test -w src/<server> -- __tests__/<file>.test.ts -t "<name>"`|
216
198
| Every TypeScript server |`npm test` at the root |
199
+
| One TypeScript server's coverage gate |`npm run coverage -w src/<server>` (see below) |
217
200
| One TypeScript server's whole chain |`npm run validate -w src/<server>` (format check, lint, typecheck, build, test) |
218
201
| One Python server |`uv run pytest` in `src/<server>`|
219
202
| One Python test |`uv run pytest <path to the test file>::<name>` in `src/<server>` (for example `tests/test_server.py::test_git_checkout_existing_branch`, or `test/time_server_test.py::…` in `time`) |
@@ -226,26 +209,43 @@ files in `filesystem`, `memory` and `sequentialthinking`: their build
226
209
227
210
## `test` versus `coverage`
228
211
229
-
`test` is `vitest run`. `coverage` is `vitest run --coverage`: the same suite,
230
-
plus a v8 coverage report over the workspace's `**/*.ts` (tests and `dist/`
231
-
excluded), written to the terminal and to the ignored `coverage/` directory.
212
+
`test` is `vitest run`, the fast loop, and part of `validate`. `coverage` is
213
+
`vitest run --coverage`: the same suite, instrumented, with a v8 report over
214
+
the workspace's `**/*.ts` (tests and `dist/` excluded) written to the terminal
215
+
and to the ignored `coverage/` directory (`coverage/index.html` has the
216
+
line-by-line view).
232
217
233
218
```sh
234
219
npm run coverage -w src/<server># one server
235
220
npm run coverage # all four
236
221
```
237
222
238
-
Today the report is **informational**. No vitest config sets a threshold, so
239
-
`coverage` fails only when a test fails, and neither `validate` nor CI runs it.
240
-
The Python servers have no coverage tooling at all: `pytest-cov` is not a dev
241
-
dependency of any of them. Read the report to find what a change left
242
-
untested; do not describe it as a gate.
223
+
**`coverage` is a gate.** Each server's `vitest.config.ts` sets
224
+
`coverage.thresholds` to 90 on lines, statements, functions and branches with
225
+
`perFile: true`, so the command fails when any one file is below 90 on any
226
+
one of them, even if the totals are above. CI runs it as its own job
227
+
(`typescript.yml` → **Coverage \<server\>**) and `npm run local:gate` as its own
228
+
stage. `test` and `validate` never measure coverage, so a change can pass
229
+
them and still fail here: run `coverage` on the server you changed before you
230
+
push.
231
+
232
+
Clearing a red file:
233
+
234
+
- The `Uncovered Line #s` column is the to-do list. A branch shortfall with no
235
+
uncovered line is the untaken side of a condition, `??`, `?.` or default
236
+
parameter: write the test that takes it.
237
+
- Only code that **cannot** run gets an ignore, at the source and with its
238
+
reason: `/* v8 ignore next -- <reason> */`, or `/* v8 ignore start -- <reason> */`
239
+
… `/* v8 ignore stop */` around a block. "Hard to test" is not a reason.
240
+
- Never lower a threshold, drop `perFile`, or add a file to the coverage
241
+
`exclude` to get green.
243
242
244
243
⚠️ **A spawned server is invisible to the report.** v8 coverage measures the
245
-
vitest process, not a child it starts, so code reached only through a stdio
246
-
test reads as uncovered: `sequentialthinking`'s `index.ts` reports 0% while its
247
-
stdio tests exercise it. A low number on an entry file is not, by itself,
248
-
missing tests.
244
+
vitest process, not a child it starts, so code reached only through the spawn
245
+
smoke reads as uncovered. That is why behavior is tested in-process: a file
246
+
whose only test spawns it cannot clear the gate.
247
+
248
+
The Python servers have no coverage tooling yet (#4855).
0 commit comments