[compliance tests] initial server scenarios#4
Conversation
pcarleton
left a comment
There was a problem hiding this comment.
Left a bunch of comments, but they're non-blocking / thinking through where we go with this. I'd say let's merge this all in and iterate forward.
A couple things this prompted me to that I want to explore are:
- how to test a scenario, specifically making sure it fails when it should. I think we'll want to make this super easy to do
- what negative tests we might want for a server (i.e. server shouldn't do this)
- starting to compile the big "MUST" / "SHOULD" list and start burning down our coverage
|
|
||
| return { | ||
| client, | ||
| close: async () => { |
| try { | ||
| const connection = await connectToServer(serverUrl); | ||
|
|
||
| // The connection process already does initialization |
There was a problem hiding this comment.
this is an example of where if we use the SDK, we lose some visibility. this one seems fine as long as the error the SDK throws is easy to interpret in the check output, but just flagging the pattern where we may want to break out of the sdk
| } | ||
| } | ||
|
|
||
| export class PromptsGetSimpleScenario implements ClientScenario { |
There was a problem hiding this comment.
I'm still getting a feel for the cost of an additional scenario, but I'd be inclined to do all these in 1 vs. separate scenarios for each.
| @@ -0,0 +1,874 @@ | |||
| # MCP Server Conformance Requirements | |||
There was a problem hiding this comment.
I was imagining this being a "suite" that is called something like "Basic", and it's a list of the tests to run, where each test describes the functionality it needs.
I like that this file describes the current SDK goal in a "1-stop" way though, so let's keep this, and we can explore "suites" or profiles later. Like for SDK's, we want this full set of features, but for servers that don't want some, we can have suites that don't fail on missing optional features.
| import { ClientScenario, ConformanceCheck } from '../../types.js'; | ||
| import { connectToServer } from './client-helper.js'; | ||
|
|
||
| export class ResourcesListScenario implements ClientScenario { |
There was a problem hiding this comment.
similar here, my current take is these would be better as 1 scenario w/ multiple checks
| Progress | ||
| } from '@modelcontextprotocol/sdk/types.js'; | ||
|
|
||
| export class ToolsListScenario implements ClientScenario { |
There was a problem hiding this comment.
these ones seem good as separate, but I might group it like:
- list + call simple
- call content types (image, audio, mixed, resource)
- (rest as individual)
the rough heuristic I'm building up is "would it make sense for someone to implement a server that just does this scenario?" (open to iterating on that heuristic, I think we'll want it to be easy for folks to reason through when writing scenarios and using the scenarios). Right now, we're proposing folks have an everything-server that does all the things, but I don't want it to be a requirement that it's all 1 server since some scenarios will be mutually exclusive (i.e. an SDK should be able to support writing a server that doesn't support certain stateful features if that unlocks other transport things).
Some of this might also be fine to "start out everything super small, and then group things up later based on what's annoying or not as we use it more".
| console.log(`\n=== Running scenario: ${scenarioName} ===`); | ||
| try { | ||
| const result = await runServerConformanceTest(serverUrl, scenarioName); | ||
| allResults.push({ scenario: scenarioName, checks: result.checks }); |
There was a problem hiding this comment.
hypothesis here is we'll want results logged as we go, and summary at the end for faster feedback loops
7635a76 to
7b3666a
Compare
Address review findings on the auth/dpop-nonce posture: - Record the token-request proof observation BEFORE the §8 nonce challenge, so a client that is challenged but does not retry is no longer mis-reported by token-request-proof as "never completed a token request" (its proof was just validated). (#2) - Record the resource-request jti/replay on ANY valid proof (before the §9 nonce gate), so fresh-proof is never asserted on zero evidence and a client that never honours the RS nonce fails ONLY rs-nonce. (modelcontextprotocol#5) - Require challengeIssued && honored for as-nonce/rs-nonce SUCCESS, closing a vacuous-pass path (a client that pre-sends the nonce is never challenged). (#3a) - Gate the AS nonce observation on the authorization_code exchange, matching recordTokenRequestProof (a refresh exchange must not satisfy §8). (#3c) - Collapse duplicate shared token-flow check IDs (token-request, pkce-*) that the §8 round-trip re-POST would otherwise emit twice. (modelcontextprotocol#4) - Example client: retry the token request only on a use_dpop_nonce error code, not any 400 carrying a DPoP-Nonce header (consistent with the RS path). (modelcontextprotocol#6) - Fix the DpopTokenRequestObservation docstring ("last write wins" → sticky-failure). (modelcontextprotocol#7) Adds an expectedSuccessSlugs option to the client test helper and tightens the two nonce negative tests to pin the now-clean behavior (token-request-proof SUCCESS for no-as-nonce; only rs-nonce fails for no-rs-nonce). Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Refresh the held DPoP nonce from every response's DPoP-Nonce header (RFC 9449 §8.2 newest-wins) instead of capturing it once, so a server that rotates or single-uses its nonce no longer turns the negative probes into false "not testable" failures (or vacuous passes). (#1) - Widen the stale iat probe to -303 s (matching the future side) so the ±1 s quantization of the whole-second Date header can't pull it onto the ±300 s boundary and be false-accepted. (#2) - Update the isNonceChallenge comment and the untestable message: probes now carry the held nonce, so a use_dpop_nonce challenge is about the nonce lifetime (rotated/stale/single-use), not a missing nonce. (#3) - Fixture: parse integer env vars NaN-safely (intEnv helper) so a malformed DPOP_CLOCK_OFFSET_SECONDS / DPOP_IAT_SKEW_SECONDS falls back to its default instead of silently disabling the iat window. (modelcontextprotocol#4) Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Refresh the held DPoP nonce from every response's DPoP-Nonce header (RFC 9449 §8.2 newest-wins) instead of capturing it once, so a server that rotates or single-uses its nonce no longer turns the negative probes into false "not testable" failures (or vacuous passes). (#1) - Widen the stale iat probe to -303 s (matching the future side) so the ±1 s quantization of the whole-second Date header can't pull it onto the ±300 s boundary and be false-accepted. (#2) - Update the isNonceChallenge comment and the untestable message: probes now carry the held nonce, so a use_dpop_nonce challenge is about the nonce lifetime (rotated/stale/single-use), not a missing nonce. (#3) - Fixture: parse integer env vars NaN-safely (intEnv helper) so a malformed DPOP_CLOCK_OFFSET_SECONDS / DPOP_IAT_SKEW_SECONDS falls back to its default instead of silently disabling the iat window. (modelcontextprotocol#4) Co-Authored-By: Claude Opus 4.8 <[email protected]>
- Refresh the held DPoP nonce from every response's DPoP-Nonce header (RFC 9449 §8.2 newest-wins) instead of capturing it once, so a server that rotates or single-uses its nonce no longer turns the negative probes into false "not testable" failures (or vacuous passes). (#1) - Widen the stale iat probe to -303 s (matching the future side) so the ±1 s quantization of the whole-second Date header can't pull it onto the ±300 s boundary and be false-accepted. (#2) - Update the isNonceChallenge comment and the untestable message: probes now carry the held nonce, so a use_dpop_nonce challenge is about the nonce lifetime (rotated/stale/single-use), not a missing nonce. (#3) - Fixture: parse integer env vars NaN-safely (intEnv helper) so a malformed DPOP_CLOCK_OFFSET_SECONDS / DPOP_IAT_SKEW_SECONDS falls back to its default instead of silently disabling the iat window. (modelcontextprotocol#4) Co-Authored-By: Claude Opus 4.8 <[email protected]>
* Add DPoP client conformance scenario + shared DPoP helpers (SEP-1932, #368) Client-anchored packaging: carries the shared DPoP foundation (dpopProof / dpopToken helpers, the createAuthServer DPoP core, and sep-1932.yaml) together with the client scenario, so it can merge first; the server (#369) and authorization-server (#370) scenarios are follow-ups that depend on it. DPoP is treated as a draft/not-yet-official feature (mirroring the WIF SEP-1933 scenario): the scenario's source is `introducedIn: DRAFT_PROTOCOL_VERSION` and it is registered in draftScenariosList (informational, not scored). It is NOT registered in EXTENSION_IDS, since the ext-auth DPoP extension is still on a branch rather than merged. Client checks (the two extension MUSTs, RFC 9449 §7.1 / §4.2-4.3): - sep-1932-client-dpop-auth-scheme — token presented with the DPoP scheme. - sep-1932-client-fresh-proof — a fresh, well-formed DPoP proof per request. The test MCP server judges the client via an opt-in DPoP middleware (dpopResourceAuth) passed through createServer's authMiddleware hook; a compliant example client (covered by the Client Draft Scenarios loop) plus deliberately- broken bearer/replay variants drive the pass/fail acceptance tests. Co-Authored-By: Claude Opus 4.8 <[email protected]> * client/dpop: split into nonce-less baseline + nonce-required scenario Server-provided nonces are optional in RFC 9449 (AS §8 "MAY", RS §9 "can also choose"), and the nonce-less flow is the common case. The single auth/dpop scenario hard-wired nonce enforcement on both the test AS and MCP server, so a plain nonce-less proof was never accepted end-to-end and a client that only does plain proofs was never exercised on its happy path. Parameterize DPoPClientScenario with `requireNonce` and register it twice: - auth/dpop (requireNonce=false) — nonce-less baseline; emits the three baseline checks (token-request-proof, dpop-auth-scheme, fresh-proof). - auth/dpop-nonce (requireNonce=true) — prior behavior; adds as-nonce and rs-nonce (five checks). Check IDs are reused (no new IDs, no sep-1932.yaml change); the two nonce checks are emitted only by auth/dpop-nonce, so traceability stays intact. Add auth-test-dpop-no-nonce.ts (nonce-incapable but otherwise compliant) asserting SUCCESS against auth/dpop — proof that a client with no nonce handling still completes DPoP when the server does not require a nonce. Baseline negatives stay on auth/dpop (nonce-independent); the two nonce negatives are re-pointed to auth/dpop-nonce. Co-Authored-By: Claude Opus 4.8 <[email protected]> * client/dpop: round-4 nonce-surface correctness fixes Address review findings on the auth/dpop-nonce posture: - Record the token-request proof observation BEFORE the §8 nonce challenge, so a client that is challenged but does not retry is no longer mis-reported by token-request-proof as "never completed a token request" (its proof was just validated). (#2) - Record the resource-request jti/replay on ANY valid proof (before the §9 nonce gate), so fresh-proof is never asserted on zero evidence and a client that never honours the RS nonce fails ONLY rs-nonce. (#5) - Require challengeIssued && honored for as-nonce/rs-nonce SUCCESS, closing a vacuous-pass path (a client that pre-sends the nonce is never challenged). (#3a) - Gate the AS nonce observation on the authorization_code exchange, matching recordTokenRequestProof (a refresh exchange must not satisfy §8). (#3c) - Collapse duplicate shared token-flow check IDs (token-request, pkce-*) that the §8 round-trip re-POST would otherwise emit twice. (#4) - Example client: retry the token request only on a use_dpop_nonce error code, not any 400 carrying a DPoP-Nonce header (consistent with the RS path). (#6) - Fix the DpopTokenRequestObservation docstring ("last write wins" → sticky-failure). (#7) Adds an expectedSuccessSlugs option to the client test helper and tightens the two nonce negative tests to pin the now-clean behavior (token-request-proof SUCCESS for no-as-nonce; only rs-nonce fails for no-rs-nonce). Co-Authored-By: Claude Opus 4.8 <[email protected]> * client/dpop: round-5 fixes — dedupe masking, wording, replay decision - dedupeSharedChecks was masking failures: it kept the last occurrence of a duplicate check ID regardless of status and ran in BOTH postures, so a FAILURE on a challenged POST (or an earlier attempt in a restarted flow) was silently dropped. Replace with an exported, unit-tested collapseDuplicateChecks that prefers the MOST-SEVERE occurrence, and only apply it in the nonce posture (the baseline keeps genuinely distinct repeated attempts). (R5 #1) - Reword the token-request-proof description: it can be SUCCESS for a client that presented a valid proof but never obtained a token (challenged, no retry), so it no longer claims "obtaining a DPoP-bound access token". (R5 #2) - Reword the as-nonce/rs-nonce failure messages to cover a client that DID retry but whose retry proof was rejected (not only "did not retry"). (R5 #3) - Document the deliberate replay decision: recording jti on challenged requests means re-sending the identical proof across the challenge/retry boundary trips replay detection — a genuine RFC 9449 §4.2 jti-uniqueness violation. (R5 #5) Adds src/scenarios/client/auth/dpop.test.ts unit-testing collapseDuplicateChecks (prefers failures, keeps INFO, preserves order) — the dedupe fix is now pinned. Co-Authored-By: Claude Opus 4.8 <[email protected]> * client/dpop: round-6 polish — pin collapse wiring; docstring - Add an integration assertion that the §8/§9 double-POST is collapsed to one token-request/pkce-* entry each in auth/dpop-nonce, so the requireNonce gating of collapseDuplicateChecks can't be silently deleted or inverted (previously mutation-survivable). runClientAgainstScenario now returns the emitted checks so callers can make occurrence-level assertions (backward-compatible). - Docstring: note the severity ladder ranks SKIPPED below SUCCESS, and that equal-severity ties keep the last occurrence. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
This is a very simplified conformance tests suit with: