Skip to content

test(admission): MCP client cancel and disconnect keep the worker's slot and land its write (#728, slice 3) - #936

Merged
github-actions[bot] merged 3 commits into
mainfrom
fix/728-mcp-protocol-cancellation
Nov 7, 2026
Merged

github-actions[bot] merged 3 commits into
mainfrom
fix/728-mcp-protocol-cancellation

Conversation

@cdeust

@cdeust cdeust commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Summary

Refs #728 (slice 3; does not close the issue).

The #728 tests merged so far (#885, #902/#903) cancel the awaiting asyncio task directly. This slice drives the cancellation the way a client sends it: a tools/call through the MCP SDK Client and MCPServer on the in-memory transport, while its handler (wrapped by the real safe_handler) runs, then either the client's notifications/cancelled or the client disconnecting. The handler writes one row to its own SQLite file under tmp_path.

Pinned for both paths:

  • the worker keeps its admission slot until it ends, not until the cancel;
  • abandoned_work counts it while it runs, then records it completed;
  • its write lands after the client stopped listening (a started write is not rolled back by a cancel);
  • the session survives a cancelled call (a second call succeeds), and teardown neither waits on nor kills the worker thread.

Test-only: no production code changes. No root cause to fix here; the defect was fixed in #885. This slice adds the protocol-level evidence asked by acceptance item 4.

Type of change

Test plan

  • pytest tests_py/server/test_mcp_protocol_cancellation.py: 2 passed, 5 consecutive runs (0.9–1.2 s each).
  • pytest tests_py/server tests_py/handlers/test_admission_cancellation.py: 29 passed.
  • Mutation check: a mutant of run_admitted that releases the slot at the caller's cancel turns both tests red (semaphore value 4 == 4 - 1 fails); a mutant that also drops the abandoned record fails them too.
  • ruff check and ruff format --check on the new file: clean. python scripts/check_craftsmanship.py: OK.
  • Orderings use threading.Event, never elapsed time; the only bound is a 5 s deadlock guard.

Audit notes

Not verified:

  • The store is a plain SQLite file, not a Cortex store: a real remember call needs the embedding model, which this environment could not install (torch download blocked). Running the same scenario against remember on an isolated SQLite Cortex store remains to do.
  • The transport is the SDK in-memory one, not stdio with a real Claude/Codex client.
  • Local env installed without torch (uv sync --locked ... --no-install-package torch); the full suite was not run.

Remaining #728 slices: the same scenario through a real mutating Cortex tool on an isolated store; a documented cooperative cancellation boundary for handlers that can stop early.

Coding-standards compliance

  • §2.2 Layer dependency direction preserved (test file only).
  • §8 Constants carry # source: annotations.
  • §9 No dead code, no TODOs.

Reviewer checklist

  • CHANGELOG.md updated under Added.
  • No secrets / credentials / PII in the diff.
  • CI passes on the latest commit (pending at opening).

Generated by Claude Code

…lot and land its write (#728, slice 3)

The earlier #728 tests cancel the awaiting task directly. These drive a
tools/call through the MCP SDK client and server on the in-memory
transport, then cancel it by notifications/cancelled, or by the client
leaving. In both paths the worker keeps its admission slot until it
ends, abandoned_work counts it and records it completed, its write to an
SQLite file under tmp_path lands after the client stopped listening, and
teardown neither waits on nor kills the worker thread.

Verified red against a mutant that releases the slot at the cancel
(semaphore value 4 for budget 4 while the worker runs), green on main.

Refs #728

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018kyZYsLJY3KEWTSwt6BCzs
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T07:36:07.716038Z 0205b08 New commits
🔒 Security Review ✅ Completed 2026-11-07T07:36:28.551302Z 0205b08 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @cdeust's task in 3m 16s —— View job


PR #936 review (review 1 of 3)

  • Read the diff
  • Ran five reviewer agents (code quality, docs accuracy, performance, security, test coverage)
  • Checked their findings against mcp_server/handlers/admission.py and mcp_server/tool_error_handler.py
  • Posted the verdict

I traced the Blocker below in the code but could not reproduce it. This runner can't install the MCP SDK or run Python, and the gh commands, including gh pr comment, need approval. So this review is posted in this job comment instead of a separate gh pr comment.

Security found nothing. The test-coverage reviewer confirmed two things. First, both tests really do cancel the call on the server side: record_abandoned is only reached from run_admitted's except CancelledError. Second, the mutant you described, releasing the slot at the cancel, makes _assert_abandoned_and_held fail.

Blocker

tests_py/server/test_mcp_protocol_cancellation.py:104-111: _wait_slot_returned waits a fixed number of loop turns, not until the 5 s guard, so the tests can fail at random on CI.

  • writer.ended is set in the handler's own finally (:83), before the worker thread has finished.
  • After that, the thread still has to:
    • finish run_until_complete
    • exit handler_transaction_scope
    • run loop.close() and metrics.Timer.__exit__ (tool_error_handler.py:87-110)
    • hand its result back to the main loop.
  • Only then does the to_thread future complete and _release_slot run (admission.py:93-94,103).
  • _wait_slot_returned gives this 100 turns of asyncio.sleep(0), about a millisecond. A thread waiting for the GIL can be held off for up to the 5 ms switch interval, so on a loaded runner the worker can miss that window. The test then fails with slot not returned even though the code under test is correct.
  • This also contradicts the PR's own rule that "the only bound is a 5 s deadlock guard".
  • Fix: bound this wait by time, the same way _until_abandoned is:
async def _until_slot_returned() -> None:
    sem = _get_semaphore(_TOOL)
    while sem._value != current_budget(_TOOL):  # noqa: SLF001
        await asyncio.sleep(0)

# in _finish_worker:
await asyncio.wait_for(_until_slot_returned(), _GUARD)

The completed == 1 check right after it stays sound: _release_slot and _log_abandoned run in the same callback batch when the worker finishes.

Fix this →

Notes, no action before merge

  • Only the disconnect test checks that "teardown neither waits on nor kills the worker". In the cancel test the worker has already ended (:143) before the client exits (:150), so the docstring (:15) and CHANGELOG should say this for one path, not "both".
  • "a started write is not rolled back by a cancel" (:13-14, CHANGELOG): at the cancel the handler is blocked before it writes (rows() == [] at :141). What the test actually shows is that a running handler still makes its write after the cancel.
  • # noqa: SLF001 (:108, :117) has no effect, because ruff does not select SLF. The private Semaphore._value read is the real coupling.

Verdict: block

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 354ce42dfa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +107 to +111
for _ in range(100):
if sem._value == current_budget(_TOOL): # noqa: SLF001
return
await asyncio.sleep(0)
raise AssertionError(f"slot not returned: {sem!r}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for worker completion with a time-bounded condition

When the worker thread is still finishing after writer.ended is set, 100 iterations of asyncio.sleep(0) can expire before its completion callback releases the semaphore. The event is set inside the handler, before _run_coroutine_on_thread closes its loop and the executor future completes, so it does not synchronize that callback. Repeated direct execution of the new cancellation test reproduced slot not returned ... value:3 on the third iteration with unchanged production code. Replace the fixed iteration count with a condition wait bounded by _GUARD, allowing the worker time to finish rather than making both tests scheduling-dependent.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in b1120ca. writer.ended is set inside the handler, before the worker thread closes its loop and the executor future completes, so counting 100 loop turns was a race. The waits are now _until(condition): polling at 1 ms, bounded by _GUARD through asyncio.wait_for. They cover the slot return, the abandoned outcome and the abandoned registration. 20 consecutive runs: 20 passed.


Generated by Claude Code

async def test_client_cancel_keeps_slot_and_write_lands(tmp_path: Path):
writer = _WritingHandler(tmp_path / "store.db")
try:
async with Client(_server(writer)) as client:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a stream-backed client to exercise protocol cancellation

With the locked MCP SDK 2.3.0, passing an MCPServer directly to the default Client(mode="auto") selects DirectDispatcher, not the in-memory transport; this was also confirmed at runtime. The SDK's _connect_inproc bypasses streams and JSON-RPC framing on this path, and _no_inbound_client_notifications explicitly documents cancellation as structural cancellation through the direct await, not a notification. Consequently, neither test exercises the claimed notifications/cancelled or transport-disconnect behavior, and protocol cancellation regressions can pass these tests. Use an explicit InMemoryTransport or mode="legacy" in both client constructions so the scenarios actually cross the protocol boundary.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in b1120ca. With mcp 2.3.0, Client(MCPServer) in the default "auto" mode goes through _connect_inproc → DirectDispatcher, so no JSON-RPC and no notifications/cancelled. Both tests now use Client(..., mode="legacy"), which puts the server behind InMemoryTransport + JSONRPCDispatcher. Checked with a probe on cancelled_request_id_from_params: the server receives notifications/cancelled for request id 2 in the cancel test. The slot-release mutant still turns both tests red.


Generated by Claude Code

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @cdeust's task in 1m 56s —— View job


PR #936 review (review 2 of 3)

  • Checked the review 1 Blocker against b1120ca
  • Ran five reviewer agents (code quality, docs accuracy, performance, security, test coverage) on the 354ce42..b1120ca delta
  • Checked their findings against mcp_server/handlers/admission.py, abandoned_work.py and tool_error_handler.py
  • Posted the verdict

This is posted in the job comment, the same way as review 1, rather than with gh pr comment. The MCP SDK is not installed on this runner, so I could not run the tests or check Client(mode="legacy") against SDK 2.3.0 myself. CI (Pipeline) was still running when I posted this.

Review 1 Blocker: fixed

The fixed 100-turn wait in _wait_slot_returned is gone. Its replacement, _until(condition) (test_mcp_protocol_cancellation.py:111-123), polls with asyncio.wait_for(..., _GUARD). _finish_worker now waits for the slot to come back and then for in_flight to empty, before it asserts completed == 1.

That order matches the code: _release_slot is registered before _log_abandoned (admission.py:94 vs :99). The new _POLL constant has a # source: comment.

Codex's second point is also addressed: both clients now use mode="legacy", so the call goes through the in-memory transport and JSON-RPC instead of DirectDispatcher.

The new code

  • Mutant still caught. A mutant that releases the slot at the cancel still fails _assert_abandoned_and_held. The held-slot check (:159, :187) runs before release is set (:138). A double release leaves _value at budget + 1, so _slot_returned times out.
  • No regression hidden. The extra in_flight wait does not hide a lost record (it times out), a double record_ended (a -1 count stays in the snapshot), or a wrong outcome (completed == 1 fails).
  • Security and performance: nothing found. The SQL uses parameters, the store is under tmp_path, and every wait has a time limit.

Notes, no action before merge

  • :18 and the CHANGELOG still say the teardown check holds for "both" paths, but only the disconnect test checks it.
  • :20 "never by elapsed time" is now only partly true, because orderings also depend on the condition polls in _until.
  • :143 assert snap["in_flight"] == {} can no longer fail, because :141 already waited for that exact condition.

Verdict: approve

cdeust commented Nov 7, 2026

Copy link
Copy Markdown
Owner Author

Revue Claude 1/3 (verdict block) : le Blocker (attente bornée par 100 tours de boucle au lieu de _GUARD) est celui que Codex a aussi relevé ; il est corrigé dans b1120ca (_until(condition) borné par asyncio.wait_for(..., _GUARD), 20/20 exécutions vertes).

Les trois notes sont justes et corrigées dans 0205b08 :

  • « teardown ne bloque pas » n'est prouvé que par le test de déconnexion ; la docstring et le CHANGELOG le disent maintenant pour ce seul test (le test d'annulation prouve, lui, que la session sert un nouvel appel).
  • « une écriture commencée n'est pas annulée » devient ce que le test montre : un handler encore en cours à l'annulation, bloqué avant d'écrire, fait son écriture après que le client a cessé d'écouter.
  • les # noqa: SLF001 sans effet (ruff ne sélectionne pas SLF) sont retirés.

Generated by Claude Code

@claude

claude Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @cdeust's task in 2m 56s —— View job


PR #936 review (review 2 of 3)

  • Check the review 1 Blocker against b1120ca / 0205b08
  • Run five reviewer agents on the 354ce42..0205b08 delta (code quality, docs accuracy, performance, security, test coverage)
  • Check their findings against admission.py, abandoned_work.py and tool_error_handler.py
  • Post the verdict

I couldn't run anything on this runner: Python and gh both need approval here, and the MCP SDK isn't installed. So the checks below come from reading the code, and the verdict is posted in this job comment instead of through gh pr comment.

Review 1 Blocker: fixed

_wait_slot_returned, which counted 100 loop turns, is replaced by _until(condition) (tests_py/server/test_mcp_protocol_cancellation.py:115-127). It polls every 1 ms and gives up after _GUARD via asyncio.wait_for. All three loop-side waits now use it: the slot return, the abandoned outcome and the abandoned registration.

_finish_worker (:144-148) has no race. record_ended lowers in_flight and records the completed outcome under the same lock (abandoned_work.py:41-45), so once in_flight is empty, completed is already recorded.

The Codex P2 finding (the default "auto" mode never crosses JSON-RPC) is addressed with mode="legacy" in both tests. I couldn't check this against the SDK source, but it can't make a test pass when it should fail: if no cancel reached the server, _until(in_flight) at :162 and :190 would time out. The slot-release mutant still fails _assert_abandoned_and_held in both tests.

The docs notes from review 1 are fixed too. The docstring and CHANGELOG now say which test covers the session-survives check and which covers teardown, and they describe the blocked-before-write case accurately. The # noqa: SLF001 comments, which had no effect, are gone. The new _POLL constant has a # source: comment. The file is 198 lines.

New Blockers

None.

Notes, no action before merge

  • :170-171: writer.started.clear() / writer.ended.clear() do nothing, since nothing waits on those events afterwards.
  • :147: assert snap["in_flight"] == {} can't fail, because the _until at :145 already waited for exactly that.
  • :186-187: the comment says the server cancels on read EOF. The test doesn't show that: the cancel could also come from the client task group being torn down. Either way the handler is cancelled, which is what the test checks.

Verdict: approve

@github-actions
github-actions Bot merged commit 4a196c2 into main Nov 7, 2026
18 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 7, 2026
…pletion (#728, slice 5) (#946)

* docs(admission): ADR-1125, a cancelled tool call's worker runs to completion (#728, slice 5)

Issue #728 asks to keep work counted "until completion or a documented
cooperative cancellation boundary". Slices 1-4 (#885, #903, #902, #936,
#943) implement and prove the first: the slot is released by the worker's
done callback, the caller awaits through asyncio.shield, abandoned workers
are reported in memory_stats.abandoned_workers. Nothing forwards the
cancellation into the worker thread, so no cooperative boundary exists.

ADR-1125 records those semantics, amends ADR-0329 (the slot belongs to the
worker), states what a handler that wants to stop early must specify, and
lists the tests that hold each point. Mirror regenerated with
wiki_decision_mirror.generate_mirror.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_018kyZYs8nydviS39BjwT88E

* fixup! docs(admission): ADR-1125, a cancelled tool call's worker runs to completion (#728, slice 5)

---------

Co-authored-by: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants