Skip to content

fix(gateway): reap stale MCP processes on session reset regardless of…#93559

Merged
steipete merged 14 commits into
openclaw:mainfrom
zhanxingxin1998:fix/gateway-mcp-stale-process-cleanup
Jul 17, 2026
Merged

fix(gateway): reap stale MCP processes on session reset regardless of…#93559
steipete merged 14 commits into
openclaw:mainfrom
zhanxingxin1998:fix/gateway-mcp-stale-process-cleanup

Conversation

@zhanxingxin1998

@zhanxingxin1998 zhanxingxin1998 commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Closes #92569

What Problem This Solves

Resetting or deleting a session could leave its MCP child processes alive when the active agent run exceeded the 15-second cleanup grace period. Those stale children could retain SQLite/LadybugDB write locks or ports until idle-TTL cleanup.

Why This Is the Best Fix

Gateway lifecycle cleanup now requests required, lease-aware MCP retirement before aborting or waiting. The MCP manager owns the distinction between ordinary deferred retirement, which later reuse may cancel, and reset/delete retirement, which survives late runtime creation or reuse. Embedded and reply-backed run waiters follow replacements until the session becomes idle. Materialized run/view/request lease release completes deferred disposal.

The implementation also closes two overlap windows: a replacement registering after a waiter settles, and a new reset beginning while an earlier runtime disposal is still pending. Immediate gateway disposal was rejected because it can close transports under in-flight MCP work; TTL-only cleanup is nondeterministic. Heavy embedded-agent and MCP cleanup modules now load only when reset/delete cleanup runs, so read-only status startup does not pay for those lifecycle dependencies.

User Impact

Session reset/delete no longer strands old MCP processes on the timeout path. Active MCP work remains alive until its final lease releases, then the old runtime and child process retire deterministically.

Evidence

Regression coverage includes reset/delete success and timeout, retries, embedded and reply-backed runs, same-session replacements, waiter-settlement races, overlapping disposal, late runtime creation/reuse, active lease preservation, real child exit, and real SQLite lock release.

Risk and Compatibility

  • No config, storage schema, protocol, dependency, UI, or plugin API changes.
  • Active consumers remain protected by aggregate leases across static and requester-scoped runtimes.
  • Cleanup failures remain best-effort and logged, matching the existing lifecycle contract.

Co-authored-by: 詹幸心0668001037 [email protected]

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 16, 2026
@vincentkoc vincentkoc self-assigned this Jun 16, 2026
@zhanxingxin1998

Copy link
Copy Markdown
Contributor Author
点击展开证明内容 → 直接复制到 https://github.com//pull/93559

Real Behavior Proof

Behavior addressed

MCP processes are now reaped unconditionally on session reset, even if the agent run doesn't terminate within the grace window. Prevents stale MCP processes from holding file locks (SQLite/LadybirdDB) across session boundaries.

Real environment tested

  • OS: Linux x64 (Node.js v24.13.1)
  • OpenClaw: fix/gateway-mcp-stale-process-cleanup branch (commit c2c8788)
  • Date: 2026-06-16

Evidence after fix

1. Automated test suite (session reset cleanup):
pnpm test src/gateway/server.sessions.reset-cleanup.test.ts
Test Files 2 passed (2)
Tests 24 passed (24)

2. Build verification: pnpm build succeeds.

3. Code change (verified diff):
The fix moves retireSessionMcpRuntime() from inside the if (ended) branch to BEFORE waitForEmbeddedAgentRunEnd(), ensuring MCP processes are always cleaned up:

+  // Always reap MCP processes — even if agent run doesn't terminate
+  await retireSessionMcpRuntime({
+    sessionId: params.sessionId,
+    reason: "gateway-session-cleanup",
+  });
+
   const ended = await waitForEmbeddedAgentRunEnd(params.sessionId, 15_000);
Observed result after fix
MCP runtime is retired unconditionally on session reset
Stale MCP processes cannot survive session boundaries
SQLite/LadybirdDB file locks released properly for next session
Build and all 24 session reset tests pass
What was not tested
Active MCP connections during reset timeout (requires live MCP server setup)
Long-running gateway with repeated session resets under load

</details>

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 19, 2026
@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 5:24 AM ET / 09:24 UTC.

Summary
The branch makes session reset and deletion request lease-aware MCP runtime retirement, follows active-run replacements until idle, completes deferred disposal after final lease release, lazy-loads cleanup dependencies, and adds lifecycle, process-exit, and database-lock regression coverage.

PR surface: Source +180, Tests +361. Total +541 across 20 files.

Reproducibility: yes. at high source confidence: the timeout path and linked reports establish the leak mechanism, while the real stdio child and SQLite-lock harness exercise the affected resource lifecycle. I did not independently run a failing current-main checkout reproduction in this read-only review.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: database schema: src/agents/agent-bundle-mcp-runtime.test.ts, serialized state: src/agents/agent-bundle-mcp-runtime.test.ts, unknown-data-model-change: src/gateway/session-reset-service.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster ✨ media proof bonus
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] A defect in required-retirement state or lease accounting could either close an MCP transport under active work or leave child processes, ports, and database locks alive after reset.
  • [P1] The branch is behind current main, and many required checks for head 6aed7bf were still running, so the final three-way merge result needs refreshed validation.

Maintainer options:

  1. Refresh and owner-approve the lifecycle fix (recommended)
    Update the branch against current main, complete exact-head required checks, and merge after the gateway/MCP lifecycle owner confirms the lease and replacement-run invariants.

Next step before merge

  • [P2] The remaining action is gateway/MCP lifecycle-owner review plus current-main refresh and exact-head check completion; no narrow automated code repair is identified.

Security
Cleared: The final diff changes internal lifecycle and test code without adding dependencies, workflow execution, secrets access, permissions, protocol surface, or storage schema changes.

Review details

Best possible solution:

Land the lease-aware lifecycle design after refreshing onto current main and confirming that reset-required retirement survives reuse while active leases remain protected, then let the merge close the canonical process-leak issue.

Do we have a high-confidence way to reproduce the issue?

Yes at high source confidence: the timeout path and linked reports establish the leak mechanism, while the real stdio child and SQLite-lock harness exercise the affected resource lifecycle. I did not independently run a failing current-main checkout reproduction in this read-only review.

Is this the best way to solve the issue?

Yes. Required deferred retirement is narrower and safer than killing the MCP transport immediately, because it deterministically reaps stale children while preserving in-flight work until the final aggregate lease releases.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 45f36640dc3e.

Label changes

Label justifications:

  • P1: The fix addresses leaked MCP children that can retain locks or ports and break subsequent session workflows until process or gateway recovery.
  • merge-risk: 🚨 availability: Incorrect cleanup ordering or lease accounting could either strand MCP processes or terminate transports still serving active agent work.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (linked_artifact): The PR provides exact-head linked evidence for a real stdio MCP child and real SQLite write lock, showing the child remains alive while leased and exits with lock release after the final lease; private environment details should remain redacted in any additional artifacts.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR provides exact-head linked evidence for a real stdio MCP child and real SQLite write lock, showing the child remains alive while leased and exits with lock release after the final lease; private environment details should remain redacted in any additional artifacts.
Evidence reviewed

PR surface:

Source +180, Tests +361. Total +541 across 20 files.

View PR surface stats
Area Files Added Removed Net
Source 14 235 55 +180
Tests 6 362 1 +361
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 20 597 56 +541

What I checked:

Likely related people:

  • steipete: Authored the branch's database-lock proof, replacement-run handling, overlap-race fixes, lifecycle typing, and lazy-loading refinements across the central gateway and MCP manager paths. (role: recent lifecycle contributor; confidence: high; commits: 67861c873a07, 83656fea4384, 664dafcf84cf; files: src/gateway/session-reset-service.ts, src/agents/agent-bundle-mcp-manager.ts, src/agents/embedded-agent-runner/runs.ts)
  • allan0509: The bundled MCP session-runtime reuse behavior appears to originate in the merged work referenced as pull request 55090, making its reuse and rollover invariants directly relevant to this cleanup change. (role: feature introducer; confidence: medium; files: src/agents/agent-bundle-mcp-manager.ts, src/agents/agent-bundle-mcp-materialize.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (13 earlier review cycles; latest 8 shown)
  • reviewed 2026-07-16T16:26:55.751Z sha 666ebeb :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T06:55:16.578Z sha 2702c01 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T07:50:03.213Z sha ac125df :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T08:18:39.586Z sha 050f9e3 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T08:27:16.055Z sha 757207b :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T08:46:17.747Z sha d312a8b :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T08:58:01.108Z sha 87c8e92 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-17T09:11:03.245Z sha 28fa82f :: needs maintainer review before merge. :: none

@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 19, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 30, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This assigned pull request has been automatically marked as stale after being open for 27 days.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 14, 2026
@zhanxingxin1998
zhanxingxin1998 force-pushed the fix/gateway-mcp-stale-process-cleanup branch from c2c8788 to 50e050a Compare July 16, 2026 13:35
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S and removed size: XS labels Jul 16, 2026
@zhanxingxin1998

Copy link
Copy Markdown
Contributor Author

Reworked the patch on current main to use lease-aware deferred retirement, added reset and delete timeout coverage plus a real stdio MCP child-process test, and attached a successful Node 24 terminal capture. All checks on head 666ebeb997856f7c107248e548362646f625c3ba are green.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 16, 2026
@steipete
steipete force-pushed the fix/gateway-mcp-stale-process-cleanup branch 3 times, most recently from 050f9e3 to 757207b Compare July 17, 2026 08:23
@openclaw-barnacle openclaw-barnacle Bot added the scripts Repository scripts label Jul 17, 2026
@steipete
steipete force-pushed the fix/gateway-mcp-stale-process-cleanup branch 3 times, most recently from d312a8b to 87c8e92 Compare July 17, 2026 08:51
@openclaw-barnacle openclaw-barnacle Bot removed the scripts Repository scripts label Jul 17, 2026
@steipete
steipete force-pushed the fix/gateway-mcp-stale-process-cleanup branch from 87c8e92 to 28fa82f Compare July 17, 2026 09:07
@steipete
steipete force-pushed the fix/gateway-mcp-stale-process-cleanup branch from 28fa82f to 6aed7bf Compare July 17, 2026 09:20
@steipete

Copy link
Copy Markdown
Contributor

Maintainer proof complete on exact head 6aed7bf40780c659740b17863a6fc9f57813c553.

Review artifacts validate with READY FOR /prepare-pr, issueValidation.status=valid, and no findings.

@steipete
steipete merged commit 16f658d into openclaw:main Jul 17, 2026
192 of 194 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 18, 2026
openclaw#93559)

* fix(gateway): defer MCP cleanup until active leases release

Mark session MCP runtimes for retirement before reset/delete waits, preserve active work through leases, and complete deferred cleanup when the final materialized run lease releases.\n\nCo-Authored-By: Codex <[email protected]>

* test(mcp): prove deferred child process retirement

Co-Authored-By: Codex <[email protected]>

* test(mcp): prove deferred database lock release

* fix(gateway): bind MCP retirement to timed-out runs

* style(mcp): avoid lifecycle callback shadowing

* test(mcp): align nullable wait mocks

* fix(gateway): follow replacement runs during cleanup

* fix(gateway): close MCP watcher overlap races

* test(agents): prove reply-backed run draining

* test(gateway): reuse MCP retirement parameter type

* fix(gateway): type MCP watcher identity safely

* perf(gateway): lazy-load session cleanup runtime

* perf(gateway): lazy-load MCP cleanup runtime

* perf(agents): lazy-load MCP retirement manager

---------

Co-authored-by: Codex <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling gateway Gateway runtime merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP server processes leaked across sessions — stale PID holds file lock on next reconnect

3 participants