Skip to content

feat(core): keep MCP tools after a dropped connection and reconnect on demand - #62756

Open
rekram1-node wants to merge 2 commits into
devfrom
mcp-resilience
Open

rekram1-node wants to merge 2 commits into
devfrom
mcp-resilience

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

When an MCP server's connection drops (a stdio server crashes or exits, or a legacy HTTP session expires and the reconnect fails), its tools disappear at once and stay gone until someone reconnects it from /mcps. In Code Mode the model is told the tools are "no longer available", and later that they are "new". Slow servers also delay the first step or join late, because nothing is known about their tools until they answer.

Change

Behind the mcp_resilience flag (on in local and dev builds):

  • Tools kept, reason shown. A connection that goes away on its own (process exit, closed transport, expired session) keeps offering the same tool objects, so the registry and Code Mode see no change. The server shows failed with the reason, including the end of a local server's stderr, plus "(tools kept; reconnects when next used)". The existing failed status is reused, so every client already shows the reason and older clients don't break on an unknown status.
  • Reconnect on demand, no background retries. The next request that needs the server (a tool call, prompt, resource read or listing) reconnects it under the server lock. Concurrent requests share that one attempt, which is bounded by the startup timeout. An expired session reconnects at once, because the request that found it is waiting.
  • Backoff on failure. A failed reconnect fails the request with the real error and updates the status. For 1 s after that, doubling to 30 s on consecutive failures, requests fail at once with the last error and nothing is spawned. A success resets the backoff. An auth failure goes to needs_auth and drops the tools, as before.
  • Explicit actions win. Disconnecting, removing or reconfiguring a server removes its tools at once, as before.
  • Disk cache. Each server's last listed tools are written to <cache>/mcp/tools/<sha256>.json. The hash covers McpPool.key (the transport, command or URL, resolved cwd, fingerprints of the environment, headers and OAuth secret, protocol, timeouts and client info) plus the signed-in credential ID and the workspace. Secrets reach it only as fingerprints inside the one-way hash. At startup the cached tools are offered right away and the first step doesn't wait for that server. The live list replaces them once it connects; if it fails to start, they are dropped. Servers still start eagerly.

With the flag off, behavior is unchanged.

Known limits

  • A remote server that silently loses its network is not detected: its calls fail while its tools stay listed, as today.

Tests

New cases under "MCP reconnect" in packages/core/test/mcp.test.ts. They use an in-process stdio server behind a fake spawner and TestClock:

  • a dropped server keeps its tools and shows the reason with stderr; nothing respawns in the background; the next call reconnects, and two concurrent calls share one attempt
  • a failing reconnect returns its error; within the backoff a call fails at once without spawning; afterwards a call tries again, and the backoff doubles
  • an expired session whose reconnect needs authorization goes to needs_auth, drops its tools and isn't retried
  • an explicit disconnect clears the tools; with the flag off, behavior is unchanged
  • cached tools are offered while a server starts, a config change uses a different cache file, and no secret appears in the file

bun run test in packages/core, bun typecheck, bun run typecheck:tsc and root bun run check pass.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round">
  <style>
    .plug { animation: drop 3s ease-in-out infinite; }
    .spin { transform-origin: 120px 36px; animation: spin 3s linear infinite; opacity: 0; }
    @keyframes drop { 0%, 30%, 70%, 100% { transform: translateX(0); } 40%, 60% { transform: translateX(-10px); } }
    @keyframes spin { 0%, 35% { opacity: 0; transform: rotate(0deg); } 40%, 60% { opacity: .6; } 65%, 100% { opacity: 0; transform: rotate(360deg); } }
    @media (prefers-reduced-motion: reduce) { .plug, .spin { animation: none; } }
  </style>
  <rect x="20" y="22" width="44" height="28" rx="6" opacity=".7"/>
  <g class="plug"><path d="M64 36h34"/><path d="M98 30v12"/></g>
  <path d="M142 30v12"/><path d="M142 36h20"/>
  <circle class="spin" cx="120" cy="36" r="9" stroke-dasharray="14 42"/>
  <rect x="162" y="14" width="58" height="44" rx="6" opacity=".7"/>
  <path d="M172 26h38M172 36h38M172 46h26" opacity=".5"/>
</svg>

Added by the opencode bot (AI-generated).

Related: #53226, #52237

Public API: adds the experimental.mcp_resilience config flag (backwards compatible)

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works as described with the flag on. I ran OpenCode from source against a stdio MCP server that exits after it connects. On the base the server goes to failed and is never respawned. With mcp_resilience on, it is respawned at once, stays connected and keeps its tools. With a server that dies on every restart, retries back off 1 s, 2 s, 4 s as described. The new reconnect tests passed on 6 repeated runs. typecheck:tsc passes in packages/core. The only failing test in mcp.test.ts fails on the base too.

Public API: adds the experimental.mcp_resilience boolean to the config schema, the OpenAPI spec and the generated client types. It is additive and backwards compatible. It also adds an internal cache file format, <cache>/mcp/tools/<sha256>.json.

One concern, inline: a server that will never come back (broken command, crash on start, a remote that keeps refusing) now stays pending forever with its tools still listed and retries every 30 s with no limit. Today the user sees failed plus the error in /mcps; with this change they get no signal at all.

Related: #53226, #52237

Before

After

Comment thread packages/core/src/mcp/index.ts Outdated
}),
failed: (failure) =>
Effect.gen(function* () {
// A reconnect keeps trying past failures other than auth, so the server stays pending with its tools.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once a reconnect starts, every failure other than auth is swallowed here: the status stays pending, no StatusChanged is published, and revive keeps retrying every 30 s with no limit. I checked this with a server that exits on every restart: /mcps showed it as pending indefinitely, and the error (MCP server process exited with code 1) only appears in tool-call errors. For a server that's permanently gone (command removed, bad config, a remote that's down for good), the user loses today's failed status and error, the model keeps seeing tools that always fail, and a process is spawned every 30 s for the life of the service. Could you either give up after a few attempts (or after a total time) and fall back to the existing failed path, which drops the retained tools, or show the reconnect reason in the server's status so /mcps can display it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed by removing the background reconnect loop. A dropped server now shows failed with the reason (including the end of stderr) and keeps its tools. It reconnects only when a request needs it; after a failed attempt it backs off 1 s doubling to 30 s, failing at once with the last error, and auth failures still go to needs_auth and drop the tools.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new design deals with what I raised. The revive loop is gone, so a server that is permanently down no longer spawns every 30 s. It shows failed with its error, and it only spawns when something actually uses it, at most once per backoff window. But the current head doesn't build, so I couldn't check the new behaviour by running it.

What I checked in pr/head (53a8023):

  • drop (mcp/index.ts:412-421) keeps the last tools, sets failed with the reason and publishes StatusChanged.
  • connected/revive (:336-346) reconnects under the server lock. Within the backoff it doesn't start anything.
  • failed (:569-590) sets the backoff to 1 s and doubles it up to 30 s on each failure. It keeps the tools and publishes the status. needs_auth clears disconnected and drops the tools.
  • Disconnect, remove and replace all clear disconnected and the kept tools (disposeServer, :661).

Blocker: the rebase onto e42d622 broke the build. After #62686, Flag.enabled requires Flag.Service. Line 259 only provides Config.Service, so Flag still leaks into the effect's requirements:

  • tsc -b tsconfig.json tsconfig.tests.json fails with Effect<…, Service> not assignable to Effect<…, never> at src/mcp/index.ts lines 398, 598, 605 and 886.
  • bun run test test/mcp.test.ts: 63 pass, 50 fail with Service not found: @opencode/Flag. That includes every "MCP reconnect" case. pr/base passes 106 of 107.

The fix I tried:

  • const resilient = (yield* Flag.Service).enabled("mcp_resilience"), which makes configService unused.
  • Adding Flag.node to the deps of configured().

That moves the missing service onto the test harness: test/mcp.test.ts around line 473 builds the layer without Flag. It needs Flag.testLayer(...) or Flag.layer with App, taking Flag.Overrides from overrides.flags. I didn't finish that wiring, so with my change it still has 57 failures.

Small but model-facing: unavailable() (:94) wraps the drop message. During the backoff the model gets …Its tools stay listed; it reconnects when next used.. Reconnect it from /mcps., which has a double period and contradicts itself. Make unavailable skip the "/mcps" suffix while disconnected is set, or drop the trailing period from disconnectedError.

I made no changes in repo/; it is still on the older commit 66b6e5d with bun.lock modified, which I didn't touch. I tested in a separate worktree at /tmp/opencode/pr-head (detached at pr/head), and it's clean again.

@rekram1-node rekram1-node changed the title feat(core): keep MCP tools and reconnect after a dropped connection feat(core): keep MCP tools after a dropped connection and reconnect on demand Nov 8, 2026

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new design addresses my earlier point: a dropped server now shows failed with its reason and no longer respawns in the background. I checked this end to end with a stdio server that exits after connecting. The status showed the reason with "Its tools stay listed", nothing respawned, and the next Code Mode call to its tool started a new process and succeeded.

This head doesn't build, though. After the rebase, Flag.enabled() needs Flag.Service, so bun run typecheck:tsc in packages/core fails in src/mcp/index.ts. bun test test/mcp.test.ts fails 50 tests with Service not found: @opencode/Flag, including all the new "MCP reconnect" cases (see inline).

Public API: same as before. It adds the experimental.mcp_resilience boolean to the config schema, the OpenAPI spec and the generated client types (backwards compatible), plus the internal <cache>/mcp/tools/<sha256>.json cache file.

Comment thread packages/core/src/mcp/index.ts Outdated
const configService = yield* Config.Service
const global = yield* Global.Service
const fsutil = yield* FSUtil.Service
const resilient = Flag.enabled("mcp_resilience").pipe(Effect.provideService(Config.Service, configService))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the rebase, Flag.enabled is Flag.Service.use(...), so resilient still needs Flag.Service after providing Config.Service. As a result typecheck:tsc fails at lines 398, 598, 605 and 886, and 50 tests in mcp.test.ts (including the new reconnect ones) die with Service not found: @opencode/Flag. Reading it as const flags = yield* Flag.Service / flags.enabled("mcp_resilience") and adding Flag.node to the MCP layer's deps (in place of Config.node, if nothing else here needs Config) should fix it.

…n demand

Behind the mcp_resilience flag. A server whose connection drops on its own (process exit, closed transport, expired session) keeps offering its last tools and shows the reason as its status error. The next request that needs it reconnects, sharing one attempt; after a failed attempt requests fail at once with the last error for 1 s, doubling to 30 s. Auth failures go to needs_auth and drop the tools. Each server's last tool list is cached on disk, keyed by a hash of its connection identity, and offered while it starts.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier threads are fixed. MCP now reads the flag through Flag.Service and depends on Flag.node, so typecheck:tsc in packages/core is clean. test/mcp.test.ts passes except terminates MCP descendants after the wrapper exits successfully, which also failed on the base earlier.

Re-checked end to end with the flag on, using a stdio server that exits after connecting. The status showed failed with "(tools kept; reconnects when next used)", nothing respawned in the background, and the next Code Mode call to its tool started a new process and succeeded.

Public API is unchanged since the last review: it adds the experimental.mcp_resilience boolean to the config schema, the OpenAPI spec and the generated client types (backwards compatible), plus the internal <cache>/mcp/tools/<sha256>.json cache file.

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commit is sound: a tool call to a server that dropped on its own now tells the model it reconnects on the next call, with the last reason, instead of pointing it to /mcps, which it can't use. The existing failed/needs_auth messages are unchanged. test/mcp.test.ts passes except terminates MCP descendants after the wrapper exits successfully, which also failed on the base earlier, and typecheck:tsc in packages/core is clean. Both earlier threads were already addressed in the previous push.

Public API is unchanged since the last review: it adds the experimental.mcp_resilience boolean to the config schema, the OpenAPI spec and the generated client types (backwards compatible), plus the internal <cache>/mcp/tools/<sha256>.json cache file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant