mcp: concurrent OAuth refreshes across locations invalidate rotating refresh tokens and delete the credential #60612

Activity

1. opencodejr commented on Oct 30, 2026

   

   opencodejrBot

   on Oct 30, 2026

   Contributor

   More actions

   Thanks for the detailed report and logs. I could reproduce this on the latest dev (65c0d83), using a test against OpenCode's MCP OAuth code rather than the real Grafana or Buildkite servers. That code is unchanged since v2.2.0.

   The test sets up a local token server that accepts only the newest refresh token, and two connections in different locations that share one stored credential with an expired access token:

   1. The first location refreshes successfully (r1 -> r2).
   2. The second location starts its refresh just after the first one is answered, while the new tokens are still being saved. It reads the old refresh token r1 from storage and sends it again.
   3. The server rejects it with invalid_grant, OpenCode logs mcp oauth credential invalidated, and the shared credential is removed. The first location was fine, but the server now needs authentication everywhere.

   Concurrent refreshes only share one request while that request is still running. Once the response arrives, the shared request is dropped before the new tokens are saved, so a connection that reads the stored credential in that gap gets the old token. The check that avoids deleting a credential another connection just refreshed doesn't help here, because the new tokens haven't been saved yet. Nothing coordinates refreshes between separate processes either.

   Related:

   • fix(core): reuse MCP OAuth refresh answers for spent tokens #53275 (open PR): keeps successful refresh answers briefly, so a connection that sends the old token gets the same result.
   • V2 MCP OAuth: serialize credential refresh across locations/processes #34520 tracked this race and was closed in favour of fix(core): serialize MCP OAuth refreshes across processes #50994 (a lock across processes), which was closed without being merged.
   • V2 provider OAuth refresh races across locations and processes #50759: the same race for provider OAuth credentials.

   The script below reproduces it (run it with bun test in packages/core of a dev checkout).

   Reproduction script (mcp-oauth-race.test.ts)

   // Reproduction for anomalyco/opencode#60612: two locations sharing one MCP OAuth credential refresh
   // against a server that rotates refresh tokens. The second location reads the stored refresh token
   // after the first refresh was answered but before its new tokens were saved, replays the spent token,
   // gets invalid_grant, and the shared credential is deleted.
   import { expect, test } from "bun:test"
   import { auth } from "@modelcontextprotocol/client"
   import { ConfigMCP } from "@opencode/schema/config/mcp"
   import { Credential } from "@opencode/core/credential"
   import { Integration } from "@opencode/core/integration"
   import { McpOAuth } from "@opencode/core/mcp/oauth"
   import { Effect } from "effect"
   
   const integrationID = Integration.ID.make("mcp_test")
   const methodID = Integration.MethodID.make("oauth")
   const id = Credential.ID.make("cred_test")
   const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
   
   test("concurrent refreshes from two locations keep the shared credential", async () => {
     // Authorization server that rotates refresh tokens: only the newest one is accepted.
     let current = "r1"
     let generation = 1
     const log: string[] = []
     const server = Bun.serve({
       port: 0,
       async fetch(request) {
         const url = new URL(request.url)
         if (url.pathname === "/.well-known/oauth-authorization-server")
           return Response.json({
             issuer: url.origin,
             authorization_endpoint: `${url.origin}/authorize`,
             token_endpoint: `${url.origin}/token`,
             response_types_supported: ["code"],
             grant_types_supported: ["authorization_code", "refresh_token"],
           })
         if (request.method === "POST" && url.pathname === "/token") {
           const body = new URLSearchParams(await request.text())
           const presented = body.get("refresh_token")
           if (presented !== current) {
             log.push(`refresh ${presented} -> invalid_grant`)
             return Response.json(
               { error: "invalid_grant", error_description: "invalid, expired, or already rotated refresh token" },
               { status: 400 },
             )
           }
           generation++
           current = `r${generation}`
           log.push(`refresh ${presented} -> ${current}`)
           return Response.json({ access_token: `a${generation}`, token_type: "Bearer", refresh_token: current, expires_in: 3600 })
         }
         return new Response(null, { status: 404 })
       },
     })
     const url = server.url.href
   
     // Shared global credential row; writes take 30 ms like a real database write under load.
     const rows = new Map<string, Credential.Info>([
       [
         id,
         new Credential.Info({
           id,
           integrationID,
           label: "test",
           value: {
             type: "oauth",
             methodID,
             access: "a1",
             refresh: "r1",
             expires: Date.now() - 1000,
             metadata: { serverUrl: url, tokenType: "Bearer" },
           },
         }),
       ],
     ])
     const unused = () => Effect.die("unused")
     const service = Credential.Service.of({
       lock: unused,
       all: unused,
       create: unused,
       activate: unused,
       list: (i) => Effect.sync(() => [...rows.values()].filter((r) => r.integrationID === i)),
       get: (i) => Effect.sync(() => rows.get(i)),
       update: (i, updates) =>
         Effect.promise(() => sleep(Number(process.env.WRITE_MS ?? 30))).pipe(
           Effect.andThen(
             Effect.sync(() => {
               const row = rows.get(i)
               if (row) rows.set(i, new Credential.Info({ ...row, ...updates }))
             }),
           ),
         ),
       remove: (i) => Effect.sync(() => void (log.push(`credential ${i} removed`), rows.delete(i))),
     })
   
     const config = new ConfigMCP.Remote({ type: "remote", url, oauth: { client_id: "client" } })
     const connect = (directory: string) =>
       Effect.runPromise(
         Effect.gen(function* () {
           const provider = yield* McpOAuth.connectProvider({ config, integrationID })
           const fetchFn = yield* McpOAuth.loggedFetch({ server: "test", directory })
           return { provider, fetchFn }
         }).pipe(Effect.provideService(Credential.Service, service)),
       )
     const main = await connect("/main")
     const worktree = await connect("/worktree")
   
     // Location 1 refreshes; location 2 hits its expired token right as location 1's refresh is answered.
     const first = auth(main.provider, { serverUrl: url, fetchFn: main.fetchFn }).catch((e) => `error: ${e}`)
     // Wait until the server has answered location 1 (its new tokens are not saved yet).
     while (log.length === 0) await sleep(1)
     const second = auth(worktree.provider, { serverUrl: url, fetchFn: worktree.fetchFn }).catch((e) => `error: ${e}`)
     const results = await Promise.all([first, second])
     await sleep(100)
     server.stop(true)
   
     console.log("results:", results)
     console.log("server log:", log)
     console.log("stored credential:", rows.get(id)?.value)
   
     // Expected: one refresh shared by both locations and the credential kept with the rotated token.
     expect(rows.has(id)).toBe(true)
   })

   Run it: bun mcp-oauth-race.test.ts
2. opencodejr

   added

   reproduced

   severity:high

   on Oct 30, 2026
3. Antonio-Faure commented on Oct 31, 2026

   

   Antonio-Faure

   on Oct 31, 2026

   More actions

   Additional analysis from reading packages/core/src/mcp/oauth.ts:

   • send deduplicates only in-flight refreshes (l.31-55) and removes the entry as soon as the response arrives, before saveTokens (l.306-316) persists the rotated token.
   • A second location can therefore read the old refresh token, wait for the first request to finish, then send the same token. invalidate (l.292-298) sees the row still holding the old token and removes the shared credential before the winner's update runs.
   • Minimal fix: keep the successful refresh response for ~60s keyed by the consumed refresh token and replay it to callers still presenting that token; guard saveTokens so a late write cannot overwrite a newer rotation; do not remove the credential on invalid_grant.
   • Test: two connections sharing one in-memory credential, delay the first saveTokens, assert one network refresh and refresh === "r2" afterwards.

   The cross-process race remains because this path does not use Credential.lock (credential.ts:46-50,98-114), so that would be a separate follow-up.
4. ggeoffreyginer-hub commented on Nov 4, 2026

   

   ggeoffreyginer-hub

   on Nov 4, 2026

   More actions

   Independent confirmation on macOS / stable channel, with a third rotating-refresh-token provider (Noota, https://mcp.noota.io/mcp).

   • opencode 1.18.32, macOS; one mcp-auth.json shared by several long-running sessions opened in different working directories.
   • Same signature as this report: two failed to get resources errors 1-47 ms apart within the same run, then the credential is gone (.tokens removed from ~/.local/share/opencode/mcp-auth.json) and every session shows "needs authentication". Recurs every 2-5 days of normal multi-session use; full browser re-auth required each time (logs since Sep 19).
   • Additional trigger observed: after an MCP stream reconnect (global event disconnected / global event connected), the next resource fetch fails with Unauthorized and the credential is purged, even though the stored access token was still within its stated 4 h validity window at that moment.

   The provider's engineering team confirmed the following to us in writing, which directly supports the fix proposed above:

   • When an already-consumed refresh token is presented, only that request is rejected (400 invalid_grant); the refreshed tokens obtained by the winning request remain fully valid.
   • There is no grace period, and no server-side revocation of other tokens.
   • Their documented recovery path matches @Antonio-Faure's analysis: after an invalid_grant, the client should re-read the freshly stored tokens and retry with the new refresh token; a single-flight refresh with a shared lock on the token storage is the recommended durable design.

   Happy to provide anonymized logs or help reproduce against the Noota MCP if useful.
5. Antonio-Faure commented on Nov 4, 2026

   

   Antonio-Faure

   on Nov 4, 2026

   More actions

   Thanks for the independent confirmation and for getting the provider's statement in writing.

   This matches the mechanism in packages/core/src/mcp/oauth.ts: the in-flight map only shares a refresh while the request is running, and the row is removed before the winning saveTokens has persisted the rotated token. A second process that read the old token can therefore replay it, get invalid_grant, and — because the stored row still holds the old token at that point — delete the shared credential.

   The durable fix is the one in #60944: run the refresh under Credential.lock (in-process sharing plus the flock file lock), re-read the row after acquiring the lock, skip the refresh if another connection already replaced the rejected access token, and save the rotated tokens under the same lock. That is the single-flight plus shared-storage lock the provider recommends.

   One practical blocker: #60944 is currently 152 commits behind dev and GitHub reports mergeable_state: dirty, so it needs a rebase before it can land.

   If you can share the anonymized logs — timestamps of the two failed to get resources errors, whether the .tokens entry was removed, and the invalid_grant response body — that would let us verify the fix against Noota as well.
6. ggeoffreyginer-hub commented on Nov 4, 2026

   

   ggeoffreyginer-hub

   on Nov 4, 2026

   More actions

   Thanks for the quick analysis, and for pointing to #60944: the Credential.lock + re-read-after-acquire design matches exactly what the provider recommends (single-flight refresh plus shared-storage lock).

   Here is the anonymized evidence you asked for.

   1. Timestamps of the failed resource fetches (opencode 1.18.32, macOS; UTC, from ~/.local/share/opencode/log/opencode.log; both lines of each pair belong to the same run):

   2026-09-19T17:06:47.371Z / 17:06:47.371Z   same ms          error="MCP error -32001: Request timed out" x2
   2026-10-04T14:53:55.136Z / 14:53:55.249Z   113 ms apart     error=Unauthorized x2
   2026-10-11T20:49:33.179Z / 20:49:33.180Z   1 ms apart       error="fetch failed" x2
   2026-10-12T21:10:53.740Z / 21:10:53.741Z   1 ms apart       error="fetch failed" x2
   2026-10-17T15:33:49.574Z / 15:33:49.710Z   136 ms apart     error=Unauthorized x2
   2026-10-21T21:25:13.405Z / 21:25:13.406Z   1 ms apart       error=Unauthorized x2
   2026-10-21T22:37:22.679Z / 22:37:22.679Z   same ms          error="MCP error -32001: Request timed out" x2
   2026-10-22T16:13:53.813Z / 16:13:54.771Z   958 ms apart     timed out, then Unauthorized
   2026-11-04T00:53:19.958Z                                    error=Unauthorized
   2026-11-04T00:55:34.272Z / 00:55:34.319Z   47 ms apart      error=Unauthorized x2
   

   The 2026-11-04 00:53/00:55 UTC events were each preceded by global event disconnected then global event connected (00:53:14.424 -> 00:53:17.783, and 00:55:25.781 -> 00:55:31.882).

   2. Whether the .tokens entry was removed: yes.

   • 2026-11-04 00:53-00:55 UTC: the events above.
   • 2026-11-04 02:40 local: our external watchdog first observed the server as "needs authentication".
   • 2026-11-04 12:22 local: jq '.noota.tokens.expiresAt' ~/.local/share/opencode/mcp-auth.json returned null, i.e. the tokens object was gone from the stored entry, while the entry itself (clientInfo, serverUrl) was still present.
   • 2026-11-04 12:23 local: after a manual OAuth re-auth, tokens is back (accessToken, expiresAt, refreshToken).

   3. invalid_grant response body: not available from this installation.

   1.18.32 logs only the resource-fetch level errors (Unauthorized, fetch failed, Request timed out) and does not log the refresh request/response. Grepping the full log for invalid_grant returns nothing.

   Happy to run a build with #60944 (once it is rebased) against the Noota MCP with debug logging, to capture the response body and verify the fix against Noota as well.
7. Antonio-Faure commented on Nov 4, 2026

   

   Antonio-Faure

   on Nov 4, 2026

   More actions

   Thanks — this is exactly the evidence needed.

   The .tokens removal while clientInfo and serverUrl remain is the current invalidateCredentials path: it deletes the credential when the stored refresh token still equals the one this connection presented. The same-millisecond / 1 ms / 47 ms pairs, and the 00:53 and 00:55 UTC events right after global event disconnected / connected, match the mid-session 401 path, not just connect-time. No need to chase the invalid_grant body; the stored-token disappearance plus the provider's statement is conclusive.

   #60944 is the right fix but is currently 152 commits behind dev and GitHub reports it as dirty. I flagged that on the PR. Once it is rebased, your offer to run a Noota build with debug logging would be the best verification; I will reply here when a rebased branch is available.
8. rekram1-node

   closed this as completedin #62827on Nov 9, 2026