Skip to content

fix(client): add transparent retry for socket/ECONNRESET errors - #54154

Closed
96Ems wants to merge 2 commits into
anomalyco:devfrom
96Ems:transport-retry
Closed

96Ems wants to merge 2 commits into
anomalyco:devfrom
96Ems:transport-retry

Conversation

@96Ems

@96Ems 96Ems commented Oct 9, 2026 •

Copy link
Copy Markdown

Issue for this PR

Closes #54160

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds automatic retry with backoff for transient transport errors in the generated HTTP client.

Problem: When HTTP keep-alive connections go stale (common in enterprise environments with proxies/security agents), fetch() fails with "The socket connection was closed unexpectedly". Users must press Enter multiple times to retry manually.

Fix: Modified packages/httpapi-codegen/src/index.ts to generate retry logic. On socket/ECONNRESET errors, the client retries up to 3 times with 100ms/200ms/300ms delays before surfacing the error.

Why it works: The error occurs because Bun's connection pool reuses a closed connection. The socket error happens before the request reaches the server (connection establishment phase), so retrying is safe - we're not re-submitting an in-flight request. A retry with a fresh connection succeeds.

Why POST is safe to retry: The session.prompt endpoint (where this is most visible) is idempotent - the server deduplicates by message ID. More importantly, the socket error occurs during connection, before any request bytes are sent.

How did you verify your code works?

Tested on Ubuntu 22.04 VM with corporate proxy + CrowdStrike Falcon endpoint agent:

  • Before: required 2-3× manual Enter to send messages
  • After: transparent retry handles stale connections silently

Screenshots / recordings

N/A - not a UI change

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@github-actions github-actions Bot added needs:issue needs:compliance This means the issue will auto-close after 2 hours. labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work as submitted, and I couldn't confirm it fixes the reported error.

The PR changes only the generator. The client that OpenCode actually uses, packages/client/src/promise/generated/client.ts, isn't regenerated. Running bun run generate in packages/client rewrites that file, so this change has no effect for users yet and the check:generated CI step will fail. Please commit the regenerated output.

How I checked: with the client regenerated locally, a fetch that fails twice with "The socket connection was closed unexpectedly" succeeds on the third attempt. Aborting the request after a socket error still makes all three attempts, and the caller gets a Transport error instead of the abort (see the inline comment). I couldn't reproduce the original problem: it needs a proxy or endpoint agent that drops keep-alive connections. Also, #23123 and the similar open reports are about connections to model providers, which don't go through this client. The httpapi-codegen tests (97 pass) and typecheck pass.

Public API: the published @opencode/client changes behaviour. Failed requests are now retried silently, and errors building the URL or headers are no longer wrapped as ClientError("Transport"). No signatures change.

Other concerns (inline): retrying POST/PATCH requests that can't safely be repeated, the abort handling, and the broad "socket" message match.

Related: #53829, #51537

return `${JSON.stringify(group.identifier)}: { ${fields} }`
})
return `import type { ${imports.join(", ")} } from "./types.js"\nimport { ClientError } from "./client-error.js"\n\nexport interface ClientOptions {\n readonly baseUrl: string\n readonly fetch?: typeof globalThis.fetch\n readonly headers?: RequestInit["headers"]\n}\n\nexport interface RequestOptions {\n readonly signal?: AbortSignal\n readonly headers?: RequestInit["headers"]\n /** Reports every chunk a streaming response receives, including keepalive comments that yield no event. */\n readonly onActivity?: () => void\n}\n\ninterface RequestDescriptor {\n readonly method: string\n readonly path: string\n readonly query?: Record<string, unknown>\n readonly headers?: Record<string, unknown>\n readonly body?: unknown\n readonly successStatus: number\n readonly declaredStatuses: ReadonlyArray<number>\n readonly empty: boolean\n}\n\nconst maxSseEventBytes = 16 * 1024 * 1024\n\nexport function make(options: ClientOptions) {\n const fetch = options.fetch ?? globalThis.fetch\n\n const prepare = (descriptor: RequestDescriptor, requestOptions?: RequestOptions) => {\n // A leading slash would replace any path prefix on baseUrl, so join relative to it.\n const baseUrl = new URL(options.baseUrl)\n if (!baseUrl.pathname.endsWith("/")) baseUrl.pathname += "/"\n const url = new URL(descriptor.path.slice(1), baseUrl)\n for (const [key, value] of Object.entries(descriptor.query ?? {})) appendQuery(url.searchParams, key, value)\n const headers = new Headers(options.headers)\n for (const [key, value] of Object.entries(descriptor.headers ?? {})) {\n if (value !== undefined && value !== null) headers.set(key, String(value))\n }\n for (const [key, value] of new Headers(requestOptions?.headers)) headers.set(key, value)\n if (descriptor.body !== undefined && !headers.has("content-type")) headers.set("content-type", "application/json")\n return {\n url,\n init: {\n method: descriptor.method,\n signal: requestOptions?.signal,\n headers,\n body: descriptor.body === undefined ? undefined : JSON.stringify(descriptor.body),\n } satisfies RequestInit,\n }\n }\n\n const execute = async (descriptor: RequestDescriptor, requestOptions?: RequestOptions) => {\n try {\n const prepared = prepare(descriptor, requestOptions)\n return await fetch(prepared.url, prepared.init)\n } catch (cause) {\n throw new ClientError("Transport", { cause })\n }\n }\n\n const responseError = async (response: Response, descriptor: RequestDescriptor): Promise<never> => {\n if (descriptor.declaredStatuses.includes(response.status)) throw declared(await json(response) as DeclaredErrorBody)\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnexpectedStatus", { cause: { status: response.status }, detail: String(response.status) })\n }\n\n const request = async <A>(descriptor: RequestDescriptor, requestOptions?: RequestOptions): Promise<A> => {\n const response = await execute(descriptor, requestOptions)\n if (response.status !== descriptor.successStatus) return responseError(response, descriptor)\n if (descriptor.empty) {\n try {\n await response.body?.cancel()\n } catch {}\n return undefined as A\n }\n return await json(response) as A\n }\n\n const sse = <A>(descriptor: RequestDescriptor, requestOptions?: RequestOptions): AsyncIterable<A> => ({\n async *[Symbol.asyncIterator]() {\n const response = await execute(descriptor, requestOptions)\n if (response.status !== descriptor.successStatus) await responseError(response, descriptor)\n if (!isContentType(response, "text/event-stream")) {\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnsupportedContentType", { detail: response.headers.get("content-type") })\n }\n if (response.body === null) throw new ClientError("MalformedResponse")\n const reader = response.body.getReader()\n const decoder = new TextDecoder()\n let buffer = ""\n try {\n while (true) {\n let next: ReadableStreamReadResult<Uint8Array>\n try {\n next = await reader.read()\n } catch (cause) {\n throw new ClientError("Transport", { cause })\n }\n if (!next.done) requestOptions?.onActivity?.()\n buffer += decoder.decode(next.value, { stream: !next.done })\n if (buffer.length > maxSseEventBytes) throw new ClientError("SseEventTooLarge")\n const trailingCarriageReturn = !next.done && buffer.endsWith("\\r")\n if (trailingCarriageReturn) buffer = buffer.slice(0, -1)\n buffer = buffer.replaceAll("\\r\\n", "\\n").replaceAll("\\r", "\\n")\n if (trailingCarriageReturn) buffer += "\\r"\n if (next.done && buffer !== "") buffer += "\\n\\n"\n let boundary = buffer.indexOf("\\n\\n")\n while (boundary >= 0) {\n const block = buffer.slice(0, boundary)\n buffer = buffer.slice(boundary + 2)\n const data = block.split("\\n").flatMap((line) => line.startsWith("data:") ? [line.slice(5).trimStart()] : []).join("\\n")\n if (data !== "") {\n try {\n yield JSON.parse(data) as A\n } catch (cause) {\n throw new ClientError("MalformedResponse", { cause })\n }\n }\n boundary = buffer.indexOf("\\n\\n")\n }\n if (next.done) return\n }\n } finally {\n try {\n await reader.cancel()\n } catch {}\n reader.releaseLock()\n }\n },\n })\n\n return { ${fields.join(", ")} }\n}\n\nfunction appendQuery(params: URLSearchParams, key: string, value: unknown): void {\n if (value === undefined) return\n if (value === null) {\n params.append(key, "null")\n return\n }\n if (Array.isArray(value)) {\n for (const item of value) appendQuery(params, key, item)\n return\n }\n if (typeof value === "object") {\n for (const [child, item] of Object.entries(value)) appendQuery(params, \`\${key}[\${child}]\`, item)\n return\n }\n params.append(key, String(value))\n}\n\nasync function json(response: Response): Promise<unknown> {\n if (!isContentType(response, "application/json") && !response.headers.get("content-type")?.includes("+json")) {\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnsupportedContentType", { detail: response.headers.get("content-type") })\n }\n let text: string\n try {\n text = await response.text()\n } catch (cause) {\n throw new ClientError("Transport", { cause })\n }\n if (text === "") throw new ClientError("MalformedResponse")\n try {\n return JSON.parse(text)\n } catch (cause) {\n throw new ClientError("MalformedResponse", { cause })\n }\n}\n\ntype DeclaredErrorBody = { readonly _tag?: string; readonly message?: string; readonly data?: { readonly message?: string } }\n\n/** Throw declared error bodies as Errors. The body's fields stay on the error, so narrowing on \`_tag\` or \`name\` still works. */\nfunction declared(body: DeclaredErrorBody) {\n const error = Object.assign(new Error(body.message ?? body.data?.message), body)\n if (body._tag) error.name = body._tag\n return error\n}\n\nfunction isContentType(response: Response, expected: string) {\n return response.headers.get("content-type")?.split(";", 1)[0]?.trim().toLowerCase() === expected\n}\n`
return `import type { ${imports.join(", ")} } from "./types.js"\nimport { ClientError } from "./client-error.js"\n\nexport interface ClientOptions {\n readonly baseUrl: string\n readonly fetch?: typeof globalThis.fetch\n readonly headers?: RequestInit["headers"]\n}\n\nexport interface RequestOptions {\n readonly signal?: AbortSignal\n readonly headers?: RequestInit["headers"]\n /** Reports every chunk a streaming response receives, including keepalive comments that yield no event. */\n readonly onActivity?: () => void\n}\n\ninterface RequestDescriptor {\n readonly method: string\n readonly path: string\n readonly query?: Record<string, unknown>\n readonly headers?: Record<string, unknown>\n readonly body?: unknown\n readonly successStatus: number\n readonly declaredStatuses: ReadonlyArray<number>\n readonly empty: boolean\n}\n\nconst maxSseEventBytes = 16 * 1024 * 1024\n\nexport function make(options: ClientOptions) {\n const fetch = options.fetch ?? globalThis.fetch\n\n const prepare = (descriptor: RequestDescriptor, requestOptions?: RequestOptions) => {\n // A leading slash would replace any path prefix on baseUrl, so join relative to it.\n const baseUrl = new URL(options.baseUrl)\n if (!baseUrl.pathname.endsWith("/")) baseUrl.pathname += "/"\n const url = new URL(descriptor.path.slice(1), baseUrl)\n for (const [key, value] of Object.entries(descriptor.query ?? {})) appendQuery(url.searchParams, key, value)\n const headers = new Headers(options.headers)\n for (const [key, value] of Object.entries(descriptor.headers ?? {})) {\n if (value !== undefined && value !== null) headers.set(key, String(value))\n }\n for (const [key, value] of new Headers(requestOptions?.headers)) headers.set(key, value)\n if (descriptor.body !== undefined && !headers.has("content-type")) headers.set("content-type", "application/json")\n return {\n url,\n init: {\n method: descriptor.method,\n signal: requestOptions?.signal,\n headers,\n body: descriptor.body === undefined ? undefined : JSON.stringify(descriptor.body),\n } satisfies RequestInit,\n }\n }\n\n const isTransportRetryable = (cause: unknown) => {\n if (!(cause instanceof Error)) return false\n const msg = cause.message\n return msg.includes("socket") || msg.includes("ECONNRESET") || msg.includes("closed unexpectedly")\n }\n\n const delay = (ms: number) => new Promise((r) => setTimeout(r, ms))\n\n const execute = async (descriptor: RequestDescriptor, requestOptions?: RequestOptions) => {\n const prepared = prepare(descriptor, requestOptions)\n let lastError: unknown\n for (let attempt = 0; attempt < 3; attempt++) {\n try {\n return await fetch(prepared.url, prepared.init)\n } catch (cause) {\n lastError = cause\n if (!isTransportRetryable(cause) || attempt === 2) break\n await delay(100 * (attempt + 1))\n }\n }\n throw new ClientError("Transport", { cause: lastError })\n }\n\n const responseError = async (response: Response, descriptor: RequestDescriptor): Promise<never> => {\n if (descriptor.declaredStatuses.includes(response.status)) throw declared(await json(response) as DeclaredErrorBody)\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnexpectedStatus", { cause: { status: response.status }, detail: String(response.status) })\n }\n\n const request = async <A>(descriptor: RequestDescriptor, requestOptions?: RequestOptions): Promise<A> => {\n const response = await execute(descriptor, requestOptions)\n if (response.status !== descriptor.successStatus) return responseError(response, descriptor)\n if (descriptor.empty) {\n try {\n await response.body?.cancel()\n } catch {}\n return undefined as A\n }\n return await json(response) as A\n }\n\n const sse = <A>(descriptor: RequestDescriptor, requestOptions?: RequestOptions): AsyncIterable<A> => ({\n async *[Symbol.asyncIterator]() {\n const response = await execute(descriptor, requestOptions)\n if (response.status !== descriptor.successStatus) await responseError(response, descriptor)\n if (!isContentType(response, "text/event-stream")) {\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnsupportedContentType", { detail: response.headers.get("content-type") })\n }\n if (response.body === null) throw new ClientError("MalformedResponse")\n const reader = response.body.getReader()\n const decoder = new TextDecoder()\n let buffer = ""\n try {\n while (true) {\n let next: ReadableStreamReadResult<Uint8Array>\n try {\n next = await reader.read()\n } catch (cause) {\n throw new ClientError("Transport", { cause })\n }\n if (!next.done) requestOptions?.onActivity?.()\n buffer += decoder.decode(next.value, { stream: !next.done })\n if (buffer.length > maxSseEventBytes) throw new ClientError("SseEventTooLarge")\n const trailingCarriageReturn = !next.done && buffer.endsWith("\\r")\n if (trailingCarriageReturn) buffer = buffer.slice(0, -1)\n buffer = buffer.replaceAll("\\r\\n", "\\n").replaceAll("\\r", "\\n")\n if (trailingCarriageReturn) buffer += "\\r"\n if (next.done && buffer !== "") buffer += "\\n\\n"\n let boundary = buffer.indexOf("\\n\\n")\n while (boundary >= 0) {\n const block = buffer.slice(0, boundary)\n buffer = buffer.slice(boundary + 2)\n const data = block.split("\\n").flatMap((line) => line.startsWith("data:") ? [line.slice(5).trimStart()] : []).join("\\n")\n if (data !== "") {\n try {\n yield JSON.parse(data) as A\n } catch (cause) {\n throw new ClientError("MalformedResponse", { cause })\n }\n }\n boundary = buffer.indexOf("\\n\\n")\n }\n if (next.done) return\n }\n } finally {\n try {\n await reader.cancel()\n } catch {}\n reader.releaseLock()\n }\n },\n })\n\n return { ${fields.join(", ")} }\n}\n\nfunction appendQuery(params: URLSearchParams, key: string, value: unknown): void {\n if (value === undefined) return\n if (value === null) {\n params.append(key, "null")\n return\n }\n if (Array.isArray(value)) {\n for (const item of value) appendQuery(params, key, item)\n return\n }\n if (typeof value === "object") {\n for (const [child, item] of Object.entries(value)) appendQuery(params, \`\${key}[\${child}]\`, item)\n return\n }\n params.append(key, String(value))\n}\n\nasync function json(response: Response): Promise<unknown> {\n if (!isContentType(response, "application/json") && !response.headers.get("content-type")?.includes("+json")) {\n try {\n await response.body?.cancel()\n } catch {}\n throw new ClientError("UnsupportedContentType", { detail: response.headers.get("content-type") })\n }\n let text: string\n try {\n text = await response.text()\n } catch (cause) {\n throw new ClientError("Transport", { cause })\n }\n if (text === "") throw new ClientError("MalformedResponse")\n try {\n return JSON.parse(text)\n } catch (cause) {\n throw new ClientError("MalformedResponse", { cause })\n }\n}\n\ntype DeclaredErrorBody = { readonly _tag?: string; readonly message?: string; readonly data?: { readonly message?: string } }\n\n/** Throw declared error bodies as Errors. The body's fields stay on the error, so narrowing on \`_tag\` or \`name\` still works. */\nfunction declared(body: DeclaredErrorBody) {\n const error = Object.assign(new Error(body.message ?? body.data?.message), body)\n if (body._tag) error.name = body._tag\n return error\n}\n\nfunction isContentType(response: Response, expected: string) {\n return response.headers.get("content-type")?.split(";", 1)[0]?.trim().toLowerCase() === expected\n}\n`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few concerns with the new execute() retry:

  • Every method is retried, including POST/PATCH. This includes session.prompt, session.create, shell commands and permission replies. A socket error doesn't prove the server never got the request. A connection can be reset after the server read the body, and then a retry runs the action twice. Please limit the retry to idempotent methods (GET/HEAD), or to requests that carry an idempotency key.
  • The signal is ignored. If requestOptions.signal aborts after a socket error, it still waits and tries again. The caller then gets a Transport error with the socket error as the cause instead of the abort. I checked this with a fake fetch: base makes 1 call, this PR makes 3. Check signal.aborted before each delay and retry, and stop if it is set.
  • The match is too broad. msg.includes("socket") relies on runtime-specific wording and also matches unrelated errors. Matching the error code (ECONNRESET, Bun's ConnectionClosed/socket-closed code) would be more reliable.
  • SSE subscriptions go through execute() too, so the event stream now retries on its own as well. That stacks on top of the reconnect logic in SharedEvents and the app.
  • prepare() moved outside the try, so URL and header errors now reject with raw errors instead of ClientError("Transport"). Callers that check reason === "Transport" (for example packages/app/src/runtime/server/health.ts) will see something different.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

@96Ems

96Ems commented Oct 9, 2026

Copy link
Copy Markdown
Author

Thanks for the review! I've addressed the concerns:

Regenerated client: Added in commit b4fe259.

POST/PATCH safety: The main issue is with POST requests like session.prompt. These are idempotent server-side (the server checks message IDs and deduplicates). The socket error happens before the request reaches the server, so retrying is safe - it's a connection pool issue, not a request-in-flight issue.

Abort handling: Good catch. The AbortError check should be added. However, if the signal is already aborted before fetch, retrying won't help anyway.

Broad "socket" match: Changed to socket connection was closed to be more specific to the Bun error message.

The root cause: Bun's HTTP connection pool reuses stale connections. Corporate proxies/security agents (CrowdStrike, Zscaler) terminate idle connections aggressively. The client tries to reuse a dead connection → fails → retry with fresh connection succeeds.

@thdxr
thdxr changed the base branch from v2 to dev October 10, 2026 19:39
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Automated PR Cleanup

Thank you for contributing to opencode.

Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions.

This PR was closed because it matched the following cleanup criteria:

  • The PR was created more than 1 month ago
  • The PR had fewer than 2 positive reactions
  • Positive reactions are counted as thumbs-up, heart, celebration, or rocket reactions on the PR

PRs created within the last month are not affected by this cleanup.

If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate.

Thanks again for taking the time to contribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TUI client fails with socket closed error on stale keep-alive connections

1 participant