Skip to content
This repository was archived by the owner on Nov 7, 2026. It is now read-only.

fix(opencode): bound stalled provider streams and interrupts - #303

Merged
brentyates-swx merged 6 commits into
swxtchfrom
fm/opencode-302-stall
Oct 20, 2026
Merged

brentyates-swx merged 6 commits into
swxtchfrom
fm/opencode-302-stall

Conversation

@brentyates-swx

@brentyates-swx brentyates-swx commented Oct 19, 2026 •

Copy link
Copy Markdown

Closes #302

Change

wrapSSE retains the existing per-read timeout and also bounds time since non-empty generation progress for OpenAI-compatible SSE. Responses reasoning/text/tool-argument deltas and Chat content/reasoning/refusal/tool-argument/function-call deltas reset the generation deadline. Chat reasoning_details progress includes OpenRouter's non-empty reasoning.text, reasoning.summary, and reasoning.encrypted payloads. Mistral array-valued text and thinking parts count as progress. Transport-only comments and metadata do not.

SSE framing accepts LF, CRLF, CR, and mixed blank-line separators while treating CRLF as one indivisible line ending, including multiline data: events. The generation deadline reuses provider chunkTimeout (300,000 ms by default), preserving configured overrides and false opt-out. A stalled stream aborts the combined provider request, cancels the reader, and fails as ProviderResponseStreamError; the existing raw-read timeout remains unchanged.

The existing SessionPrompt → SessionRunState → Runner cancellation path is exercised through SessionPrompt.cancel; this change does not add a second cancellation controller. The processor regression verifies the classified timeout survives retries and a one-shot persistence failure through the existing #112 publication path.

Live route evidence (2026-10-19)

The incident report records three high-effort gpt-6.1-sol agent attempts that streamed for 2–4 minutes and then froze. It is symptom evidence, not a route trace. The referenced 28,853-byte brief was used read-only in a bounded direct LLM stream through LLM.Service → Provider → @ai-sdk/openai; tools were disabled, so this was not a complete reproduction of the brief-writer workflow.

The probe used Bun 1.4.2, @ai-sdk/openai 3.0.88, model swx-azure/gpt-6.1-sol, reasoning effort high, and the default 300,000 ms header and chunk timeouts. The measured provider implementation was committed as 9eeb67cab8013bf9d4e74031d6d297a216789381.

The request was POST https://swxcodexmodels.openai.azure.com/openai/v1/responses; the response was 200 text/event-stream; charset=utf-8, served by istio-envoy in East US. It completed in 18.642 seconds. The first raw chunk arrived at 1.418 seconds and the last at 18.608 seconds. The body had 66 raw chunks (105,757 bytes), 275 non-empty generation-progress frames (240 reasoning-summary deltas and 35 output-text deltas), and no SSE comment/keepalive frames. First raw/decoded generation progress arrived at 5.802/5.805 seconds, last at 18.604/18.605 seconds, with a 5.649-second maximum progress gap.

For this measured request, the response/SDK path was observed through the configured fetch; candidate B (an unwrapped or SDK-bypassed response) was not observed. Candidate A (keepalives continuing after generation stops) remains LIVE-UNOBSERVED: the probe completed normally and had no keepalive comments. The captain authorized proceeding from the controlled fixture alone; this PR does not claim candidate A was observed.

The isolated two-ESC TUI probe returned HTTP 200 from session.abort and rendered the session interrupted with the spinner gone; that live probe did not observe provider-body closure. The producer-driven SessionPrompt.cancel regression separately observes the provider fetch signal abort and loop/idle settlement.

Regression proof and focused checks

From packages/opencode:

TMPDIR=/tmp/opencode bun test --timeout 85000 test/session/llm.test.ts -t "bounds Foundry Responses|preserves slow Foundry reasoning progress|preserves Mistral array-valued thinking|recognizes Chat Completions progress fields|keeps OpenRouter structured reasoning_details"
# 5 pass, 76 assertions
TMPDIR=/tmp/opencode bun test --timeout 30000 test/provider/header-timeout.test.ts -t "chunkTimeout|headerTimeout"
# 8 pass, 14 assertions
TMPDIR=/tmp/opencode bun test --timeout 65000 test/session/prompt.test.ts -t "SessionPrompt.cancel aborts a progressing provider request and settles the drain|first-provider diagnostic identifies the held actual runner under its unchanged bound"
# 2 pass, 11 assertions
TMPDIR=/tmp/opencode bun test --timeout 110000 test/session/processor-effect.test.ts -t "session.processor retains the stalled Foundry error"
# 1 pass, 14 assertions

The mixed/multiline-CRLF Chat test failed at its completion assertion with the previous splitter, then passed with CRLF-atomic framing. Chat tests cover the progress fields and require empty content, text, summary, and encrypted reasoning payloads to retain the generation timeout. The OpenRouter test exercises the installed @openrouter/ai-sdk-provider through LLM.Service; the old classifier failed before recognizing reasoning.encrypted, and the updated test passes for text, encrypted, and summary deltas. The Mistral SDK-path regression verifies array-valued reasoning and text deltas over an interval longer than chunkTimeout.

Bounded absence proofs: when OpenAI progress recognition was disabled, the stalled LLM regression failed at its internal bound (stalled provider attempt did not fail within its bound). When state.cancel(sessionID) was temporarily replaced with Effect.void, the cancellation regression failed at its internal bound (cancelled prompt did not settle). Both implementations were restored and the focused tests passed green.

bun typecheck passed in packages/opencode and packages/core; the push hook ran bun turbo typecheck successfully in 30/30 packages. The PR-context changeset check passed (ok: #303 is recorded). Selected Prettier checks passed for the LLM regression file, provider config, generated SDK types, and OpenAPI schema. Full-file Prettier still reports pre-existing unrelated formatting suggestions in the existing provider, changeset, processor-test, and prompt-test files; no unrelated formatting was applied. ./packages/sdk/js/script/build.ts, bun dev generate > ../sdk/openapi.json, and bun run generate from packages/client regenerated/verified the public config descriptions.

The previous unit CI run on head 2429de1c83 had one prompt-test setup failure (Agent not found: build) before the provider boundary. After merging the updated base containing #301, the focused first-provider diagnostic and cancellation tests passed. Unit, typecheck, and e2e checks for the current head are running.

Independent review

Codex's actionable findings on multiline CRLF framing and OpenRouter reasoning_details text, summary, and encrypted payloads were fixed with red/green regressions. Its final response said no actionable regressions were identified, but the structured review gate marked it UNVERIFIED because the response was untagged. Gemini 3.1 Pro reviewed the full diff in read-only mode and reported no actionable regressions.

Follow-up and coordination

Issue #308 tracks a sanitized live trace after #302 lands and fm-spp resumes. It determines whether transport-only keepalives continued after progress stopped; it is post-landing verification, not a blocker. Issue #310 tracks broader non-text SSE progress and opaque-reasoning compatibility; duplicate #312 was closed in favor of #310.

PR #301 merged into swxtch as bf66f27de555be40f64c949699104c9d82e08758 and is included in this branch. Issue #302 was unassigned at pickup.

PR #303 remains draft against swxtch. The pushed head is 7359b71cd312e43d436270a42c2dc1e7827bb973 and matches the live PR head; the base is bf66f27de555be40f64c949699104c9d82e08758. GitHub unit, typecheck, and e2e checks are pending on this head. Do not mark ready or merge; firstmate owns review convergence and the final suite gate.

Plan for swxtchio/swx-opencode#302:
- Probe an isolated swx-azure/gpt-6.1-sol request with the incident workload; retain sanitized build, route, timeout, headers, raw-arrival, decoded-progress, and cancellation evidence.
- Trace the observed SDK consumption and confirmed ESC cancellation through the provider fetch, session runner, and processor; do not select a candidate from source shape alone.
- Change only the established liveness or cancellation boundary, preserving small and genuinely progressing turns and the existing timeout contracts.
- Add bounded producer-driven regressions at the provider stream, SessionPrompt.cancel, and failure-finalization owners as required by the changed behavior; prove each changed regression reds when its guard is absent.
- Run focused tests, package typecheck, lint, and crew identity checks, then push this branch and leave the issue-linked PR draft.
@brentyates-swx

Copy link
Copy Markdown
Author

Session ID: ses_eabb87911ffe0yQWvh8D3Ujpim
Working repo: /home/byates/fm-worktrees/fm-llmrouter/opencode-302-stall/swx-opencode

@brentyates-swx

brentyates-swx commented Oct 19, 2026 •

Copy link
Copy Markdown
Author

Round-1 fixes are pushed through 7359b71cd3.

  • abdd8f9bab adds non-empty Mistral array-valued text/thinking detection, Chat Completions progress coverage, a generation-specific timeout assertion, and slower-fixture margins.
  • 7359b71cd3 makes CRLF atomic in SSE frame splitting and counts non-empty OpenRouter reasoning_details text, summary, and encrypted payloads as progress; multiline CRLF and empty-payload regressions cover the boundary.
  • debacd3992 merges current swxtch base bf66f27de5 (fix(opencode): sandbox the SDK replacement fixture #301).

The focused LLM, header-timeout, prompt cancellation/first-provider, and processor retry tests passed; the push hook typechecked 30/30 packages. Current GitHub checks are running. Codex's final response found no actionable regressions but its structured gate was unverified due to untagged output; Gemini reviewed the supplied diff and found none. Candidate A remains LIVE-UNOBSERVED, with #308 tracking post-landing verification. PR remains draft.

@brentyates-swx

Copy link
Copy Markdown
Author

done: ready for review in branch fm/opencode-302-stall

@brentyates-swx

Copy link
Copy Markdown
Author

done: ready for review in branch fm/opencode-302-stall

@brentyates-swx

Copy link
Copy Markdown
Author

done: ready for review in branch fm/opencode-302-stall

@brentyates-swx

Copy link
Copy Markdown
Author

Integration coordination with auto-compaction PR #313: please merge #303 before #313. We ran a real three-way merge test between the current heads (7359b71cd312e43d436270a42c2dc1e7827bb973 and cddd8db7cb2deea144718a48f3c296eb30fad5f3); it reports one content conflict in CHANGESET.md, while packages/opencode/test/session/prompt.test.ts and generated SDK types auto-merge. After #303 lands, #313 will merge the updated swxtch base and resolve the changeset while preserving both entries. Both PRs remain draft.

@brentyates-swx

Copy link
Copy Markdown
Author

Updated integration-order evidence for auto-compaction PR #313: the current head is dbd73f1eb8901f31f09b3e6565bb370fb1857838. The three-way merge test against #303 head 7359b71cd312e43d436270a42c2dc1e7827bb973 still reports a content conflict only in CHANGESET.md; prompt tests and SDK generated types auto-merge. Please merge #303 before #313; #313 will merge updated swxtch and preserve both changeset entries. Both PRs remain draft.

@brentyates-swx
brentyates-swx marked this pull request as ready for review October 20, 2026 04:52
@brentyates-swx

Copy link
Copy Markdown
Author

done: ready for review in branch fm/opencode-302-stall

@brentyates-swx

Copy link
Copy Markdown
Author

Updated integration-order evidence after auto-compaction head 4dccc3e263c11b0649a1e7a7309b6f25d72193a4: the three-way merge test against #303 head 7359b71cd312e43d436270a42c2dc1e7827bb973 still has only the CHANGESET.md content conflict; prompt.test.ts and generated SDK types auto-merge. Please merge #303 before #313; #313 will merge updated swxtch and preserve both changeset entries.

@brentyates-swx

Copy link
Copy Markdown
Author

Updated merge-order note for auto-compaction head 4dccc3e263c11b0649a1e7a7309b6f25d72193a4: a three-way merge still conflicts only in CHANGESET.md; packages/opencode/test/session/prompt.test.ts and generated SDK types auto-merge. Merge #303 before #313, then #313 will merge updated swxtch and retain both changeset entries.

@brentyates-swx
brentyates-swx merged commit 2402409 into swxtch Oct 20, 2026
3 checks passed
@brentyates-swx
brentyates-swx deleted the fm/opencode-302-stall branch October 20, 2026 12:37
@brentyates-swx

Copy link
Copy Markdown
Author

Integration complete: #303 merged into swxtch as 24024090d2; auto-compaction PR #313 merged that base on its branch in commit 8a87a6d756. The branch now carries both changeset entries and passed the focused post-merge prompt/LLM/provider tests; fresh PR #313 checks are running.

brentyates-swx added a commit that referenced this pull request Nov 6, 2026
* docs: plan transport and lifecycle backport

Implement the assigned transport and lifecycle behavior in existing fork owners.

- Pick each donor in its own -x commit; omit shell-failure UI and absent CLI/MCP payload.
- Disable Bun’s independent fetch deadline only for long network clients and native provider requests; preserve caller signals, domain deadlines, and the #303 watchdog.
- Track detached POSIX groups only while cleanup is owed, preserving #380 escalation and release of intentional descendants; route terminal SIGHUP through the TUI lifecycle.
- Add real-server and child-process regressions with removal proofs, then record donor coverage and focused checks in CHANGESET.md.

* fix(tui): keep long-pending requests past Bun's fetch deadline (anomalyco#55149)

Give the TUI client timeout: false, share the no-timeout fetch helper across run, mini and the TUI via @opencode/util/no-timeout-fetch, and surface existing-session shell failures with a toast and composer restore.

Closes anomalyco#54974

Co-authored-by: Finn763 <[email protected]>
(cherry picked from commit 52a10f5)

* fix(core): disable Bun's fetch idle timeout for provider requests (anomalyco#55007)

(cherry picked from commit 460c626)

* fix(util): kill spawned process groups when opencode exits (anomalyco#59411)

(cherry picked from commit e2bc79d)

* fix(core): type Bun fetch timeout overrides

* test(opencode): verify mini attach timeout wiring

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(core): verify abrupt owner process-group cleanup

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(tui): verify SDK long-call fetch policy

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* fix(tui): preserve nohup SIGHUP behavior

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(core): verify native fetch timeout override

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(core): prove native requests outlive default idle timeout

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(opencode): verify attached runs survive Bun idle timeout

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* docs: record transport and lifecycle backport

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* docs: clarify backport regression ownership

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(opencode): allow Bun 1.4.4 server startup window

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test: shorten idle-timeout acceptance tests

Run the native and attached held-response witnesses in bounded Bun 1.4.4 child processes with BUN_CONFIG_HTTP_IDLE_TIMEOUT=2 so they stay out of the multi-minute range.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(tui): prove SDK long calls survive Bun idle expiry

Exercise SDKProvider's external SDK and SSE paths with a held event stream, a real timeout control, and a completed event after the injected short idle expiry.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(opencode): prove mini attach survives Bun idle expiry

Hold the real mini attach session response in a Bun 1.4.4 child process, observe an unmodified control fetch time out first, then release the expected 404 response.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* test(tui): make SIGHUP fixture reports truthful

Keep the nohup fixture scoped to listener policy and process survival, and add a non-TTY non-nohup default-disposition witness while retaining lifecycle assertions only from the real TUI producer.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* fix(tui): clean Worker-owned process groups on exit

Acknowledge Worker group ownership and release events, kill and await remaining groups before the shutdown reply, and use the recorded PGID ledger as the bounded TUI-owner backstop when shutdown is unconfirmed.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* docs: finalize transport exit review coverage

Record the acknowledged TUI Worker group lifecycle, truthful SIGHUP behavior, and short Bun 1.4.4 behavioral witnesses in the backport ledger.

Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5

* fix(tui): centralize Worker process-group ownership

* docs: plan transport exit review fixes

1. Make worker and parent ownership ledgers converge on each local acquire, release, and cleanup decision even when acknowledgement is rejected or delayed; validate recorded identities before parent-side signalling.
2. Add rejected/delayed acknowledgement regressions for acquire, release, cleanup announcement, and release/reacquire identity fallback.
3. Harden fixture cleanup and deadlines; add Linux-only guards; correct CHANGESET ownership and SIGHUP claims; remove only test hooks no longer needed.
4. Verify focused changed cases and then each importing test file once under checksum-verified Bun 1.4.4; commit without pushing.

* fix(core): reconcile process-group acknowledgement failures

Reconcile Worker and parent ownership ledgers on local acquire, release, and cleanup facts, independent of rejected or delayed acknowledgements. Validate recorded member identities before signalling and preserve released member identities across reacquisition.

Bound identity-owned process fixture cleanup, gate procfs child fixtures to Linux, retain portable SDK cancellation coverage, and correct the #410 SIGHUP and ownership ledger.

Claude-Session: unknown

* test(opencode): gate shell output on spawn readiness

Wait for the process spawn acknowledgement before the ShellTool first-output deadline and reap the child on readiness failure. Tighten process-group fixture types for package typecheck.

Claude-Session: unknown

* fix(core): restore cross-platform process group cleanup

Claude-Session: unavailable (OpenCode session)

* fix(core): unify process-group ownership decisions

Claude-Session: unavailable (OpenCode session)

* fix(core): publish process-group ownership updates

Claude-Session: unavailable (OpenCode session)

* test(opencode): cover cancellation-overlap ownership

Claude-Session: unavailable (OpenCode session)

* test(core): wait for process-group cleanup

Claude-Session: unavailable (OpenCode session)

* test(opencode): harden process-group test results

Claude-Session: unavailable (OpenCode session)

* test(opencode): validate cancellation result shape

Claude-Session: unavailable (OpenCode session)

* test(opencode): type fixture results as unknown

Claude-Session: unavailable (OpenCode session)

---------

Co-authored-by: Finn763 <[email protected]>
Co-authored-by: Finn763 <[email protected]>
Co-authored-by: Aiden Cline <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Foundry long-generation stall: frozen turn survives 20+ min (chunk timeout defeated), ESC cannot abort the in-flight request

1 participant