Repository navigation
fix(opencode): bound stalled provider streams and interrupts - #303
Conversation
Plan for swxtchio/swx-opencode#302: - Probe an isolated swx-azure/gpt-6.1-sol request with the incident workload; retain sanitized build, route, timeout, headers, raw-arrival, decoded-progress, and cancellation evidence. - Trace the observed SDK consumption and confirmed ESC cancellation through the provider fetch, session runner, and processor; do not select a candidate from source shape alone. - Change only the established liveness or cancellation boundary, preserving small and genuinely progressing turns and the existing timeout contracts. - Add bounded producer-driven regressions at the provider stream, SessionPrompt.cancel, and failure-finalization owners as required by the changed behavior; prove each changed regression reds when its guard is absent. - Run focused tests, package typecheck, lint, and crew identity checks, then push this branch and leave the issue-linked PR draft.
|
Session ID: ses_eabb87911ffe0yQWvh8D3Ujpim |
|
Round-1 fixes are pushed through
The focused LLM, header-timeout, prompt cancellation/first-provider, and processor retry tests passed; the push hook typechecked 30/30 packages. Current GitHub checks are running. Codex's final response found no actionable regressions but its structured gate was unverified due to untagged output; Gemini reviewed the supplied diff and found none. Candidate A remains LIVE-UNOBSERVED, with #308 tracking post-landing verification. PR remains draft. |
|
done: ready for review in branch fm/opencode-302-stall |
|
done: ready for review in branch fm/opencode-302-stall |
|
done: ready for review in branch fm/opencode-302-stall |
|
Integration coordination with auto-compaction PR #313: please merge #303 before #313. We ran a real three-way merge test between the current heads ( |
|
Updated integration-order evidence for auto-compaction PR #313: the current head is |
|
done: ready for review in branch fm/opencode-302-stall |
|
Updated integration-order evidence after auto-compaction head |
|
Updated merge-order note for auto-compaction head |
* docs: plan transport and lifecycle backport Implement the assigned transport and lifecycle behavior in existing fork owners. - Pick each donor in its own -x commit; omit shell-failure UI and absent CLI/MCP payload. - Disable Bun’s independent fetch deadline only for long network clients and native provider requests; preserve caller signals, domain deadlines, and the #303 watchdog. - Track detached POSIX groups only while cleanup is owed, preserving #380 escalation and release of intentional descendants; route terminal SIGHUP through the TUI lifecycle. - Add real-server and child-process regressions with removal proofs, then record donor coverage and focused checks in CHANGESET.md. * fix(tui): keep long-pending requests past Bun's fetch deadline (anomalyco#55149) Give the TUI client timeout: false, share the no-timeout fetch helper across run, mini and the TUI via @opencode/util/no-timeout-fetch, and surface existing-session shell failures with a toast and composer restore. Closes anomalyco#54974 Co-authored-by: Finn763 <[email protected]> (cherry picked from commit 52a10f5) * fix(core): disable Bun's fetch idle timeout for provider requests (anomalyco#55007) (cherry picked from commit 460c626) * fix(util): kill spawned process groups when opencode exits (anomalyco#59411) (cherry picked from commit e2bc79d) * fix(core): type Bun fetch timeout overrides * test(opencode): verify mini attach timeout wiring Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(core): verify abrupt owner process-group cleanup Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(tui): verify SDK long-call fetch policy Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * fix(tui): preserve nohup SIGHUP behavior Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(core): verify native fetch timeout override Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(core): prove native requests outlive default idle timeout Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(opencode): verify attached runs survive Bun idle timeout Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * docs: record transport and lifecycle backport Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * docs: clarify backport regression ownership Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(opencode): allow Bun 1.4.4 server startup window Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test: shorten idle-timeout acceptance tests Run the native and attached held-response witnesses in bounded Bun 1.4.4 child processes with BUN_CONFIG_HTTP_IDLE_TIMEOUT=2 so they stay out of the multi-minute range. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(tui): prove SDK long calls survive Bun idle expiry Exercise SDKProvider's external SDK and SSE paths with a held event stream, a real timeout control, and a completed event after the injected short idle expiry. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(opencode): prove mini attach survives Bun idle expiry Hold the real mini attach session response in a Bun 1.4.4 child process, observe an unmodified control fetch time out first, then release the expected 404 response. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * test(tui): make SIGHUP fixture reports truthful Keep the nohup fixture scoped to listener policy and process survival, and add a non-TTY non-nohup default-disposition witness while retaining lifecycle assertions only from the real TUI producer. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * fix(tui): clean Worker-owned process groups on exit Acknowledge Worker group ownership and release events, kill and await remaining groups before the shutdown reply, and use the recorded PGID ledger as the bounded TUI-owner backstop when shutdown is unconfirmed. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * docs: finalize transport exit review coverage Record the acknowledged TUI Worker group lifecycle, truthful SIGHUP behavior, and short Bun 1.4.4 behavioral witnesses in the backport ledger. Claude-Session: c368ed23-1e31-4f2c-b305-b84f21d79cc5 * fix(tui): centralize Worker process-group ownership * docs: plan transport exit review fixes 1. Make worker and parent ownership ledgers converge on each local acquire, release, and cleanup decision even when acknowledgement is rejected or delayed; validate recorded identities before parent-side signalling. 2. Add rejected/delayed acknowledgement regressions for acquire, release, cleanup announcement, and release/reacquire identity fallback. 3. Harden fixture cleanup and deadlines; add Linux-only guards; correct CHANGESET ownership and SIGHUP claims; remove only test hooks no longer needed. 4. Verify focused changed cases and then each importing test file once under checksum-verified Bun 1.4.4; commit without pushing. * fix(core): reconcile process-group acknowledgement failures Reconcile Worker and parent ownership ledgers on local acquire, release, and cleanup facts, independent of rejected or delayed acknowledgements. Validate recorded member identities before signalling and preserve released member identities across reacquisition. Bound identity-owned process fixture cleanup, gate procfs child fixtures to Linux, retain portable SDK cancellation coverage, and correct the #410 SIGHUP and ownership ledger. Claude-Session: unknown * test(opencode): gate shell output on spawn readiness Wait for the process spawn acknowledgement before the ShellTool first-output deadline and reap the child on readiness failure. Tighten process-group fixture types for package typecheck. Claude-Session: unknown * fix(core): restore cross-platform process group cleanup Claude-Session: unavailable (OpenCode session) * fix(core): unify process-group ownership decisions Claude-Session: unavailable (OpenCode session) * fix(core): publish process-group ownership updates Claude-Session: unavailable (OpenCode session) * test(opencode): cover cancellation-overlap ownership Claude-Session: unavailable (OpenCode session) * test(core): wait for process-group cleanup Claude-Session: unavailable (OpenCode session) * test(opencode): harden process-group test results Claude-Session: unavailable (OpenCode session) * test(opencode): validate cancellation result shape Claude-Session: unavailable (OpenCode session) * test(opencode): type fixture results as unknown Claude-Session: unavailable (OpenCode session) --------- Co-authored-by: Finn763 <[email protected]> Co-authored-by: Finn763 <[email protected]> Co-authored-by: Aiden Cline <[email protected]>
Closes #302
Change
wrapSSEretains the existing per-read timeout and also bounds time since non-empty generation progress for OpenAI-compatible SSE. Responses reasoning/text/tool-argument deltas and Chat content/reasoning/refusal/tool-argument/function-call deltas reset the generation deadline. Chatreasoning_detailsprogress includes OpenRouter's non-emptyreasoning.text,reasoning.summary, andreasoning.encryptedpayloads. Mistral array-valued text and thinking parts count as progress. Transport-only comments and metadata do not.SSE framing accepts LF, CRLF, CR, and mixed blank-line separators while treating CRLF as one indivisible line ending, including multiline
data:events. The generation deadline reuses providerchunkTimeout(300,000 ms by default), preserving configured overrides andfalseopt-out. A stalled stream aborts the combined provider request, cancels the reader, and fails asProviderResponseStreamError; the existing raw-read timeout remains unchanged.The existing SessionPrompt → SessionRunState → Runner cancellation path is exercised through
SessionPrompt.cancel; this change does not add a second cancellation controller. The processor regression verifies the classified timeout survives retries and a one-shot persistence failure through the existing #112 publication path.Live route evidence (2026-10-19)
The incident report records three high-effort
gpt-6.1-solagent attempts that streamed for 2–4 minutes and then froze. It is symptom evidence, not a route trace. The referenced 28,853-byte brief was used read-only in a bounded direct LLM stream throughLLM.Service→Provider→@ai-sdk/openai; tools were disabled, so this was not a complete reproduction of the brief-writer workflow.The probe used Bun 1.4.2,
@ai-sdk/openai3.0.88, modelswx-azure/gpt-6.1-sol, reasoning efforthigh, and the default 300,000 ms header and chunk timeouts. The measured provider implementation was committed as9eeb67cab8013bf9d4e74031d6d297a216789381.The request was
POST https://swxcodexmodels.openai.azure.com/openai/v1/responses; the response was200 text/event-stream; charset=utf-8, served byistio-envoyinEast US. It completed in 18.642 seconds. The first raw chunk arrived at 1.418 seconds and the last at 18.608 seconds. The body had 66 raw chunks (105,757 bytes), 275 non-empty generation-progress frames (240 reasoning-summary deltas and 35 output-text deltas), and no SSE comment/keepalive frames. First raw/decoded generation progress arrived at 5.802/5.805 seconds, last at 18.604/18.605 seconds, with a 5.649-second maximum progress gap.For this measured request, the response/SDK path was observed through the configured fetch; candidate B (an unwrapped or SDK-bypassed response) was not observed. Candidate A (keepalives continuing after generation stops) remains LIVE-UNOBSERVED: the probe completed normally and had no keepalive comments. The captain authorized proceeding from the controlled fixture alone; this PR does not claim candidate A was observed.
The isolated two-ESC TUI probe returned HTTP 200 from
session.abortand rendered the session interrupted with the spinner gone; that live probe did not observe provider-body closure. The producer-drivenSessionPrompt.cancelregression separately observes the provider fetch signal abort and loop/idle settlement.Regression proof and focused checks
From
packages/opencode:The mixed/multiline-CRLF Chat test failed at its completion assertion with the previous splitter, then passed with CRLF-atomic framing. Chat tests cover the progress fields and require empty content, text, summary, and encrypted reasoning payloads to retain the generation timeout. The OpenRouter test exercises the installed
@openrouter/ai-sdk-providerthroughLLM.Service; the old classifier failed before recognizingreasoning.encrypted, and the updated test passes for text, encrypted, and summary deltas. The Mistral SDK-path regression verifies array-valued reasoning and text deltas over an interval longer thanchunkTimeout.Bounded absence proofs: when OpenAI progress recognition was disabled, the stalled LLM regression failed at its internal bound (
stalled provider attempt did not fail within its bound). Whenstate.cancel(sessionID)was temporarily replaced withEffect.void, the cancellation regression failed at its internal bound (cancelled prompt did not settle). Both implementations were restored and the focused tests passed green.bun typecheckpassed inpackages/opencodeandpackages/core; the push hook ranbun turbo typechecksuccessfully in 30/30 packages. The PR-context changeset check passed (ok: #303 is recorded). Selected Prettier checks passed for the LLM regression file, provider config, generated SDK types, and OpenAPI schema. Full-file Prettier still reports pre-existing unrelated formatting suggestions in the existing provider, changeset, processor-test, and prompt-test files; no unrelated formatting was applied../packages/sdk/js/script/build.ts,bun dev generate > ../sdk/openapi.json, andbun run generatefrompackages/clientregenerated/verified the public config descriptions.The previous unit CI run on head
2429de1c83had one prompt-test setup failure (Agent not found: build) before the provider boundary. After merging the updated base containing #301, the focused first-provider diagnostic and cancellation tests passed. Unit, typecheck, and e2e checks for the current head are running.Independent review
Codex's actionable findings on multiline CRLF framing and OpenRouter
reasoning_detailstext, summary, and encrypted payloads were fixed with red/green regressions. Its final response said no actionable regressions were identified, but the structured review gate marked it UNVERIFIED because the response was untagged. Gemini 3.1 Pro reviewed the full diff in read-only mode and reported no actionable regressions.Follow-up and coordination
Issue #308 tracks a sanitized live trace after #302 lands and fm-spp resumes. It determines whether transport-only keepalives continued after progress stopped; it is post-landing verification, not a blocker. Issue #310 tracks broader non-text SSE progress and opaque-reasoning compatibility; duplicate #312 was closed in favor of #310.
PR #301 merged into
swxtchasbf66f27de555be40f64c949699104c9d82e08758and is included in this branch. Issue #302 was unassigned at pickup.PR #303 remains draft against
swxtch. The pushed head is7359b71cd312e43d436270a42c2dc1e7827bb973and matches the live PR head; the base isbf66f27de555be40f64c949699104c9d82e08758. GitHub unit, typecheck, and e2e checks are pending on this head. Do not mark ready or merge; firstmate owns review convergence and the final suite gate.