Skip to content

Session permanently frozen: drain() failures are logged but never surfaced to the client (no session.error event) #49740

Closed as not planned

Description

@TVcasualty

Summary

SessionRunner.drain() failures are caught server-side and logged (Failed to drain Session <cause>), but the failure is never surfaced to the client through any session event — no session.error, no durable Step.Failed write, nothing a TUI or downstream consumer can react to. A session that hits this path is left permanently open: no assistant turn ever completes, no error is shown, and nothing about the session's own state indicates it needs attention. The only fix from the user side is to notice the freeze out-of-band (watching the server log) and manually send a new prompt/"continue" to the same session.

This looks closely related to #34839 ("V2: pre-runner errors are not surfaced as assistant message failures"), which was reported against a single error family (SessionRunnerModel.ModelUnavailableError) at pre-stream/model-resolution time and is tracked as likely addressed by #35272 ("finalize session event lifecycle", merged 2026-07-06, bf01264). Filing this separately because on v2.0.8 (which is far ahead of bf01264, ahead_by: 3256 commits per the compare API — the fix is present) the same log-only, never-surfaced behavior still reproduces, and across a noticeably wider set of causes than the original report covered.

Evidence — a real instance's log, one server, ~4 days

Scanning one long-running server's log turned up 54 occurrences of "Failed to drain Session", breaking down by cause:

Cause Count
AI.Error (various — see below) 28
Session.StepFailedError: Compaction summary reached the output token limit 16
SessionRunnerModel.ModelUnavailableError 1
(uncategorized/other AI.Error subtypes not itemized above) 9

The AI.Error family alone spans at least these distinct underlying causes, all reaching the identical dead-end:

  • litellm.Timeout: BedrockException: Timeout Error ... (connection timeout to the upstream provider) — 10 occurrences
  • litellm.BadRequestError: BedrockException - {"message":"...does not support assistant message prefill..."} — 7 occurrences
  • API key is invalid. — 4 occurrences
  • Provider request failed with HTTP 403 — 3 occurrences
  • ConnectionRefused: Unable to connect. Is the computer able to access the url? — 2 occurrences
  • Provider request failed with HTTP 502 — 1 occurrence
  • ECONNRESET: The socket connection was closed unexpectedly. — 1 occurrence

Every one of these is caught in SessionRunner.drain() / SessionStep.attempt() (per the stack traces in the log), logged with level=ERROR, and then dropped — the session simply stops advancing, with no user-visible signal that anything went wrong. Session.StepFailedError: Compaction summary reached the output token limit is a second recurring cause with the exact same dead-end, and it isn't a provider/auth/network problem at all — it's an internal failure during automatic compaction, which makes it clearly in-scope for packages/core/src/session/runner's own control flow rather than an upstream-provider edge case.

This is not one flaky provider or one flaky config — it's the general shape of the bug: any exception that reaches SessionRunner.drain()'s catch path, regardless of origin, is swallowed into a log line instead of becoming a durable session event.

Corroborating evidence: a third-party plugin exists purely to work around this

The community plugin opencode-auto-resume exists specifically to detect a stalled/silently-dead session and re-send a "continue" prompt automatically — i.e., it is a plugin-level workaround for the exact absence of a surfaced failure event described here. That a third party built and maintains a whole plugin around "the server goes silent and someone has to resend the message" is itself evidence this is a real, user-facing gap, independent of my own repro.

(Separately, and not the subject of this issue: as of npm's published 1.1.17 and the unreleased main/1.1.18, this plugin's own dist/index.js still exports the pre-V2 function-style (ctx, options) => Hooks plugin shape rather than the object-style Plugin interface @opencode-ai/plugin's V2 SDK (dist/v2/promise/plugin.d.ts / dist/v2/effect/plugin.d.ts) now requires via define({ id, setup }) / define({ id, effect }). It fails to load against 2.0.8 with PluginModule.LoadError: Plugin must export a default definition with an id and an effect or setup function. (cause: SchemaError(Expected object at ["default"])). Mentioned only because it means the community workaround is currently non-functional against 2.0.x, which raises the stakes on fixing the underlying gap directly rather than relying on that plugin.)

Expected behavior

Whenever SessionRunner.drain()'s step loop fails for any reason — pre-stream setup, mid-stream provider error, or a post-stream failure like a compaction overflow — the runner should still write a durable failure event (a Step.Failed-equivalent, or whatever toSessionError/the SessionError envelope added in #35272 produces) before returning, not just call log.error. Clients/TUIs need something in the session's own event stream to render, and downstream integrations (channel bots, automation, anything polling session state) need it even more, since they have no log tail to watch.

Repro steps (generic — this reproduces from any cause that reaches the drain catch path)

  1. Start a session against a provider/model configuration that will fail mid-drain — easiest reliable trigger is an invalid/expired provider credential, or a model temporarily returning 403/503, or forcing automatic compaction to overflow its own output token limit on a very long session.
  2. Send a prompt and let the runner begin draining.
  3. Watch the server log: Failed to drain Session <cause> appears at level=ERROR.
  4. Check the client / TUI / any subscriber of session events: no error is visible, the session just stops progressing. GET /api/session/{sessionID} and the message list show no failure — the last user message simply has no completed assistant turn.
  5. The only way to unstick it is to notice this out-of-band and send a fresh prompt into the same session (what opencode-auto-resume attempts to automate, and what I ended up scripting myself against the local HTTP API's session.prompt operation as a stopgap).

Version info

  • OpenCode: v2.0.8 (confirmed current — opencode upgrade reports up to date, and I diffed v2.0.7...v2.0.8/v2.0.0...v2.0.8 via the GitHub compare API for anything touching drain/compaction/session-event code; nothing in that range addresses this)
  • Provider: an OpenAI-compatible gateway in front of Amazon Bedrock Anthropic models (litellm), reached through a local proxy — but per the cause breakdown above the failure shape is provider-agnostic; the same dead-end is hit by auth errors, network errors, HTTP-level provider errors, and a purely-internal compaction failure.
  • Confirmed bf01264 (feat(core): finalize session event lifecycle #35272's merge commit) is an ancestor of v2.0.8 (ahead_by: 3256, not diverged), so this is not the pre-fix behavior V2: pre-runner errors are not surfaced as assistant message failures #34839 was about — it's the same class of gap surviving past that fix, for a wider set of causes.

Activity

  1. holny commented on Sep 19, 2026

    @holny
    Contributor

    Confirmed on current v2 head — packages/core/src/session/execution.ts:105 still funnels every drain failure into Effect.logError("Failed to drain Session", cause) with nothing durable written, so any mid-drain failure (the compaction StepFailedError overflow variant included) leaves the session hanging with no event for clients, even after #35272.

    I can take this one. Plan: on the drain catch path, shape the cause through the same toSessionError handling #35272 introduced and write a durable failure event before the runner gives up, with tests covering at least the compaction-overflow and provider-error cases. I'd keep it scoped to the drain catch path since the pre-runner slice (#34839) already has an open PR.

  2. holny commented on Sep 26, 2026

    @holny
    Contributor

    Friendly nudge — this has been open for a week with green checks and no review. The durable failure event only lands when hasInterruptsOnly misclassifies a mixed interrupt+failure cause, so the blast radius is small and contained; #49891 rides on the same branch and stays unaddressed until this lands. Glad to rebase or split the two fixes if that's easier to review.

  3. opencode-agent commented on Oct 18, 2026

    @opencode-agent
    Contributor

    Thanks for the report. Failed runs are already surfaced: the server publishes a session.execution.failed event with the error, and the TUI shows a "Session failed" notification for it. This was already in 2.0.8. The Failed to drain Session log line is extra diagnostics, not the only handling. After a failure the session goes idle, and sending a new prompt continues it. If you're consuming sessions through the API, listen for session.execution.failed.

    Closing. If a failure on a current build leaves no visible error, please open a new issue with that case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions