Skip to content

gateway: a reply translated from a dying stream no longer reads complete - #78

Merged
yetone merged 1 commit into
yetone:mainfrom
charan-rathore:fix/translate-severed-stream
Sep 25, 2026
Merged

yetone merged 1 commit into
yetone:mainfrom
charan-rathore:fix/translate-severed-stream

Conversation

@charan-rathore

Copy link
Copy Markdown
Contributor

Root cause

Server.translate speaks one provider API to the client while streaming another from the provider, reading the provider's SSE stream with readSSE (internal/gateway/sse.go). readSSE returns the stream's terminal error, but both call sites in translate ignore that return (internal/gateway/gateway.go).

So when a provider's connection dies mid-reply — upstream reset, truncated body — the failure vanishes:

  • A buffered client (stream:false) gets HTTP 200 with whatever partial text had arrived, rendered with finish_reason: "stop" — a fragment indistinguishable from a complete answer.
  • A streaming client gets a synthetic happy ending: a stop chunk, final usage, and [DONE] over a stream the provider never finished.

Repro on current main (f298bb4): an httptest upstream that hijacks the connection, promises 100 more bytes than it sends, delivers one message_start and one text delta, then closes. Buffered: 200 with "content":"hel" and finish_reason:"stop". Streamed: a finish_reason:"stop" chunk, usage, [DONE], and no error anywhere.

Fix

Check readSSE's return in both branches:

  • Buffered: a stream error now fails the request with 502 naming the provider (UP: unexpected EOF) instead of returning the partial answer. A partial answer is not an answer.
  • Streamed: the 200 is already committed, so translate emits the client protocol's own error event — {"error": ...} for Chat, response.failed for Responses, event: error for Anthropic, an error chunk for Gemini, each encoder's existing KError rendering — and skips the normal completion instead of appending a synthetic one. A stream whose provider already signalled an error likewise no longer gets a completion event after it.

Complete streams are untouched: a control test runs a full Anthropic stream through both branches and asserts the same 200, translated body and [DONE] as before.

Tests

  • New internal/gateway/translate_severed_test.go:
    • TestTranslateBufferedSeveredUpstream — fails on main (200 with partial body), passes with the fix (502 naming the provider).
    • TestTranslateStreamSeveredUpstream — fails on main (synthetic stop + [DONE]), passes with the fix (error event, no completion).
    • TestTranslateStreamCompleteUpstream — control: a properly finished stream still translates end to end, streamed and buffered.
  • go vet ./internal/gateway/ clean; go test -race -count=1 ./internal/gateway/ passes; go test ./internal/... passes except internal/gui, which fails to compile in this Linux container for want of GTK4/WebKitGTK 6 development libraries — pre-existing, since the unmodified tree fails the same way, and CI installs GTK with its own tags.

translate ignored readSSE's return, so an upstream that severed mid-reply left a buffered client holding the partial text under finish_reason "stop" and a streaming client with a synthetic stop chunk, final usage and [DONE]; the buffered path now answers 502 and the streamed path emits the client protocol's error event and skips the normal completion, covered by tests that hijack an httptest upstream and close it mid-stream plus a control that a properly finished stream still translates end to end

@yetone yetone left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — reviewed and verified locally.

  • go test ./internal/gateway -count=1 passes; the three new tests fail on main (the severed stream ends with finish_reason: stop + [DONE]) and pass with the change.
  • Checked the side effect of skipping finish() once failed is set, which now also covers an upstream error event: every encoder already writes a terminal error on KError (Anthropic error, Responses response.failed, chat/Gemini error chunk), so previously a failed reply was followed by message_stop / response.completed / [DONE] — contradicting itself. Dropping that is a fix too.
  • A client that hangs up mid-stream also makes readSSE return an error, but the caller already turns a canceled context into 499 "the agent canceled the request", so it isn't counted against the provider.

Merging. (The subscription path in serveSubscription still calls finish() after an error; I'll align it separately.)

@yetone
yetone merged commit fa25a50 into yetone:main Sep 25, 2026
4 checks passed
yetone added a commit that referenced this pull request Sep 25, 2026
…de Code, Cursor, Grok and Devin runs still wrote message_stop / response.completed / [DONE] after the error event, so a client could take the half reply as complete; the same fix #78 made for translated streams
@yetone

yetone commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Released in v0.1.93, together with the matching fix for subscription runs (6da20f5). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants