Skip to content

realtime-transcription: reconnect give-up cap and exponential backoff never trigger on a flapping-after-ready websocket #102251

Description

@yetval

Summary

Realtime transcription websocket sessions never give up on a flapping upstream. The reconnect give-up cap (maxReconnectAttempts, default 5) and the exponential backoff are both silently defeated: a session that reaches ready and then drops re-opens at the base delay forever instead of backing off and stopping after 5 tries. This affects every websocket transcription provider (openai, mistral, xai, elevenlabs, deepgram).

Environment

  • Commit: 534ace4 (origin/main at time of filing)
  • Surface: src/realtime-transcription/websocket-session.ts, shared by all realtime transcription provider plugins

Steps to reproduce

  1. Configure any realtime transcription provider (for example openai) and start a session.
  2. Let the session reach ready (provider handshake completes).
  3. Have the upstream flap: accept the socket, then drop it shortly after ready, repeatedly (a degraded provider edge, load-balancer draining, or network churn does this in production).
  4. Observe the client reconnecting once per base delay (1000 ms for all bundled providers) without bound. The 5-attempt give-up never fires and the backoff never grows.

Expected

After a drop, the session should reconnect with exponential backoff (base, 2x, 4x, 8x, 16x) and, after maxReconnectAttempts consecutive failures, emit the reconnect-limit error and stop. A persistently flapping upstream should not produce an unbounded reconnect stream.

Actual

The counter that the cap and the backoff both read is reset to 0 on every socket open, so each post-open drop restarts the count from zero. The cap check never reaches the limit and the backoff exponent stays at 0, so the delay stays pinned at the base value. The session reconnects roughly once per second indefinitely.

Root cause

src/realtime-transcription/websocket-session.ts. The open handler unconditionally zeroes the same counter the cap and backoff depend on:

this.ws.on("open", () => {
  opened = true;
  this.connected = true;
  this.reconnectAttempts = 0; // <- reset on every open, even a reconnect that opens then drops
  this.captureLocalOpen();
  ...
});

while the give-up and backoff both read that counter in attemptReconnect:

if (this.reconnectAttempts >= this.maxReconnectAttempts) {
  this.emitError(new Error(... "reconnect limit reached"));
  return;
}
this.reconnectAttempts += 1;
const delay = this.reconnectDelayMs * 2 ** (this.reconnectAttempts - 1);

The cap and backoff were meant to bound reconnect attempts. Because the counter resets whenever a socket reaches open (a raw TCP open, before the provider is even ready), they only bound failures that never open at all. Any connection that reaches open and then drops resets the count, so reconnectAttempts never climbs past 1, the delay never grows past the base, and the limit branch is unreachable. A correct fix resets the counter only after the connection has stayed genuinely ready/stable, or bounds total reconnects, rather than on every raw open.

Sibling surfaces

All five bundled websocket transcription providers route through this single file and inherit the defect: openai, mistral, xai, elevenlabs, and deepgram. All ship base delay 1000 ms and cap 5. deepgram uses readyOnOpen so open equals ready; the other four mark ready on a provider handshake message, but the reset is on raw open, so a socket that opens then drops before or after the handshake resets the counter either way. The gateway client has a related but distinct reconnect-limit gap tracked in #45469 (src/gateway/client.ts scheduleReconnect, which has no cap at all); this issue is a different file and a different failure mode (a cap that exists but is nullified), so it is not covered by #45469 or its fix PRs #77556 / #77961.

Cost note (honest scope)

This is primarily a resource and connection-storm reliability bug, not a guaranteed billing bug. Each reconnect re-runs the provider handshake (onOpen sends session.update) and, on the OAuth path, re-mints a fresh ephemeral client secret (a control-plane request), so the storm inflates provider request volume. When a reconnect reaches ready it also re-flushes queued audio (flushQueuedAudio), so audio buffered during the down window is re-sent on each recovery; if a drop lands mid-utterance this can re-submit overlapping audio for transcription. The direct paid exposure is bounded and conditional (audio-token billing on the re-flushed buffer), not a flat per-reconnect charge, so it is framed as a resource storm with a secondary spend exposure rather than a clean money bug.

Real behavior proof

Behavior addressed: realtime transcription reconnect give-up cap and exponential backoff on a flapping-after-ready upstream
Real environment tested: real runtime, createRealtimeTranscriptionWebSocketSession driven against a real local ws WebSocketServer on 127.0.0.1, at commit 534ace4
Exact steps or command run after this patch: stand up a ws server that accepts each connection, sends {"type":"session.updated"} so the provider markReady fires (session reaches ready), then closes the socket to simulate an upstream flap; configure the session with maxReconnectAttempts 5 and reconnectDelayMs 30; drive session.connect and observe reconnect count, inter-connection gaps, and whether the reconnect-limit error fires over a 1500 ms window; then remove the single line that zeroes reconnectAttempts in the open handler and drive the identical scenario
Evidence after fix:

# OBSERVED (buggy, origin/main 534ace4d, reset on every open)
{ "connectCount": 45,
  "gaps": [46,36,35,36,35,34,35,34,35,35,33,33,36,34,34,34,35,34,33,34,33,34,33,34,33,33,33,34,33,34,34,33,34,33,33,34,33,33,34,33,34,33,34,34],
  "hitReconnectLimit": false }
# EXPECTED (identical drive, the open-handler reset removed)
{ "connectCount": 6,
  "gaps": [42,65,126,248,488],
  "hitReconnectLimit": true }

Observed result after fix: on origin/main the flapping-after-ready session re-opens 45 times in the 1500 ms window with gaps pinned near the 30 base value and never emits the reconnect-limit error, proving the give-up cap and exponential backoff are both defeated; with the open-handler reset removed the identical drive stops after exactly 5 reconnects (6 opens total) with gaps growing 30x1, 30x2, 30x4, 30x8, 30x16 and the reconnect-limit error firing.
What was not tested: not exercised against a live provider endpoint (openai/mistral/xai/elevenlabs/deepgram) or the OAuth ephemeral-secret minting path under real flapping; the exact per-provider audio-token re-billing on the re-flush path was not measured against a real bill.

Repro harness

A throwaway harness driving the real session against a real local ws server, plus the captured before/after numbers, is archived at /tmp/openclaw-bug-repros/realtime-transcription-reconnect-cap/ on the reporter box. It uses the smallest input that triggers the defect: accept, mark ready, drop, repeat.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automation

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions