Skip to content

fix(realtime): preserve reconnect backoff across flaps#102348

Merged
steipete merged 1 commit into
openclaw:mainfrom
HOYALIM:codex/issue-102251-realtime-backoff-cap
Jul 9, 2026
Merged

fix(realtime): preserve reconnect backoff across flaps#102348
steipete merged 1 commit into
openclaw:mainfrom
HOYALIM:codex/issue-102251-realtime-backoff-cap

Conversation

@HOYALIM

@HOYALIM HOYALIM commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Realtime transcription WebSocket sessions reset reconnectAttempts on every raw socket open. A provider that opened, reached ready, and then repeatedly dropped therefore stayed at the base reconnect delay forever and never reached maxReconnectAttempts.

Closes #102251.

Why This Change Was Made

The shared session helper now records the provider-ready timestamp and preserves the retry budget across short ready-then-close flaps. A connection that remains provider-ready for 30 seconds earns a fresh budget when it later disconnects.

The maintainer rewrite is based on current main and keeps the inbound payload cap from #102443. It uses a timestamp checked at disconnect instead of a separate stable-reset timer, avoiding another timer/cleanup state machine and avoiding a new public tuning option.

User Impact

A flapping realtime transcription upstream backs off and stops at the configured retry limit instead of creating an unbounded handshake/audio-reflush loop. Healthy long-lived sessions can still recover from later transient disconnects because a genuinely stable connection refreshes the retry budget.

Evidence

  • Real loopback WebSocket regression: repeated provider-ready flaps stop after the initial connection plus three configured retries.
  • Stable-recovery regression: with a one-retry budget, a connection that remains ready for 30 seconds earns exactly one fresh retry; the next short flap reaches the limit.
  • Blacksmith Testbox tbx_01kx37y4wrj3aja5vab6tdce8y: corepack pnpm test src/realtime-transcription/websocket-session.test.ts — 14/14 passed.
  • Same Testbox: remote pnpm check:changed child gate — exit 0 across core prod/test types, lint, and relevant architecture guards.
  • Fresh Codex autoreview: no findings; patch correct, confidence 0.94.
  • oxfmt and git diff --check: clean.

No live provider endpoint was deliberately flapped; the shared runtime path was exercised against a real local WebSocket server.

Co-authored-by: Ho Lim [email protected]

Copilot AI review requested due to automatic review settings July 9, 2026 01:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@clawsweeper

clawsweeper Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 10:10 PM ET / 02:10 UTC.

Summary
The branch changes the shared realtime transcription websocket session to preserve reconnect attempts across short flaps, reset them only after a stable ready window, clear that timer on close paths, and add a regression test.

PR surface: Source +27, Tests +44. Total +71 across 2 files.

Reproducibility: yes. at source level: current main resets the reconnect counter on raw websocket open, while close handling calls reconnect logic that uses the same counter for the retry cap and delay. I did not execute the regression in this read-only review, but the linked issue and PR both include real local websocket proof.

Review metrics: none identified.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #102251
Summary: This PR is the candidate fix for the canonical realtime transcription websocket reconnect-counter issue; gateway reconnect reports are adjacent reliability work in a different client path.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] The PR head is mergeable but behind current main, so final landing should rely on the normal exact-head CI and merge freshness gate.

Maintainer options:

  1. Decide the mitigation before merge
    Land the shared-helper fix after exact-head freshness checks, keeping provider-specific APIs and config unchanged.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No automated repair is needed; the remaining action is normal maintainer review and exact-head landing of this open implementation PR.

Security
Cleared: The diff only changes reconnect timer state in the shared websocket helper and a colocated test; it adds no dependency, workflow, secret, permission, package-resolution, or code-execution surface.

Review details

Best possible solution:

Land the shared-helper fix after exact-head freshness checks, keeping provider-specific APIs and config unchanged.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main resets the reconnect counter on raw websocket open, while close handling calls reconnect logic that uses the same counter for the retry cap and delay. I did not execute the regression in this read-only review, but the linked issue and PR both include real local websocket proof.

Is this the best way to solve the issue?

Yes. The shared websocket session helper is the best layer because all affected bundled realtime transcription providers route through it; provider-specific fixes or a new config knob would duplicate the invariant.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0307deacfac7.

Label changes

Label justifications:

  • P2: This is a normal-priority reliability fix for shared realtime transcription reconnect behavior with clear source evidence and limited provider-family blast radius.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body reports after-fix real local websocket behavior, the exact observed reconnect-limit result, and validation commands for this non-visual runtime fix.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body reports after-fix real local websocket behavior, the exact observed reconnect-limit result, and validation commands for this non-visual runtime fix.
Evidence reviewed

PR surface:

Source +27, Tests +44. Total +71 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 28 1 +27
Tests 1 44 0 +44
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 72 1 +71

What I checked:

Likely related people:

  • vincentkoc: Current-main blame for the websocket open handler and reset line points to grafted commit 5a5b4c5, and live PR metadata for refactor(slack): localize internal declarations #102044 ties that recent broad commit to vincentkoc. (role: recent area contributor; confidence: medium; commits: 5a5b4c5b105f; files: src/realtime-transcription/websocket-session.ts, src/realtime-transcription/websocket-session.test.ts)
  • Peter Steinberger: The available feature-history commit 0e7bcf7 created the shared realtime transcription websocket session, its tests, and refactored bundled providers onto it, though the current main ancestry is grafted from a later snapshot. (role: introduced shared helper in available history; confidence: medium; commits: 0e7bcf7588d2; files: src/realtime-transcription/websocket-session.ts, src/realtime-transcription/websocket-session.test.ts, extensions/deepgram/realtime-transcription-provider.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-07-09T02:02:37.256Z sha c37203f :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 9, 2026
@steipete steipete self-assigned this Jul 9, 2026
Reset reconnect attempts only after a provider-ready connection stays stable, and cover both flap exhaustion and stable recovery.

Co-authored-by: Ho Lim <[email protected]>
@steipete
steipete force-pushed the codex/issue-102251-realtime-backoff-cap branch from c37203f to 6f87b40 Compare July 9, 2026 10:57
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Maintainer rewrite is ready to land at 6f87b40d3f40e39f3a0061939d3976d7f3e38792.

What changed:

Proof:

  • Blacksmith Testbox tbx_01kx37y4wrj3aja5vab6tdce8y: corepack pnpm test src/realtime-transcription/websocket-session.test.ts — 14/14 passed.
  • Same Testbox: remote pnpm check:changed child gate — exit 0.
  • Fresh Codex autoreview — no findings; patch correct, confidence 0.94.
  • oxfmt and git diff --check — clean.
  • Exact-head CI run 29013284308 — green.

Known gap: no external provider endpoint was deliberately flapped; the shared production session path was exercised against a real loopback WebSocket server.

@steipete
steipete merged commit 99d82a2 into openclaw:main Jul 9, 2026
99 of 103 checks passed
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 10, 2026
Reset reconnect attempts only after a provider-ready connection stays stable, and cover both flap exhaustion and stable recovery.

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

realtime-transcription: reconnect give-up cap and exponential backoff never trigger on a flapping-after-ready websocket

3 participants