Skip to content

feat(ai): retry transient failures on queued generation reads - #51635

Merged
nexxeln merged 1 commit into
v2from
poll-retry
Sep 27, 2026
Merged

nexxeln merged 1 commit into
v2from
poll-retry

Conversation

@nexxeln

@nexxeln nexxeln commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Why the change

A single 429, 5xx, or network blip while polling a queued generation (Veo, Runway, fal, BFL, AssemblyAI, …) failed await() even though the provider job kept running and billing, so this retries those read-only calls within the caller's poll.timeout.

Special things to note

  • Only the status and result HTTP exchange retries, never the decoded document: a terminal failed generation also decodes to ProviderInternal (HTTP 200), and re-reading it would spin until the deadline. start (submit, which can bill twice) and cancel never retry. Inline and stream routes are unchanged.
  • The retryability rule is not new: isRetryable moves unchanged from packages/core/src/session/runner/retry.ts to @opencode/ai (provider-error.ts), and SessionRunnerRetry re-exports it so every Core caller stays the same.
  • resultEvents (behind Video.stream / queued Image.stream) used to fetch the result with no deadline. That fetch is now bounded by the same poll.timeout, so retries cannot run past it. The policy is internal: there is no new option.

Change outline

The retry policy (src/route/media.ts). A single blip recovers in about 1s, which is less than the default 5s poll interval. Each gap is capped at 30s, close to how often polls already run. The 8-retry cap (about 2 minutes) bounds a direct result() or resume, which has no deadline of its own.

READ_RETRY = max(min(exponential(1s), spaced(30s)), recurs(8))
  |> jittered
  |> delay = min(max(delay, reason.retryAfterMs ?? 0), 30s)   // honor retry-after, capped

Where it applies inside MediaRoute.queued:

 generationRoute(token, http, execute)
   poll(operation)
     transport.call("GET", path)
+      .retry({ schedule: READ_RETRY, while: isRetryable })   // RateLimit, ProviderInternal, Transport, …
     operation.decode(response)                               // not retried
   status = poll(protocol.status)
   result = poll(protocol.result)
   cancel = transport.call(cancel.method, …)                  // not retried
 start → transport.submit(…)                                  // not retried

Deadlines in src/generation.ts: await and events already interrupt retries at poll.timeout. The fix is the result fetch in the stream path:

 resultEvents(generation, expand, options)
+  start = now
   generation.events(options)            // each poll bounded by the deadline (unchanged)
-  ++ generation.result()                // no deadline
+  ++ within(generation.result(), deadline = start + poll.timeout)

Files:

 packages/ai/
 ├── src/provider-error.ts      + isRetryable (moved from core, exported from index)
 ├── src/route/media.ts         + READ_RETRY on status/result GETs
 ├── src/generation.ts          ~ shared `within` deadline helper; result fetch in resultEvents bounded
 ├── docs/media-design.md       + retry policy; interrupting a wait does not cancel the job
 └── README.md                  + same, user-facing
 packages/core/src/session/runner/retry.ts   - isRetryable body, re-exports the ai one
 packages/ai/test/video.test.ts              + 7 tests (Runway, TestClock)

Tests: 503 status and 503 result, then success · 429 retry-after: 10 (no poll at 9s, polls at 10s) · 400 not retried · persistent 503 ends with Timeout at poll.timeout · streamed result retries end with Timeout · submit 503 not retried · cancel 503 not retried. The existing Veo and Runway tests where a terminal failed generation surfaces as ProviderInternal still pass without advancing the clock, which confirms decoded failures are not retried.

@nexxeln
nexxeln merged commit c0d49f1 into v2 Sep 27, 2026
7 of 8 checks passed
@nexxeln
nexxeln deleted the poll-retry branch September 27, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant