Repository navigation
feat(ai): retry transient failures on queued generation reads - #51635
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the change
A single 429, 5xx, or network blip while polling a queued generation (Veo, Runway, fal, BFL, AssemblyAI, …) failed
await()even though the provider job kept running and billing, so this retries those read-only calls within the caller'spoll.timeout.Special things to note
failedgeneration also decodes toProviderInternal(HTTP 200), and re-reading it would spin until the deadline.start(submit, which can bill twice) andcancelnever retry. Inline and stream routes are unchanged.isRetryablemoves unchanged frompackages/core/src/session/runner/retry.tsto@opencode/ai(provider-error.ts), andSessionRunnerRetryre-exports it so every Core caller stays the same.resultEvents(behindVideo.stream/ queuedImage.stream) used to fetch the result with no deadline. That fetch is now bounded by the samepoll.timeout, so retries cannot run past it. The policy is internal: there is no new option.Change outline
The retry policy (
src/route/media.ts). A single blip recovers in about 1s, which is less than the default 5s poll interval. Each gap is capped at 30s, close to how often polls already run. The 8-retry cap (about 2 minutes) bounds a directresult()orresume, which has no deadline of its own.Where it applies inside
MediaRoute.queued:generationRoute(token, http, execute) poll(operation) transport.call("GET", path) + .retry({ schedule: READ_RETRY, while: isRetryable }) // RateLimit, ProviderInternal, Transport, … operation.decode(response) // not retried status = poll(protocol.status) result = poll(protocol.result) cancel = transport.call(cancel.method, …) // not retried start → transport.submit(…) // not retriedDeadlines in
src/generation.ts:awaitandeventsalready interrupt retries atpoll.timeout. The fix is the result fetch in the stream path:Files:
Tests: 503 status and 503 result, then success · 429
retry-after: 10(no poll at 9s, polls at 10s) · 400 not retried · persistent 503 ends withTimeoutatpoll.timeout· streamed result retries end withTimeout· submit 503 not retried · cancel 503 not retried. The existing Veo and Runway tests where a terminalfailedgeneration surfaces asProviderInternalstill pass without advancing the clock, which confirms decoded failures are not retried.