Repository navigation
Labels
Description
Context
#174 (for #112) makes a 429 provider usage limit fail fast. To match the scope of #112 ("Changing retry policy for all provider errors" is out of scope), it does not change retry delays for other errors. Review of #174 found two remaining ways a headless run can still wait far longer than the 30-second delay cap suggests.
Problem
- A long
Retry-Afteron a status other than 429 is honoured without a cap.SessionRetry.delay()returns the provider'sRetry-After/retry-after-msas is, andSessionRetry.sleep()clamps it only at 2^31-1 ms (about 24.8 days). Take a retryable 503/529 "overloaded" response withretry-after: 18000: each attempt sleeps about 5 hours, for up to 10 attempts. The test "keeps retrying an overloaded provider that asks for a long wait" inpackages/cli/test/session/retry.test.tspins this behaviour, and it can still be classified as retryable. - When the response has headers, the backoff fallback is uncapped. In
delay(), a response with headers but no retry hint uses2000 * 2^(attempt-1)with no cap. That gives 1024 s at attempt 10 and about 34 minutes over all attempts. TheRETRY_MAX_DELAY_NO_HEADERS(30 s) cap applies only whenresponseHeadersis missing, and HTTP errors from the AI SDK always carry a headers record. So a quota 429 that the fix(session): fail fast on long-reset provider usage limits (#112) #174 heuristics miss retries for about 34 minutes.
Proposed approach
Decide one bounded per-attempt delay policy for all retryable errors. For example:
- cap the header-present backoff fallback at
RETRY_MAX_DELAY_NO_HEADERS, and - treat any
Retry-Afterlonger thanSessionRetry.USAGE_LIMIT_MAX_WAITas terminal (or cap it), whatever the status.
Then keep EVENTS.md in line with the policy. This is a retry-policy change, so it needs a product decision on the bounds.
Acceptance criteria
- No single retry sleep exceeds a documented bound, whatever the status code or headers.
- The total retry duration in the worst case is documented in
EVENTS.md. - Regression tests cover a long
Retry-Afteron 503/529, and the header-present fallback without a hint.
References
- Fix: fail fast on rolling provider usage limits #112, fix(session): fail fast on long-reset provider usage limits (#112) #174 (review round 2 findings: "Long Retry-After on non-429 still parks retries for hours" and "5-minute retry-budget premise breaks when 429 lacks hint")
Activity
byapparov commentedon Oct 17, 2026
Related data point from production telemetry over the last ~4 months, aggregate only. Besides the long-reset quota failures that #174 now fails fast on, a separate cluster of runs ended with
Max retry attempts (10) reached: Rate limit reached for requests. That is a short, request-rate limit, not a quota. In those runs the existing backoff used up all 10 attempts without recovering, so each run spent its full retry budget and still failed. When picking bounds for this issue, consider whether the short-limit backoff schedule (delays and attempt count) actually spans the provider's rate-limit window, or whether those runs should also end earlier with a distinct, actionable reason.fix(retry): treat a non-positive Retry-After as no hint; name RETRY_M…
fix(retry): bound every retry wait at 5 minutes (#175)
fix(retry): treat a non-positive Retry-After as no hint; name RETRY_M…
Merge pull request #217 from aictrl-dev/fix/issue-175-bound-retry-delays