Skip to content

Commit 7942348

Browse files
byapparovclaude
andcommitted
fix(retry): bound every retry wait at 5 minutes (#175)
A retryable error of any status whose Retry-After / retry-after-ms asks for longer than USAGE_LIMIT_MAX_WAIT (5 minutes) is no longer retried: 429s still fail as a usage limit (#174), other statuses fail with the provider error. delay() caps a requested wait at the same bound, and the hintless exponential backoff stops at 30 s whether or not the response has headers. RETRY_MAX_DELAY_NO_HEADERS is renamed RETRY_MAX_BACKOFF since it now applies to every response. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_015nwvfUHdM3bgMgD9Zpz7Mj
1 parent 47ede76 commit 7942348

5 files changed

Lines changed: 73 additions & 24 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
### Fixes
66

77
- **Messages written in the same millisecond keep their order** — Reading a session's messages now breaks `time_created` ties by message ID. Before, messages written in the same millisecond came back oldest first. A compaction writes its request, its summary and the continue message in quick succession, so the prompt loop could miss the finished summary and compact again, sending an extra model request. (#200)
8+
- **Retry waits are bounded for every status** — No single wait between retries exceeds 5 minutes. A retryable error of any status whose `Retry-After` / `retry-after-ms` asks for longer is no longer retried: a 429 still fails as `usage_limit` (#174), and any other status (for example a 503 or 529 asking for `retry-after: 18000`, which used to sleep about 5 hours per attempt) fails at once with the provider error, `session_error.reason` from its status (`provider` for a 5xx) and exit 1, with no `retry` event. Exponential backoff without a retry hint now stops at 30 s even when the response has headers, as every AI SDK HTTP error does, so a hintless 429 or 5xx retries for about 3.5 minutes in all instead of about 34. With 10 retries the waits total at most 50 minutes. (closes #175)
89

910
### Security
1011

‎EVENTS.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -177,12 +177,12 @@ Emitted immediately before `session_complete` when the session terminates abnorm
177177
- `message` (string, **required**) — human-readable error message.
178178
- `resetAt` (number, optional) — for `usage_limit`, when the provider said the limit resets, in Unix milliseconds. It comes from `Retry-After` / `retry-after-ms`, or from the reset time in a Z.ai quota message. Omitted when the provider gave no reset time the CLI can read, or one too far away to be a valid date.
179179

180-
**Provider usage limits.** A provider usage or quota limit that will not reset soon fails the run immediately instead of retrying. The bound is 5 minutes: `MAX_RETRY_ATTEMPTS` (10) times the 30-second header-less delay cap. A limit is recognised when:
180+
**Provider usage limits.** A provider usage or quota limit that will not reset soon fails the run immediately instead of retrying. The bound is 5 minutes: `MAX_RETRY_ATTEMPTS` (10) times the 30-second backoff cap. A limit is recognised when:
181181

182182
- an HTTP 429 response has a `Retry-After` or `retry-after-ms` header asking for a single wait longer than 5 minutes, or
183183
- an HTTP 429 response has a known quota-exhausted shape. Currently this is the Z.ai GLM Coding Plan rolling five-hour (`1308`) and weekly or monthly (`1310`) quota codes, or a `Usage limit reached for <n> hour` message. Z.ai gives the reset only in the message, as `reset at YYYY-MM-DD HH:MM:SS` with no time zone. The CLI reads it as UTC+8 (a fixed `+08:00` offset). A quota whose reset is 5 minutes away or less still retries. A quota whose reset is missing or cannot be read fails fast without `resetAt`.
184184

185-
The run emits no `retry` event. The failed turn ends with an `error` status, `session_error` has `reason: "usage_limit"` and `code: "PROVIDER_USAGE_LIMIT"`, and `invocation_complete.status` is `error`. Headless execution exits 1. The message starts with `Provider usage limit reached` and keeps the provider's text, which can name its own reset time. Any other 429 still retries as `rate_limited`: one that asks for a wait of 5 minutes or less, a known quota that resets within 5 minutes, or one with no reset hint and no known quota shape. Those retries keep the existing delays. Each attempt waits for the provider's requested delay, or for exponential backoff when the response gives none, so a run of retries can last longer than 5 minutes in total.
185+
The run emits no `retry` event. The failed turn ends with an `error` status, `session_error` has `reason: "usage_limit"` and `code: "PROVIDER_USAGE_LIMIT"`, and `invocation_complete.status` is `error`. Headless execution exits 1. The message starts with `Provider usage limit reached` and keeps the provider's text, which can name its own reset time. Any other 429 still retries as `rate_limited`: one that asks for a wait of 5 minutes or less, a known quota that resets within 5 minutes, or one with no reset hint and no known quota shape. Those retries follow the bounded delays described under [`retry`](#retry).
186186

187187
```json
188188
{
@@ -198,7 +198,7 @@ The run emits no `retry` event. The failed turn ends with an `error` status, `se
198198

199199
### `retry`
200200

201-
Emitted when a primary-session model attempt fails with a retryable error and the CLI schedules another attempt. This includes a thought-only `unknown` finish with no tool call or non-empty text. It is not emitted after retries are exhausted, for a provider usage limit (see `session_error`), or for an `unknown` finish that includes text or a tool call.
201+
Emitted when a primary-session model attempt fails with a retryable error and the CLI schedules another attempt. This includes a thought-only `unknown` finish with no tool call or non-empty text. It is not emitted after retries are exhausted, for a provider usage limit (see `session_error`), for a provider-requested wait over 5 minutes (see Retry delays below), or for an `unknown` finish that includes text or a tool call.
202202

203203
```json
204204
{
@@ -218,6 +218,8 @@ Emitted when a primary-session model attempt fails with a retryable error and th
218218
- `retry.code` `rate_limited` maps to `session_error.reason` `rate_limit`; `no_finish_reason`, `overloaded`, `server_error`, and `network` are retry-only codes.
219219
- `next` (number, **required**) — scheduled next-attempt time in Unix milliseconds.
220220

221+
**Retry delays.** No single wait between attempts exceeds 5 minutes, whatever the status code or headers. Each retry waits for the provider's `Retry-After` / `retry-after-ms` when it sends one, or else for exponential backoff of 2 s, 4 s, 8 s and 16 s, then 30 s for each later attempt. A retryable error whose provider asks for a wait longer than 5 minutes is not retried: a 429 fails as a provider usage limit (see `session_error`), and any other status (for example a 503 or 529 "overloaded" with `retry-after: 18000`) fails with that provider error, so `session_error.reason` is classified from its status as usual (`provider` for a 5xx). Neither emits a `retry` event. With at most 10 retries, the waits total at most 50 minutes when every response asks for 5 minutes, and 3.5 minutes (210 s) when no response gives a retry hint.
222+
221223
### `message_complete`
222224

223225
Emitted when a primary-session assistant message finishes (one per LLM turn). Child-session assistant messages are
@@ -290,6 +292,7 @@ An `unknown` finish is triaged by delivered output. Nonempty text becomes `stop`
290292
| Other nonempty finish | Existing behavior: end the loop without inferring failure from an unfamiliar reason. |
291293
| Thrown provider error | Existing retry/error handling; unrecoverable failures emit the failure lifecycle. |
292294
| Provider usage limit | Not retried for a 429 that waits over 5 minutes or is a known quota without a sooner reset; `usage_limit` and exit 1. |
295+
| Long provider-requested wait | Not retried for any other status that waits over 5 minutes; the provider error fails the run and exits 1. |
293296
| Cancellation / stream timeout | Existing cancellation and timeout lifecycle; not reclassified as a provider finish error. |
294297

295298
With pinned Google SDK 2.0.54, `IMAGE_SAFETY`, `RECITATION`, `SAFETY`, `BLOCKLIST`, `PROHIBITED_CONTENT`, and `SPII` map to `content-filter`; `MALFORMED_FUNCTION_CALL` maps to `error`. `OTHER` and `FINISH_REASON_UNSPECIFIED` map to `other`, while `LANGUAGE` maps to `unknown`. These last mappings retain the behavior above; they do not prove successful task completion. Consumers can distinguish `other` from `stop` using `message_complete.finish`.

‎packages/cli/src/session/retry.ts‎

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ export namespace SessionRetry {
3131

3232
export const RETRY_INITIAL_DELAY = 2000
3333
export const RETRY_BACKOFF_FACTOR = 2
34-
export const RETRY_MAX_DELAY_NO_HEADERS = 30_000 // 30 seconds
34+
// Exponential backoff, used when the provider gives no retry hint, stops here.
35+
export const RETRY_MAX_BACKOFF = 30_000 // 30 seconds
3536
export const RETRY_MAX_DELAY = 2_147_483_647 // max 32-bit signed integer for setTimeout
3637
export const MAX_RETRY_ATTEMPTS = 10
3738

@@ -78,20 +79,23 @@ export namespace SessionRetry {
7879
}
7980
}
8081

81-
export function delay(attempt: number, error?: MessageV2.APIError) {
82-
if (error) {
83-
const headers = error.data.responseHeaders
84-
if (headers) {
85-
return retryAfter(headers) ?? RETRY_INITIAL_DELAY * Math.pow(RETRY_BACKOFF_FACTOR, attempt - 1)
86-
}
87-
}
82+
function requestedWait(error: MessageV2.APIError | undefined) {
83+
const headers = error?.data.responseHeaders
84+
return headers ? retryAfter(headers) : undefined
85+
}
8886

89-
return Math.min(RETRY_INITIAL_DELAY * Math.pow(RETRY_BACKOFF_FACTOR, attempt - 1), RETRY_MAX_DELAY_NO_HEADERS)
87+
// The wait before the next attempt: the provider's requested wait, at most
88+
// USAGE_LIMIT_MAX_WAIT, or else exponential backoff up to RETRY_MAX_BACKOFF.
89+
export function delay(attempt: number, error?: MessageV2.APIError) {
90+
const wait = requestedWait(error)
91+
if (wait !== undefined) return Math.min(wait, USAGE_LIMIT_MAX_WAIT)
92+
return Math.min(RETRY_INITIAL_DELAY * Math.pow(RETRY_BACKOFF_FACTOR, attempt - 1), RETRY_MAX_BACKOFF)
9093
}
9194

92-
// A 429 asking for a single wait longer than every attempt would spend at
93-
// the header-less delay cap (5 minutes) is a usage limit, not a rate limit.
94-
export const USAGE_LIMIT_MAX_WAIT = MAX_RETRY_ATTEMPTS * RETRY_MAX_DELAY_NO_HEADERS
95+
// The longest single wait a provider can ask for and still be retried: every
96+
// attempt at the backoff cap (5 minutes). Any status asking for longer is not
97+
// retried, and a 429 asking for longer is a usage limit, not a rate limit.
98+
export const USAGE_LIMIT_MAX_WAIT = MAX_RETRY_ATTEMPTS * RETRY_MAX_BACKOFF
9599
export const USAGE_LIMIT_CODE = "PROVIDER_USAGE_LIMIT"
96100

97101
export type UsageLimit = {
@@ -104,7 +108,7 @@ export namespace SessionRetry {
104108
export function usageLimit(error: ReturnType<NamedError["toObject"]>): UsageLimit | undefined {
105109
if (!MessageV2.APIError.isInstance(error)) return undefined
106110
if (error.data.statusCode !== 429) return undefined
107-
const wait = error.data.responseHeaders ? retryAfter(error.data.responseHeaders) : undefined
111+
const wait = requestedWait(error)
108112
if (wait !== undefined && wait > USAGE_LIMIT_MAX_WAIT) return { resetAt: resetTime(wait) }
109113
const quota = zaiUsageLimit(error)
110114
if (!quota) return undefined
@@ -186,6 +190,9 @@ export namespace SessionRetry {
186190
if (usageLimit(error)) return undefined
187191
if (MessageV2.APIError.isInstance(error)) {
188192
if (!error.data.isRetryable) return undefined
193+
// nor will any other error whose provider asks to wait longer than a retry may
194+
const wait = requestedWait(error)
195+
if (wait !== undefined && wait > USAGE_LIMIT_MAX_WAIT) return undefined
189196
return error.data.message.includes("Overloaded") ? "Provider is overloaded" : error.data.message
190197
}
191198

‎packages/cli/test/cli/run-usage-limit.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,3 +145,19 @@ describe("headless provider usage limits (#112)", () => {
145145
expect(result.events.find((event) => event.type === "invocation_complete")?.status).toBe("completed")
146146
}, 20000)
147147
})
148+
149+
describe("headless retry wait bound (#175)", () => {
150+
test("fails fast when an overloaded provider asks for a wait over 5 minutes", async () => {
151+
const result = await run([
152+
{ status: 503, headers: { "retry-after": "18000" }, body: { error: { message: "Overloaded" } } },
153+
])
154+
expect(result.exit, result.output).toBe(1)
155+
expect(result.requests).toBe(1)
156+
expect(result.events.filter((event) => event.type === "retry")).toHaveLength(0)
157+
const errors = result.events.filter((event) => event.type === "session_error")
158+
expect(errors).toHaveLength(1)
159+
expect(errors[0]).toMatchObject({ reason: "provider", code: "503" })
160+
expect(errors[0].message).toContain("Overloaded")
161+
expect(result.events.find((event) => event.type === "invocation_complete")?.status).toBe("error")
162+
}, 20000)
163+
})

‎packages/cli/test/session/retry.test.ts‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -58,12 +58,24 @@ describe("session.retry.delay", () => {
5858
expect(SessionRetry.delay(1, error)).toBe(2000)
5959
})
6060

61-
test("uses retry-after values even when exceeding 10 minutes with headers", () => {
61+
test("caps backoff at 30 seconds when the response has headers but no retry hint", () => {
62+
const error = apiError({ "content-type": "application/json" })
63+
const delays = Array.from({ length: 10 }, (_, index) => SessionRetry.delay(index + 1, error))
64+
expect(delays).toStrictEqual([2000, 4000, 8000, 16000, 30000, 30000, 30000, 30000, 30000, 30000])
65+
})
66+
67+
test("uses retry-after values longer than the backoff cap", () => {
6268
const error = apiError({ "retry-after": "50" })
6369
expect(SessionRetry.delay(1, error)).toBe(50000)
70+
})
6471

65-
const longError = apiError({ "retry-after-ms": "700000" })
66-
expect(SessionRetry.delay(1, longError)).toBe(700000)
72+
test.each([
73+
["retry-after-ms", "700000"],
74+
["retry-after", "18000"],
75+
["retry-after-ms", "Infinity"],
76+
])("never waits longer than 5 minutes for %s: %s", (header, value) => {
77+
const error = apiError({ [header]: value })
78+
expect(SessionRetry.delay(1, error)).toBe(SessionRetry.USAGE_LIMIT_MAX_WAIT)
6779
})
6880

6981
test("sleep caps delay to max 32-bit signed integer to avoid TimeoutOverflowWarning", async () => {
@@ -238,16 +250,26 @@ describe("session.retry.usageLimit", () => {
238250
expect(SessionRetry.retryable(error)).toBe("Rate limit exceeded")
239251
})
240252

241-
test("keeps retrying an overloaded provider that asks for a long wait", () => {
242-
const error = providerError({ statusCode: 503, message: "Overloaded", headers: { "retry-after": "18000" } })
253+
test.each([
254+
[503, { "retry-after": "18000" }],
255+
[529, { "retry-after": "18000" }],
256+
[500, { "retry-after-ms": String(SessionRetry.USAGE_LIMIT_MAX_WAIT + 1) }],
257+
])("fails fast without a usage limit when a %d asks for a wait over 5 minutes", (statusCode, headers) => {
258+
const error = providerError({ statusCode, message: "Overloaded", headers })
259+
expect(error.data.isRetryable).toBe(true)
243260
expect(SessionRetry.usageLimit(error)).toBeUndefined()
261+
expect(SessionRetry.usageLimitError(error)).toBeUndefined()
262+
expect(SessionRetry.retryable(error)).toBeUndefined()
263+
})
264+
265+
test("keeps retrying an overloaded provider that asks for a wait of 5 minutes", () => {
266+
const error = providerError({ statusCode: 503, message: "Overloaded", headers: { "retry-after": "300" } })
244267
expect(SessionRetry.retryable(error)).toBe("Provider is overloaded")
268+
expect(SessionRetry.delay(1, error)).toBe(SessionRetry.USAGE_LIMIT_MAX_WAIT)
245269
})
246270

247271
test("bounds the wait by the existing retry budget", () => {
248-
expect(SessionRetry.USAGE_LIMIT_MAX_WAIT).toBe(
249-
SessionRetry.MAX_RETRY_ATTEMPTS * SessionRetry.RETRY_MAX_DELAY_NO_HEADERS,
250-
)
272+
expect(SessionRetry.USAGE_LIMIT_MAX_WAIT).toBe(SessionRetry.MAX_RETRY_ATTEMPTS * SessionRetry.RETRY_MAX_BACKOFF)
251273
})
252274

253275
test("builds a nonretryable terminal error that names the usage limit", () => {

0 commit comments

Comments
 (0)