Skip to content

Commit 62f0222

Browse files
authored
fix(codex): promote a healthy detour instead of releasing it, and honour Retry-After (#4546) (#4616)
* fix(codex): promote a healthy detour instead of releasing it, and honour Retry-After (#4546) Refs #4546. PRD R07. When a transient hold outlived its window, routing deleted the whole affinity entry -- including the detour account that had actually been serving the thread -- and re-picked cold. The timer expiring restores the right to re-decide; it is not itself a recovery, and treating it as one threw away the single piece of evidence the request had. A still-healthy detour is now promoted to the binding instead, with the move recorded as rebound/transient_hold_expired so the reason is visible. A detour that has itself gone unhealthy still falls through to the cold path. Retry-After is a lower bound on the transient path. The local maximum delay bounds our own exponential backoff and has no business shortening a wait the provider stated: sending early is a request we already know will be refused, which is the storm the header exists to prevent. It is opt-in per caller so the change lands on the transient path first rather than silently lengthening every adapter's backoff, and an honoured wait is ceilinged at one minute so an hour-long Retry-After cannot park a request. * docs(devlog): record the R07 detour-promotion outcome (#4546)
1 parent 9f39e75 commit 62f0222

4 files changed

Lines changed: 106 additions & 2 deletions

File tree

‎devlog/_plan/260914_cost_guard_stabilization/020_backoff_preserves_binding.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,3 +77,22 @@ the generation check and the pinned-account guard in
7777
4. A 429 on the bound account still releases the binding immediately, unchanged.
7878
5. A late transient failure from an account the thread already left does not touch
7979
the current binding.
80+
81+
## R07 outcome: expiry is permission to re-decide, not a recovery
82+
83+
The rule above bounded the hold correctly and then threw away its own evidence. On expiry the
84+
entry was deleted whole -- `transientDetourAccountId` with it -- and the thread re-picked cold,
85+
so an account that had been serving the conversation happily for ten minutes got no more
86+
consideration than any other. A timer running out restores the right to re-decide; it is not
87+
itself a reason to prefer a stranger.
88+
89+
A still-healthy detour is now promoted to the binding, recorded as `rebound` with reason
90+
`transient_hold_expired`. Promotion is refused when the release reason is generation
91+
invalidation or a quota refusal: those are hard invalidations, and a detour that merely looks
92+
healthy must not rescue them.
93+
94+
What this still does not do: a soft-avoided account receives no traffic at all, so the
95+
two-consecutive-success clearing rule can only be met through the "held" fallback, which hands
96+
the failing account back to every pinned thread at once. A half-open probe lease -- one thread
97+
probes, the rest keep detouring -- is the missing piece and needs a lease keyed on the health
98+
domain rather than the quota cooldown domain the existing one uses.

‎src/codex/routing.ts‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2946,6 +2946,32 @@ export function resolveCodexAccountForThreadDetailed(
29462946
// A model-only exclusion does not invalidate the shared task binding. Health,
29472947
// generation, pause, cooldown, and failure evidence still retire it normally.
29482948
if (!modelScopedSelection || !healthyForSharedAffinity) {
2949+
// A hold that outlived its window is not the same as a conversation with nowhere to go.
2950+
// If the account that has actually been serving this thread is still healthy, promote it
2951+
// instead of deleting the entry and re-picking cold: releasing here threw away the one
2952+
// piece of evidence the request had -- that B works -- and handed the thread back to a
2953+
// fresh strategy choice, which is the cold-prefix cost #4546 is about. A timer expiring
2954+
// restores the right to re-decide; it is not itself a recovery.
2955+
const expiredDetour = entry.transientDetourAccountId;
2956+
if (
2957+
isTransientHoldExpired(entry, now)
2958+
&& generationLive
2959+
&& !quotaRefused
2960+
&& expiredDetour !== undefined
2961+
&& expiredDetour !== entry.accountId
2962+
&& isCodexAccountSelectable(config, expiredDetour, now, quotaScope, selectionOptions)
2963+
&& !hasUnrecoveredCodexQuotaRefusal(expiredDetour, quotaScope)
2964+
&& !shouldFailover(config, expiredDetour, now)
2965+
&& !isCodexAccountSoftAvoided(expiredDetour, now)
2966+
) {
2967+
if (!isIndependentCodexQuotaScope(quotaScope)) promoteActiveCodexAccount(config, expiredDetour);
2968+
bindThreadAffinity(threadId, expiredDetour, now, quotaScope);
2969+
return {
2970+
status: "selected",
2971+
accountId: expiredDetour,
2972+
affinity: { move: "rebound", reason: "transient_hold_expired" },
2973+
};
2974+
}
29492975
releaseReason = !generationLive
29502976
? "generation"
29512977
: quotaRefused

‎src/lib/upstream-retry.ts‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,19 @@ export interface RetryBackoffOptions {
133133
baseDelayMs: number;
134134
maxDelayMs: number;
135135
headers?: Headers;
136+
/**
137+
* Treat a provider's `Retry-After` as the earliest legal send rather than something the
138+
* local maximum may shorten. Opt-in per caller so the change lands on the transient path
139+
* first instead of silently lengthening every adapter's backoff.
140+
*/
141+
retryAfterIsLowerBound?: boolean;
142+
/** Hard ceiling for an honoured `Retry-After`, so an hour-long wait cannot park a request. */
143+
retryAfterCeilingMs?: number;
136144
}
137145

146+
/** One minute, matching the same-target 429 ceiling the key-failover path already uses. */
147+
export const RETRY_AFTER_CEILING_MS = 60_000;
148+
138149
export function abortError(signal?: AbortSignal): unknown {
139150
return signal?.reason ?? new DOMException("The operation was aborted", "AbortError");
140151
}
@@ -284,9 +295,20 @@ function retryAfterDelayMs(headers: Headers): number | undefined {
284295

285296
export function retryBackoffDelayMs(attempt: number, opts: RetryBackoffOptions): number {
286297
const retryAfter = opts.headers ? retryAfterDelayMs(opts.headers) : undefined;
287-
if (retryAfter !== undefined) return Math.min(retryAfter, opts.maxDelayMs);
288298
const exp = Math.min(opts.baseDelayMs * (2 ** attempt), opts.maxDelayMs);
289-
return Math.floor(exp * (0.8 + Math.random() * 0.4));
299+
const jittered = Math.floor(exp * (0.8 + Math.random() * 0.4));
300+
if (retryAfter === undefined) return jittered;
301+
if (opts.retryAfterIsLowerBound !== true) {
302+
// Historical behaviour, still the default for every caller that has not opted in.
303+
return Math.min(retryAfter, opts.maxDelayMs);
304+
}
305+
// A provider that names a wait is stating when it will serve again; sending earlier is a
306+
// request we already know will be refused, and refusing it twice is the retry storm the
307+
// header exists to prevent. The local maximum bounds our OWN exponential backoff and has no
308+
// business shortening someone else's instruction. The ceiling is separate: it stops an
309+
// hour-long Retry-After from parking a request forever.
310+
const ceiling = opts.retryAfterCeilingMs ?? RETRY_AFTER_CEILING_MS;
311+
return Math.min(Math.max(retryAfter, jittered), ceiling);
290312
}
291313

292314
export function cancelResponseBodyBestEffort(res: Response): void {
@@ -504,6 +526,7 @@ export async function fetchWithTransientRetry(
504526
baseDelayMs: TRANSIENT_RETRY_BASE_DELAY_MS,
505527
maxDelayMs: TRANSIENT_RETRY_MAX_DELAY_MS,
506528
headers: res.headers,
529+
retryAfterIsLowerBound: true,
507530
});
508531
cancelResponseBodyBestEffort(res);
509532
// Throws on abort (see sleepWithAbort): the rejection propagates, and the body we just

‎tests/lib/upstream-retry.test.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,42 @@ describe("retryBackoffDelayMs", () => {
250250
}
251251
});
252252

253+
test("treats Retry-After as a lower bound when the caller opts in (#4546)", () => {
254+
const headers = new Headers({ "Retry-After": "30" });
255+
// The local maximum bounds our OWN exponential backoff. Shortening a provider's stated
256+
// wait to 5s just sends a request we already know will be refused, which is the storm the
257+
// header exists to prevent.
258+
expect(retryBackoffDelayMs(0, {
259+
baseDelayMs: 250,
260+
maxDelayMs: 5_000,
261+
headers,
262+
retryAfterIsLowerBound: true,
263+
})).toBe(30_000);
264+
});
265+
266+
test("an honoured Retry-After is still ceilinged so it cannot park a request (#4546)", () => {
267+
const headers = new Headers({ "Retry-After": "3600" });
268+
expect(retryBackoffDelayMs(0, {
269+
baseDelayMs: 250,
270+
maxDelayMs: 5_000,
271+
headers,
272+
retryAfterIsLowerBound: true,
273+
retryAfterCeilingMs: 60_000,
274+
})).toBe(60_000);
275+
});
276+
277+
test("opting in never shortens a wait below the local backoff (#4546)", () => {
278+
const headers = new Headers({ "Retry-After": "0" });
279+
// A past or zero Retry-After means "no enforced wait", not "send immediately with no
280+
// backoff at all" -- the count and ratio budgets still apply and so does our own pacing.
281+
expect(retryBackoffDelayMs(0, {
282+
baseDelayMs: 1_000,
283+
maxDelayMs: 5_000,
284+
headers,
285+
retryAfterIsLowerBound: true,
286+
})).toBeGreaterThanOrEqual(800);
287+
});
288+
253289
test("falls back to capped exponential jitter when Retry-After is absent", () => {
254290
const randomSpy = spyOn(Math, "random").mockReturnValue(0);
255291
try {

0 commit comments

Comments
 (0)