Skip to content

Commit a06a416

Browse files
committed
fix(proxy): extend retryOn429 to all key-auth surfaces (audit round-3 xhigh)
- core: the Responses passthrough wire (openai-responses key-auth gateways, e.g. the built-in DeepSeek preset) now replays 429 on the same key pre-relay, before the forward-pool logic; Anthropic terminal-guard continuations replay before key/account failover - images/loop + web-search/loop: same-target replay before on429 key rotation via a new retryOn429Policy dep (abort-aware, heartbeat seams preserved) - key-failover: the fixed fallback is now capped at maxIntervalMs (a single wait never exceeds the cap); authMode local runtimes are gated out alongside oauth/forward, so the knob matches the documented API-key scope exactly - tests: key-auth openai-responses passthrough e2e, terminal-guard continuation replay, image/web-search same-key replay (rotation stays zero), fallback cap, local-mode gating - docs: coverage and cap wording in devlog 010, structure/04, and all five configuration locale rows
1 parent 2efb887 commit a06a416

16 files changed

Lines changed: 427 additions & 69 deletions

File tree

‎devlog/_plan/260802_429_same_target_retry/010_design.md‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,25 @@ existing multi-key failover runs. Default off → zero behavior change for exist
2828
dropped, never a config-rejecting error; the outer provider schema stays passthrough).
2929
- `src/providers/key-failover.ts`: `rateLimitRetryPolicyFor` (normalize/default) and
3030
`rateLimitRetryDelayMs` (Retry-After seconds/HTTP-date → capped, else `intervalMs`),
31-
reusing the existing `parseRetryAfterMs` cooldown parser. Key-auth providers only: OAuth and
32-
forward credentials are never replayed on the same token.
31+
reusing the existing `parseRetryAfterMs` cooldown parser; the fixed fallback is capped at
32+
`maxIntervalMs` too, so a single wait never exceeds the cap. API-key providers only
33+
(`authMode: "key"`): OAuth/forward credentials are never replayed on the same token, and
34+
local runtimes have no remote key to preserve.
3335
- `src/usage/log.ts`: new `AttemptRecoveryKind` member `"rate-limit-429"`.
3436
- `src/server/responses/core.ts`: in the pre-stream recovery loop, BEFORE the multi-key
3537
failover `while`, wait then `rebuildAndRefetch("rate-limit-429")`. Abort during the wait
3638
cancels the client request (the unread 429 body is released first). The retry budget lives
3739
OUTSIDE the recovery loop, so a 413/401 replay that comes back 429 cannot re-arm a fresh
3840
budget — bounded to `attempts` per request. After attempts are exhausted the existing
39-
failover and error mapping run unchanged. Covers Responses, chat completions, and routed
40-
Claude messages (they all enter `handleResponses`).
41+
failover and error mapping run unchanged. The same wait-and-replay applies to the other
42+
key-auth surfaces that bypass that loop:
43+
- Responses passthrough wire (`openai-responses` key-auth gateways, e.g. the built-in
44+
DeepSeek preset) — pre-relay, before the forward-pool logic;
45+
- image/video bridge and web-search sidecar loops (`src/images/loop.ts`,
46+
`src/web-search/loop.ts`) — before their `on429` key rotation;
47+
- Anthropic terminal-guard continuations — before key/account failover.
48+
Covers Responses, chat completions, and routed Claude messages (they all enter
49+
`handleResponses`).
4150

4251
## Safety
4352

@@ -68,4 +77,9 @@ existing multi-key failover runs. Default off → zero behavior change for exist
6877
- `tests/usage-log.test.ts` — the `rate-limit-429` recovery kind survives persisted usage logs.
6978
- `tests/server-rate-limit-retry-e2e.test.ts` — single-key replay to success, immediate
7079
passthrough without the knob, exhausted attempts surface 429, and retry-before-failover
71-
ordering with a 2-key pool.
80+
ordering with a 2-key pool, plus key-auth `openai-responses` passthrough replaying 429 on
81+
the same key.
82+
- `tests/terminal-guard-server.test.ts` — an Anthropic terminal-guard continuation that 429s
83+
is replayed on the same key before the error surfaces (3 upstream sends).
84+
- `tests/images/loop.test.ts` + `tests/web-search.test.ts` — the bridge loops replay 429 on
85+
the same key before `on429` rotation runs (same-key sends counted, rotations zero).

‎docs-site/src/content/docs/ja/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,7 +194,7 @@ timing side channel を防ぐため定数時間(`timingSafeEqual`)で比較
194194
| `parallelToolCalls?` | `boolean` | 並列ツール呼び出しをオン/オフします。OpenAI Chat はデフォルト on で、chat 以外のアダプターは明示的な `true` でのみサポートを公表します。 |
195195
| `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` で `auto` または `none` だけを受け付けるモデル。強制/指定選択は downgrade します。 |
196196
| `preserveReasoningContentModels?` | `string[]` | 前の assistant `reasoning_content` を chat history に維持すべきモデル。 |
197-
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | key-auth プロバイダーのみ。オプトインの同一ターゲット 429 リトライ: 待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は 600000 で上限)、`respectRetryAfter: true`。 |
197+
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。オプトインの同一ターゲット 429 リトライ: `retryOn429` が無ければ無効で、オブジェクトがあれば `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 |
198198
| `thinkingToggleModels?` | `string[]` | effort 段階の代わりに vendor `thinking.enabled` toggle を使う chat モデル。 |
199199
| `thinkingBudgetModels?` | `string[]` | 整数 `thinking_budget` を使う chat モデル。effort を budget 比率にマッピングします。 |
200200
| `noVisionModels?` | `string[]` | テキスト専用モデル。[ビジョンサイドカー](/ja/guides/sidecars/) が画像を説明します。Ollama の `:size` タグも一致させます。 |

‎docs-site/src/content/docs/ko/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ timing side channel을 막기 위해 상수 시간(`timingSafeEqual`)으로 비
204204
| `parallelToolCalls?` | `boolean` | 병렬 툴 호출을 켜거나 끕니다. OpenAI Chat은 기본 on이며, chat 외 어댑터는 명시적인 `true`에서만 지원을 알립니다. |
205205
| `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice`에서 `auto` 또는 `none`만 받는 모델. 강제/지정 선택은 downgrade합니다. |
206206
| `preserveReasoningContentModels?` | `string[]` | 이전 assistant `reasoning_content`를 chat history에 유지해야 하는 모델. |
207-
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | key-auth 프로바이더 전용. 동일 대상 429 재시도: 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 600000으로 상한), `respectRetryAfter: true`. |
207+
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key 프로바이더 전용(`authMode: "key"`). 동일 대상 429 재시도: `retryOn429`가 없으면 기능이 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 429 시 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다 — 일반 텍스트 턴 복구 루프, Responses passthrough, 이미지/비디오 브리지, web-search 사이드카, 터미널 연속 요청을 모두 포함합니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 `maxIntervalMs`로 상한, 그 자체는 600000으로 상한), `respectRetryAfter: true`. |
208208
| `thinkingToggleModels?` | `string[]` | effort 단계 대신 vendor `thinking.enabled` toggle을 쓰는 chat 모델. |
209209
| `thinkingBudgetModels?` | `string[]` | 정수 `thinking_budget`을 쓰는 chat 모델. effort를 budget 비율로 매핑합니다. |
210210
| `noVisionModels?` | `string[]` | 텍스트 전용 모델. [비전 사이드카](/ko/guides/sidecars/)가 이미지를 설명합니다. Ollama의 `:size` 태그도 일치시킵니다. |

‎docs-site/src/content/docs/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,7 @@ or bind the forward explicitly to loopback (`ssh -L 127.0.0.1:20100:localhost:10
350350
| `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | Provider-local, disabled-by-default passthrough SSE repair for exact `message` / `reasoning` placeholder ids and missing terminal ids. Downstream only; function-call ids and `call_id` are never rewritten. |
351351
| `autoToolChoiceOnlyModels?` | `string[]` | Models whose `tool_choice` accepts only `auto` or `none`; forced/named choices are downgraded. |
352352
| `preserveReasoningContentModels?` | `string[]` | Models that require prior assistant `reasoning_content` to remain in chat history. |
353-
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Key-auth providers only. Opt-in same-target 429 retry: wait (upstream `Retry-After` or the fixed interval) and replay the identical request on the same key before any key failover. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (a single wait is capped at 600000), `respectRetryAfter: true`. |
353+
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key providers only (`authMode: "key"`). Opt-in same-target 429 retry: when `retryOn429` is absent the feature is off; object presence enables it unless `enabled: false`. On 429 the proxy waits (upstream `Retry-After` or the fixed interval) and replays the identical request on the same key before any key failover — across the main text-turn recovery loop, the Responses passthrough wire, the image/video bridge, the web-search sidecar, and terminal continuations. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (any single wait is capped at `maxIntervalMs`, itself capped at 600000), `respectRetryAfter: true`. |
354354
| `thinkingToggleModels?` | `string[]` | Chat models using a vendor `thinking.enabled` toggle instead of an effort ladder. |
355355
| `thinkingBudgetModels?` | `string[]` | Chat models using an integer `thinking_budget`; effort is mapped to a budget fraction. |
356356
| `noVisionModels?` | `string[]` | Text-only models — the [vision sidecar](/guides/sidecars/) describes images for them. Matching tolerates an Ollama `:size` tag. |

‎docs-site/src/content/docs/ru/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ Responses и Chat Completions принимают только выделенны
228228
| `parallelToolCalls?` | `boolean` | Включает/отключает параллельные вызовы инструментов. В OpenAI Chat включено по умолчанию; не-chat-адаптеры объявляют поддержку только при явном `true`. |
229229
| `autoToolChoiceOnlyModels?` | `string[]` | Модели, у которых `tool_choice` принимает только `auto` или `none`; принудительные/именованные выборы понижаются. |
230230
| `preserveReasoningContentModels?` | `string[]` | Модели, требующие сохранять предыдущий assistant `reasoning_content` в истории чата. |
231-
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с key-auth. Опциональный повтор при 429 на том же таргете: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. По умолчанию: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (одно ожидание ограничено 600000), `respectRetryAfter: true`. |
231+
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с API-ключом (`authMode: "key"`). Опциональный повтор при 429 на том же таргете: если `retryOn429` отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. При 429: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей — покрывает основной цикл восстановления текстовых ходов, passthrough-канал Responses, мост изображений/видео, sidecar web-search и терминальные продолжения. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. По умолчанию: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (любое ожидание ограничено `maxIntervalMs`, который сам ограничен 600000), `respectRetryAfter: true`. |
232232
| `thinkingToggleModels?` | `string[]` | Chat-модели, использующие вендорский переключатель `thinking.enabled` вместо лестницы уровней рассуждений. |
233233
| `thinkingBudgetModels?` | `string[]` | Chat-модели, использующие целочисленный `thinking_budget`; уровень отображается в долю бюджета. |
234234
| `noVisionModels?` | `string[]` | Модели только для текста — [vision-сайдкар](/ru/guides/sidecars/) описывает для них изображения. Сопоставление допускает тег Ollama `:size`. |

‎docs-site/src/content/docs/zh-cn/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,7 +196,7 @@ Codex Direct 透传,两个 bearer 域不能混淆。仪表盘的 API 标签页
196196
| `parallelToolCalls?` | `boolean` | 启用或禁用并行工具调用。OpenAI Chat 默认开启;非 chat adapter 只有显式为 `true` 时才公布支持。 |
197197
| `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` 只接受 `auto` 或 `none` 的模型;forced/named 选择会降级。 |
198198
| `preserveReasoningContentModels?` | `string[]` | 要求在 chat history 中保留先前 assistant `reasoning_content` 的模型。 |
199-
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 key-auth 提供商。可选的同目标 429 重试:等待(上游 `Retry-After` 或固定间隔)后在相同 key 上重放完全相同请求,再进入任何 key 故障转移。Codex 自身从不重试 429,因此这是单 key 提供商唯一的防线。默认值:`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(单次等待上限 600000)、`respectRetryAfter: true`。 |
199+
| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 API-key 提供商(`authMode: "key"`)。可选的同目标 429 重试:未配置 `retryOn429` 时功能关闭;对象存在即启用,除非 `enabled: false`。收到 429 时等待(上游 `Retry-After` 或固定间隔)后在相同 key 上重放完全相同请求,再进入任何 key 故障转移——覆盖主文本恢复循环、Responses passthrough、图像/视频桥、web-search 侧车与终结续接。Codex 自身从不重试 429,因此这是单 key 提供商唯一的防线。默认值:`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(单次等待以 `maxIntervalMs` 为上限,其本身上限 600000)、`respectRetryAfter: true`。 |
200200
| `thinkingToggleModels?` | `string[]` | 使用 vendor `thinking.enabled` toggle,而不是 effort ladder 的 chat 模型。 |
201201
| `thinkingBudgetModels?` | `string[]` | 使用整数 `thinking_budget` 的 chat 模型;effort 会映射成 budget 比例。 |
202202
| `noVisionModels?` | `string[]` | 纯文本模型;[视觉 sidecar](/zh-cn/guides/sidecars/) 会为它们描述图像。匹配时容忍 Ollama `:size` 标签。 |

0 commit comments

Comments
 (0)