Skip to content

Commit 58c36c9

Browse files
committed
fix(proxy): resolve review-bot round on retryOn429 (fail-closed auth, deadlines, recovery labels)
- key-failover: fail closed - only authMode key (or the documented omitted default) may replay; unknown modes rejected; an already-expired HTTP-date Retry-After retries immediately (like Retry-After: 0) - derive: providerConfigSeed preserves the registry auth kind (incl. local) so the gate survives the seed round-trip and routing - core: release the unread 429 body BEFORE every backoff (main loop, passthrough, continuations); passthrough and continuation replay sends record the rate-limit-429 recovery kind; continuation budget hoisted outside the failover loop so rotation/413 cannot re-arm it; continuation waits sleep on the upstream signal so an SSE body-cancel aborts them too - images/web-search loops: release the 429 body before the wait, restart the response-header deadline after each deliberate wait (backoffs never consume the connect budget or surface as 504), and record rate-limit-429 on replay sends - config: load-time degradation for invalid optional retryOn429 fields (warn + drop the field) instead of tripping the whole schema and hiding all providers behind a default config; the management write boundary still rejects - tests: fail-closed auth modes (incl. unknown), expired HTTP-date, per-request budget across 2-key pools (e2e + terminal continuation), byte-identical replay bodies/auth (passthrough + continuation), recovery telemetry in both loops, config load degradation, registry auth-kind preservation, typed usage-log entry - docs: retry-wait vs total-latency and attempts+poolKeys volume bounds, identical-replay equivalence, deadline/backoff behavior in devlog 010 and structure/04; retryOn429 boundary notes in the provider guide and adapter reference (English + ja/ko/ru/zh-cn)
1 parent a06a416 commit 58c36c9

26 files changed

Lines changed: 366 additions & 46 deletions

File tree

‎devlog/_plan/260802_429_same_target_retry/010_design.md‎

Lines changed: 34 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,17 @@ existing multi-key failover runs. Default off → zero behavior change for exist
2626
- `src/types.ts`: `RateLimitRetryPolicy` interface + `OcxProviderConfig.retryOn429`.
2727
- `src/config.ts`: zod schema entry (zod's default strip inside the object — an unknown key is
2828
dropped, never a config-rejecting error; the outer provider schema stays passthrough).
29+
Load-time degradation: one hand-edited invalid optional field (e.g. `attempts: 0`) is dropped
30+
with a warning instead of tripping the whole schema and hiding all providers behind a default
31+
config; the management write boundary still rejects invalid policies.
2932
- `src/providers/key-failover.ts`: `rateLimitRetryPolicyFor` (normalize/default) and
3033
`rateLimitRetryDelayMs` (Retry-After seconds/HTTP-date → capped, else `intervalMs`),
3134
reusing the existing `parseRetryAfterMs` cooldown parser; the fixed fallback is capped at
32-
`maxIntervalMs` too, so a single wait never exceeds the cap. API-key providers only
33-
(`authMode: "key"`): OAuth/forward credentials are never replayed on the same token, and
34-
local runtimes have no remote key to preserve.
35+
`maxIntervalMs` too, so a single wait never exceeds the cap. Fail closed: only `authMode:
36+
"key"` (or the documented omitted default for custom API-key providers) may use replays —
37+
OAuth/forward are never replayed on the same token, local runtimes have no remote key to
38+
preserve, and unknown values are rejected. `providerConfigSeed` preserves the registry auth
39+
kind (including `"local"`) so the gate survives the seed round-trip.
3540
- `src/usage/log.ts`: new `AttemptRecoveryKind` member `"rate-limit-429"`.
3641
- `src/server/responses/core.ts`: in the pre-stream recovery loop, BEFORE the multi-key
3742
failover `while`, wait then `rebuildAndRefetch("rate-limit-429")`. Abort during the wait
@@ -45,6 +50,9 @@ existing multi-key failover runs. Default off → zero behavior change for exist
4550
- image/video bridge and web-search sidecar loops (`src/images/loop.ts`,
4651
`src/web-search/loop.ts`) — before their `on429` key rotation;
4752
- Anthropic terminal-guard continuations — before key/account failover.
53+
Every surface releases the unread 429 body BEFORE the backoff, records the
54+
`rate-limit-429` recovery kind on replay sends, and (bridges) restarts the response-header
55+
deadline after each deliberate wait.
4856
Covers Responses, chat completions, and routed Claude messages (they all enter
4957
`handleResponses`).
5058

@@ -54,17 +62,34 @@ existing multi-key failover runs. Default off → zero behavior change for exist
5462
request is lossless (same invariant as the transient-5xx layer in `lib/upstream-retry.ts`).
5563
- Ordering: same-key retries run before failover, so "primary-first" users keep their key on
5664
rate-limit blips; failover still works after retries exhaust.
57-
- Latency bound: worst case `attempts × maxIntervalMs` (default 3 × 60 s = 180 s) when
58-
honoring upstream `Retry-After`; `attempts × intervalMs` (default 15 s) when
59-
`respectRetryAfter=false` or no header is present.
65+
- Retry-wait bound: the SLEEP component is at most `attempts × maxIntervalMs` (default
66+
3 × 60 s = 180 s) when honoring upstream `Retry-After`; `attempts × intervalMs`
67+
(default 15 s) when `respectRetryAfter=false` or no header is present. Total request
68+
latency is higher: every attempt also consumes its own connect/response time (bounded by
69+
`connectTimeoutMs`), so the documented bound covers deliberate waits only.
70+
- Identical replay: rebuilds are deterministic for the same parsed request (same serialized
71+
body and auth headers); the passthrough/continuation/e2e tests assert byte-identical bodies
72+
and identical auth headers across replays, not just send counts.
6073
- Abort during the wait: the sleep is abort-aware — when the server observes the client
6174
disconnect (Bun propagates this asynchronously, observed 1–10 s), the wait is interrupted,
6275
the unread 429 body is released, and the request is cancelled with 499 before any replay.
6376
Because the propagation is async, a replay can still precede the cancel if the interval
64-
elapses first; that is bounded by the same `attempts` budget.
77+
elapses first; that is bounded by the same `attempts` budget. Terminal continuations sleep
78+
on the upstream signal, so a body-cancel (SSE already streaming) aborts the wait too.
6579
- Concurrency: each request honors its own policy independently — no process-wide cooldown is
66-
shared between concurrent requests (unlike the Kiro 429 pattern). Opt-in and bounded, so a
67-
storm multiplies upstream volume by at most `attempts + 1` per request.
80+
shared between concurrent requests (unlike the Kiro 429 pattern). Upstream volume per
81+
request: same-key replays add at most `attempts` sends, then multi-key failover adds up to
82+
`poolKeys − 1` more (or Anthropic account rotations), so the combined bound is
83+
`attempts + poolKeys` sends — a storm multiplies by that factor per request, not by
84+
`attempts + 1`.
85+
- Header deadlines: the image/video and web-search bridge loops restart their response-header
86+
deadline after each deliberate wait, so backoffs never consume the connect budget and a
87+
rate-limit wait is never misattributed as a 504 header timeout.
88+
- Expired `Retry-After`: a valid HTTP-date already in the past retries immediately (same as
89+
numeric `Retry-After: 0`) instead of falling back to the fixed interval.
90+
- Recovery observability: every retry surface records the `rate-limit-429` recovery kind
91+
(normal loop, passthrough wire, image/video bridge, web-search sidecar, terminal
92+
continuations), so usage logs explain the extra sends.
6893
- Final 429 still carries `Retry-After` for clients that honor it (Claude Code).
6994

7095
## Tests

‎docs-site/src/content/docs/guides/providers.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ labels local presets separately; those normally omit both `authMode` and `apiKey
3939
| `forward` | Relays **your incoming Codex auth headers** verbatim to the provider — no key stored. This is the ChatGPT-login passthrough. | OpenAI (`openai-responses` adapter). |
4040
| `oauth` | Resolves a stored OAuth access token (auto-refreshed before expiry) and uses it as the bearer key. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. |
4141

42+
The [`retryOn429`](/reference/configuration/) same-key 429 replay applies only to API-key
43+
providers (`authMode: "key"`). OAuth, forward, and local presets are excluded — their
44+
credentials must never be replayed on the same token, and local runtimes have no remote key to
45+
preserve.
46+
4247
## 1. ChatGPT login (forward / passthrough)
4348

4449
The `openai` provider needs **no API key**. Direct forwards credentials from your existing

‎docs-site/src/content/docs/ja/guides/providers.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ max input 922,000 で `*-pro` virtual ID は公開状態を維持し、wire で
3535
| `forward` | **受け取った Codex 認証ヘッダーを**プロバイダーにそのまま中継します — キーを保存しません。ChatGPT ログインのパススルーです。 | OpenAI(`openai-responses` アダプター)。 |
3636
| `oauth` | 保存された OAuth アクセストークンを読み込み bearer キーとして使い、期限切れ前に自動更新します。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 |
3737

38+
[`retryOn429`](/ja/reference/configuration/)(同一キーでの 429 リトライ)は API キー プロバイダー
39+
(`authMode: "key"`)のみに適用されます。OAuth・forward・ローカル プリセットは除外されます —
40+
同じトークンを再送すべきではなく、ローカルランタイムには保存すべきリモートキーがありません。
41+
3842
## 1. ChatGPT ログイン(forward / パススルー)
3943

4044
デフォルトプロバイダーは**API キー不要**です。既存の `codex login` の認証情報を OpenAI Responses バックエンドに

‎docs-site/src/content/docs/ja/reference/adapters.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ interface ProviderAdapter {
3838
**対象:** OpenAI **Responses API**。**`passthrough: true`** — 元のリクエスト本文をそのまま渡し、レスポンスを **変換せずに** ストリーミングします。
3939
**認証:** `forward`(呼び出し元ヘッダー中継)または `key`。
4040

41+
`key` 認証では、[`retryOn429`](/ja/reference/configuration/) もここに適用されます: プリストリームの
42+
429 は、翻訳された `openai-chat` / Anthropic リクエスト経路と同様に、同じキーで同一リクエストを
43+
待機して再送します。カスタム `runTurn` トランスポートは HTTP リトライ ループの対象外です。
44+
4145
- `forward` URL → `{baseUrl}/responses`。`key` provider はデフォルトで従来の `{baseUrl}/v1/responses` 構築を使います。
4246
- `key` provider は検証済みの相対 `responsesPath` を設定できます。adapter は `baseUrl` 末尾の `/` を 1 つ除き、`{trimmedBaseUrl}{responsesPath}` に送信します。Ark Agent Plan では `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` と `responsesPath: "/responses"` を使います。
4347
- `forward` モードでは安全なヘッダー許可リスト(`FORWARD_HEADERS`)だけを中継します。authorization、ChatGPT account id、OpenAI beta/originator/session ヘッダーが対象です。この ChatGPT ログイン経路は [サイドカー](/ja/guides/sidecars/) にも使われます。

‎docs-site/src/content/docs/ko/guides/providers.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ shipped v1 config는 marker 2의 단일 옵션 행으로 자동 이관됩니다.
3535
| `forward` | **수신된 Codex 인증 헤더를** 프로바이더에 그대로 중계합니다 — 키를 저장하지 않습니다. ChatGPT 로그인 패스스루입니다. | OpenAI (`openai-responses` 어댑터). |
3636
| `oauth` | 저장된 OAuth 액세스 토큰을 불러와 bearer 키로 사용하며, 만료 전에 자동 갱신합니다. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor. |
3737

38+
[`retryOn429`](/ko/reference/configuration/)(동일 키 429 재시도)는 API 키 프로바이더
39+
(`authMode: "key"`)에만 적용됩니다. OAuth·forward·로컬 프리셋은 제외됩니다 — 같은 토큰을
40+
재전송해서는 안 되며, 로컬 런타임에는 보존할 원격 키가 없습니다.
41+
3842
## 1. ChatGPT 로그인 (forward / 패스스루)
3943

4044
기본 프로바이더는 **API 키가 필요 없습니다**. 기존 `codex login`의 자격 증명을 OpenAI Responses 백엔드로

‎docs-site/src/content/docs/ko/reference/adapters.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,10 @@ interface ProviderAdapter {
4545
**변환하지 않은 채** 스트리밍합니다.
4646
**인증:** `forward`(호출자 헤더 중계) 또는 `key`.
4747

48+
`key` 인증에서는 [`retryOn429`](/ko/reference/configuration/)도 여기에 적용됩니다: 사전 스트림
49+
429는 번역된 `openai-chat`/Anthropic 요청 경로와 동일하게 같은 키로 동일 요청을 대기 후
50+
재전송합니다. 커스텀 `runTurn` 전송은 HTTP 재시도 루프에 포함되지 않습니다.
51+
4852
- `forward` URL → `{baseUrl}/responses`. `key` provider는 기본적으로 기존 `{baseUrl}/v1/responses` 구성을 사용합니다.
4953
- `key` provider는 검증된 상대 `responsesPath`를 설정할 수 있습니다. adapter는 `baseUrl` 끝의 `/` 하나를 제거하고 `{trimmedBaseUrl}{responsesPath}`로 전송합니다. Ark Agent Plan은 `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"`와 `responsesPath: "/responses"`를 사용합니다.
5054
- `forward` 모드에서는 안전한 헤더 허용 목록(`FORWARD_HEADERS`)만 중계합니다. authorization,

‎docs-site/src/content/docs/reference/adapters.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,11 @@ provider — xAI, Kimi, DeepSeek, GLM, Groq, OpenRouter, Ollama (local & cloud),
4545
streams the response back **untranslated**.
4646
**Auth:** `forward` (relay the caller's headers) or `key`.
4747

48+
For `key` auth, [`retryOn429`](/reference/configuration/) applies here too: a pre-stream 429
49+
waits and replays the identical request on the same key before any other handling, exactly like
50+
the translated `openai-chat` / Anthropic request path. Custom `runTurn` transports are not part
51+
of the HTTP retry loop.
52+
4853
- `forward` URL → `{baseUrl}/responses`. A `key` provider defaults to the legacy `{baseUrl}/v1/responses` construction.
4954
- A `key` provider may set a validated relative `responsesPath`; the adapter removes one trailing slash from `baseUrl` and sends `{trimmedBaseUrl}{responsesPath}`. For Ark Agent Plan, use `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` with `responsesPath: "/responses"`.
5055
- In `forward` mode only a safe header allowlist is relayed (`FORWARD_HEADERS`): authorization,

‎docs-site/src/content/docs/ru/guides/providers.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,11 @@ description: Все способы, которыми opencodex аутентиф
4141
| `forward` | Передаёт провайдеру **входящие заголовки аутентификации Codex** без изменений — ключ не хранится. Это сквозной режим (passthrough) входа через ChatGPT. | OpenAI (адаптер `openai-responses`). |
4242
| `oauth` | Берёт сохранённый OAuth-токен доступа (автоматически обновляется до истечения срока) и использует его как bearer-ключ. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. |
4343

44+
Повтор при 429 на том же ключе ([`retryOn429`](/ru/reference/configuration/)) применим только к
45+
провайдерам с API-ключом (`authMode: "key"`). Пресеты OAuth, forward и local исключены — их
46+
учётные данные нельзя повторно отправлять по тому же токену, а у локальных сред выполнения нет
47+
удалённого ключа.
48+
4449
## 1. Вход через ChatGPT (forward / passthrough)
4550

4651
Провайдеру `openai` **не нужен API-ключ**. Direct пересылает учётные данные вашего существующего

‎docs-site/src/content/docs/ru/reference/adapters.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,10 @@ interface ProviderAdapter {
4848
запроса и стримит ответ обратно **без преобразования**.
4949
**Аутентификация:** `forward` (ретрансляция заголовков вызывающей стороны) или `key`.
5050

51+
При `key`-аутентификации [`retryOn429`](/ru/reference/configuration/) действует и здесь: 429 до
52+
начала потока ждёт и повторяет идентичный запрос на том же ключе, как и в переводимом пути
53+
`openai-chat`/Anthropic. Пользовательские транспорты `runTurn` в цикл HTTP-повторов не входят.
54+
5155
- URL для `forward` → `{baseUrl}/responses`. Провайдер с `key` по умолчанию сохраняет прежнее построение `{baseUrl}/v1/responses`.
5256
- Провайдер с `key` может задать проверенный относительный `responsesPath`: адаптер удаляет один завершающий `/` из `baseUrl` и отправляет запрос на `{trimmedBaseUrl}{responsesPath}`. Для Ark Agent Plan используйте `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` и `responsesPath: "/responses"`.
5357
- В режиме `forward` ретранслируется только безопасный allowlist заголовков (`FORWARD_HEADERS`):

‎docs-site/src/content/docs/zh-cn/guides/providers.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ shipped v1 配置自动迁移到 marker 2 的单一选项行。原配置只保
3434
| `forward` | 将**你传入的 Codex 认证请求头**原样转发给提供商——不存储任何密钥。这就是 ChatGPT 登录的透传方式。 | OpenAI(`openai-responses` adapter)。 |
3535
| `oauth` | 读取已存储的 OAuth 访问令牌(过期前自动刷新),并将其用作 bearer 密钥。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 |
3636

37+
[`retryOn429`](/zh-cn/reference/configuration/)(同 key 的 429 重试)仅适用于 API-key 提供商
38+
(`authMode: "key"`)。OAuth、forward 与本地预设均被排除——同一 token 绝不可重放,本地运行时
39+
也没有需要保留的远程 key。
40+
3741
## 1. ChatGPT 登录(forward / 透传)
3842

3943
默认提供商**不需要 API 密钥**。它将你现有 `codex login` 的凭据直接转发到 OpenAI Responses 后端:

0 commit comments

Comments
 (0)