[Bug]: Intermittent canonical OpenAI synthetic 429 upstream_reset_replay_refused on v2.91.0; request safe transport mitigation #7720

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   streamingSSE, WebSocket, terminal stream framesSSE, WebSocket, terminal stream frames

   on Oct 25, 2026
2. Ingwannu commented on Oct 25, 2026

   

   Ingwannu

   on Oct 25, 2026

   Owner

   More actions

   I checked the released v2.91.0 tree (83ec4f75a4) and current dev (6672cfb4a2). This is a local replay-safety refusal, not evidence that OpenAI rate-limited the account. The distinction in this report is correct.

   1. What can produce it: isConnectionResetError and fetchWithResetRetry recognize ECONNRESET, EPIPE, and two reset/closed-socket message shapes before the fetch has returned a response. They deliberately do not classify ordinary aborts/timeouts, DNS, connection refusal, or an ordinary HTTP error response as that first reset. The code does not prove whether a recognized reset came from idle connection reuse, an intermediary, or the origin, nor whether the POST executed. If an operator replacement was already spent, other replacement failures/statuses can also settle as the same refusal. Consequently the code alone is not a unique stage/root-cause identifier.
   2. Mitigation: applyUpstreamRecoveryInit disables keep-alive on a permitted replacement send. That is not a verified pre-send remedy for your incident, and enabling retryOnReset authorizes potentially duplicate inference. I do not recommend changing that setting or selecting a different transport as a proven fix on this evidence. The existing provider picker can remain in use while we collect the next failure.
   3. Fix status: I have not established a post-2.91.0 change that fixes the underlying reset. Current dev still has this boundary. Recent no-retry/classification changes must not be represented as a connection-reset cure. Synthetic refusals are explicitly excluded from ordinary account-rotation/quota handling; a separate cooldown observation would need its own correlated evidence.
   4. Next occurrence: please attach one redacted usage row with timestamp and timezone, exact OpenCodex/Bun/OS versions, actual transport setting at that time, status, errorCode, transportPhase, and the attempt's failureStage, failureCause, sendCount, recoveryKinds, recoveryWithheld, and content-free codexWsStage only when those fields exist. resendPermission is derived from the stage/cause, not a stored usage field; HTTP rows need not contain WS telemetry. Keep absent values absent. A locally consistent replacement for the request ID is sufficient for correlation; omit account labels, credentials, request content, full headers and raw exception text.

   Leaving this open for transport attribution. The existing refusal deliberately discards the raw reset exception, so the six supplied rows cannot distinguish socket reuse from another reset source. A narrow diagnostic improvement should retain only bounded transport facts and must preserve the no-duplicate-send boundary.
3. runshen0 commented on Oct 25, 2026

   

   runshen0

   on Oct 25, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Follow-up with four new, redacted usage-log records from Oct 25, 2026, as requested. These are from the same macOS Codex App / canonical openai provider setup as the original report. All timestamps are UTC+08:00. OpenCodex v2.91.0 was last verified in the environment; providers.openai.upstreamWebsocket=false was read back on Oct 25 at approximately 12:02 UTC+08, but there is no contemporaneous per-request transport snapshot, so I cannot assert it was the transport mode at each failure. The exact Bun and macOS build versions were not captured. No account IDs, session content, headers, raw exceptions, or raw request IDs are being shared.

   Local timestamp (+08:00)	Model	Status	errorCode	durationMs	Anonymized request correlation
   2026-10-25 18:28:28	gpt-6-luna	429	upstream_reset_replay_refused	136830	6f022c706121
   2026-10-25 18:33:58	gpt-6-luna	429	upstream_reset_replay_refused	5394	6ba7821d5b40
   2026-10-25 21:09:27	gpt-6-luna	429	upstream_reset_replay_refused	635	e0c633187191
   2026-10-25 21:09:31	gpt-6-luna	429	upstream_reset_replay_refused	993	bfcff13fd6bd

   Each record's top-level diagnostic fields are identical:

   {"failureStage":"headers-only","failureCause":"rate-limit","status":429,"errorCode":"upstream_reset_replay_refused"}

   Each has exactly one reported attempt with the same stage/cause/error, and:

   {"sendCount":1,"recoveryKinds":[]}

   The inspected output did not include transportPhase, recoveryWithheld, or a top-level codexWsStage object. They are absent from the supplied extraction; please don't infer values for them. Request correlation strings above are truncated SHA-256 hashes of local request IDs, not the original request IDs.

   Specific question prompted by these records: how can failureStage=headers-only and failureCause=rate-limit coexist with the synthetic upstream_reset_replay_refused code on a one-send attempt (sendCount=1, no recoveryKinds)? Does this indicate (a) a real upstream HTTP 429 being projected into the proxy's no-replay safety code, (b) an ambiguous transport reset after a headers-only upstream exchange, or (c) a classification/projection bug? The 136.8-second record is particularly different from the other 0.6–5.4-second failures.

   As your previous triage notes, the synthetic public 429 is not proof that OpenAI rate-limited the account, but failureCause=rate-limit also should not be silently discarded. Please advise a safe way to capture the actual upstream status / pre-header stage (without credentials or content and without ambiguous duplicate sends) and whether the available fields can distinguish these cases. The aim remains to keep the Codex App graphical third-party model picker operational.

   No retryOnReset change or live fault injection was made to produce these records.
4. Ingwannu commented on Oct 25, 2026

   

   Ingwannu

   on Oct 25, 2026

   Owner

   More actions

   The new rows expose a real attribution question; headers-only plus rate-limit should not be treated as proof of an upstream quota response. I found and reproduced one concrete projection defect on current dev and opened #7727 (Draft, 1119578e189063963234d6db16dcf6ba6299d624).

   The two transports are independent: upstreamWebsocket=false chooses HTTP upstream, but Codex App can still enter OCX through client WebSocket ingress. The HTTP ingress already stamps the in-process replay-refusal marker as terminalSource: synthetic / failureCause: transport-ambiguous; the WebSocket ingress finalizes its own log and omitted that stamp. Its numeric429 fallback can therefore report rate limiting despite an ambiguous reset. headers-only is the recorder's response-progress stage; it does not establish that OpenAI returned headers or executed the request.

   In the new hermetic fixture, canonical Codex-login and keyed-gateway HTTP fetches each throw a synthetic ECONNRESET. Both send exactly once, have no recovery and expose the replay-refusal429 to the WebSocket client. Before the patch both fail the provenance assertion (the stored code also becomes rate_limit_exceeded in this current-dev path); afterwards the row/attempt retain the refusal code and transport-ambiguous. A real upstream429 remains rate-limit. The regression plus existing attribution controls pass 19 tests / 22652 assertions. The stored-code difference from your extraction is another reason not to claim every historical row is now explained.

   This fixes diagnostic provenance only, not the underlying reset. Your missing transport fields remain unknown, and neither 136.8s nor 0.6–5.4s identifies the connection phase by itself. No duplicate send or change to retryOnReset is proposed. Until a build containing the patch is available, preserve the closed fields already supplied, the exact OCX/Bun/OS versions and whether the request used client HTTP versus WebSocket when that is actually observable; do not share credentials, headers, account IDs or content. A missing field should stay recorded as absent.

   The PR remains Draft: focused/layout/structure/privacy checks pass, while local full TypeScript and docs search-index generation hit the existing five-minute resource envelope; exact-head hosted CI is pending. The issue remains open for the original transport-reset investigation. No live fault injection, account operation, config edit or Codex Security scan was performed.
5. runshen0 commented on Oct 26, 2026

   

   runshen0

   on Oct 26, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Additional operator observation on 2026-10-26 (UTC+08:00): the user changed the VPN egress from a Singapore endpoint to a Los Angeles (US) endpoint and still sees the same Codex UI message exceeded retry limit, last status: 429 Too Many Requests during ordinary project work. This change did not provide reliable relief.

   Attribution limits: No matching post-switch usage.jsonl row has been supplied yet, and the actual OCX process-level egress/peer routing on both endpoints was not captured. We therefore cannot assert that the new user-visible 429 is specifically another upstream_reset_replay_refused, or that different public IPs were definitely used by OCX; the observation only weakens the simple "Singapore VPN geography alone" hypothesis. No raw IP or network subscription details are shared.

   I also saw the maintainer's draft #7727 and understand it fixes WebSocket ingress log attribution, not the underlying reset. Thank you for the precise distinction.

   Focused maintainer question: is OCX_FRESH_CONNECTION_HOSTS applicable and safe on the canonical ChatGPT-login OpenAI HTTP/SSE forwarding path in v2.91.0, and has anyone established it reduces pre-header ambiguous resets there? The reporter wants to preserve the GUI model picker and should not enable ambiguous POST retransmission. If not verified, please say so rather than recommend a speculative production tweak. We can provide a next failure's content-free transport fields once captured.
6. jingyihu818 commented on Oct 26, 2026

   

   jingyihu818

   on Oct 26, 2026 – with ChatGPT Codex Connector

   More actions

   Hi, I'm seeing a closely matching failure on Windows using OpenCodex with the official OpenAI (Codex login) provider.

   A fresh occurrence from the OpenCodex UI:

   • Time: 2026-10-26 12:39:56 (UTC+08:00)
   • Model: gpt-6.1-sol, high reasoning effort
   • Displayed error: 429
   • errorCode: upstream_reset_replay_refused
   • Failure reason: "connection interrupted, upstream state unknown (reached stage: headers-only)"
   • Upstream send count: 1; replay was refused because the upstream state was unknown.

   In several runs, I subjectively noticed the failure appearing about 15 seconds after response activity begins. This is a UI observation, not yet a precisely measured transport interval. I can collect redacted logs for a future occurrence.

   I understand from the maintainer discussion that this is a proxy-generated replay-safety refusal, not proof of a genuine upstream OpenAI rate limit, and that draft PR #7727 fixes log attribution rather than the underlying reset.

   A few questions:

   1. Has anyone found a verified, safe mitigation for the underlying connection resets, without risking duplicate inference via retryOnReset?
   2. Which sanitized diagnostic fields would best distinguish an OpenCodex transport/connection-management problem from upstream or network resets?
   3. Is there a known regression range or a stable earlier OpenCodex version worth testing as a controlled rollback (e.g., two versions earlier)? If so, what is the supported rollback procedure that preserves configuration/authentication?
   4. Does a failure occurring after visible response activity help narrow the connection phase, or could that be misleading from the UI alone?

   Thanks for investigating. I can follow up with content-free logs and version details.

   👍React with 👍1Reacted by paleorbit
7. lidge-jun

   added a commit that references this issue on Oct 27, 2026

   fix(logs): preserve replay refusal attribution on WebSocket ingress (#…

   ...

   7deb359
8. lidge-jun commented on Oct 27, 2026

   

   lidge-jun

   on Oct 27, 2026

   Owner

   More actions

   Partial progress merged into dev: #7830 (merge commit 7deb359, carrying #7727).

   On a direct provider route, a proxy-generated upstream_reset_replay_refused 429 now keeps its transport-ambiguous cause and a synthetic terminal source on WebSocket ingress too, matching HTTP ingress. Usage logs and account health no longer show it as a provider rate limit. A genuine upstream 429 keeps its rate-limit attribution.

   Still open:

   • The underlying connection resets and the transport mitigation this issue asks for are not addressed; this change only fixes attribution for the next occurrence.
   • Follow-up: when a combo rebuilds a failed child's response (src/server/responses/core-combo-failure.ts, consumeComboFailure), the replay-refusal marker is not carried over, and adoptFailedChildLog keeps the child attempt as recorded. A refusal served through a combo is therefore still attributed as a rate limit on both HTTP and WebSocket ingress. Fixing it needs the marker carried through the rebuild, the child attempt stamped, and regressions on both ingresses.

   Leaving this issue open.
9. lidge-jun

   added a commit that references this issue on Oct 28, 2026

   fix(responses): preserve combo replay-refusal attribution (#7864)

   ...

   f26d6e8
10. lidge-jun commented on Oct 28, 2026

    

    lidge-jun

    on Oct 28, 2026

    Owner

    More actions

    Combo follow-up merged into dev: #7864 (f26d6e8). A proxy-generated upstream_reset_replay_refused 429 served through a combo now keeps the refusal code, the transport-ambiguous cause and a synthetic terminal source on both HTTP and WebSocket ingress, on the logical row and on the serving child attempt. It no longer produces retry-after or quota reset metadata for combo cooling. A genuine upstream 429 through a combo keeps rate-limit attribution.

    Still open: the underlying connection resets and a transport mitigation. Both #7830 and #7864 fix attribution only, so the next occurrence is reported accurately.
11. leonforgit commented on Oct 28, 2026

    

    leonforgit

    on Oct 28, 2026 – with ChatGPT Codex Connector

    More actions

    Additional evidence: synthetic 429 across two unrelated upstreams in the same two-minute window

    Environment

    • OpenCodex 2.93.0 (standalone macOS app), proxy on 127.0.0.1:10100, websockets=false in config.json (no per-request transport snapshot was captured), default provider openai (authMode=forward, codexAccountMode=pool, baseUrl=https://chatgpt.com/backend-api/codex).
    • Codex CLI bundled with ChatGPT.app, 0.165.0-alpha.1.2.
    • A second provider configured in the same client: deepseek (baseUrl=https://api.deepseek.com, responsesPath=/responses, statelessResponses=true). The failing attempt itself was recorded with adapter=openai-responses and protocolTrace.mode=native / reasonCodes=[same-wire-native], so both routes were native Responses rather than a translated Chat path.
    • macOS, Wi-Fi behind a system-wide VPN tunnel. pmset -g log shows no system sleep in the window; only display assertions.

    What happened (UTC+08:00)

    Four synthetic 429s in 111 seconds, split across two different upstreams:

    Time	Provider	Model	Effort	durationMs	failureStage	failureCause
    11:34:57	openai	gpt-6.1-terra	max	5983	headers-only	transport-ambiguous
    11:35:04	openai	gpt-6.1-terra	max	521	headers-only	transport-ambiguous
    11:36:39	deepseek	deepseek-flash	max	8385	headers-only	transport-ambiguous
    11:36:48	deepseek	deepseek-flash	max	8402	headers-only	transport-ambiguous

    All four rows report errorCode=upstream_reset_replay_refused, sendCount=1, recoveryKinds=[], resendPermission=refused-ambiguous, terminalSource=synthetic.

    The point I want to add to the thread: the first two requests go to chatgpt.com and the last two go to api.deepseek.com. They share only the local egress path. A capacity 429 at one provider cannot explain a pre-header failure at the other 100 seconds later.

    Normal requests resumed at 11:37:53 on both providers with no configuration change, so this is a burst, not a steady state.

    This is not the quota class

    The same log does contain genuine rate-limit 429s, and they look different:

    • 8 rows on 2026-10-28, all openai / gpt-6.1-terra at effort=max, failureCause=rate-limit, durations 0.7-11.3 s.
    • In the same period gpt-5.6-luna (61 requests) and codex-auto-review (61 requests) had zero 429s.

    So I am not claiming the rate-limit rows are synthetic. The two classes appear distinguishable by failureCause, which is useful.

    Diagnostic gap

    The synthetic rows carry no transportPhase field; successful rows in the same log do (terminal_sse). Without it the log cannot distinguish:

    1. the connection never completed (nothing could have been sent),
    2. the body was still being written,
    3. the body was fully sent and we were waiting for headers.

    Only (1) is clearly safe to replay without the duplicate-inference risk that makes retryOnReset unacceptable as a default.

    Request

    • Record the transport phase on synthetic refusals, so an operator can tell "never sent" from "possibly executing" without guessing.
    • If a class of pre-send resets can be proven not to have written the request body, consider replaying only that class automatically. That would remove most of these dead turns without authorizing duplicate inference.

    Happy to provide more rows or run a specific probe if that helps.
12. Ingwannu commented on Oct 29, 2026

    

    Ingwannu

    on Oct 29, 2026

    Owner

    More actions

    Following up on the unanswered connection-policy and phase questions, against trusted dev 9b78a16f4fe0c300527e6b183052407269bb941d and the actual v2.91.0 source:

    • OCX_FRESH_CONNECTION_HOSTS does apply to a matching HTTP physical send, including the canonical ChatGPT HTTP/SSE path, in both revisions. src/server/responses/fetch-helpers.ts matches the final rewritten destination hostname (exact host or subdomain), and passes Connection: close plus keepalive: false to the selected fetch. It does not add a replay or choose a different model. It is an HTTP policy, not a WebSocket policy. This establishes implementation applicability, not a verified reduction of production resets: there is no controlled before/after capture here, and I cannot present it as a proven mitigation. No live configuration was changed or provider call made.
    • A UI activity period, stage: headers-only, sendCount: 1, or raw ECONNRESET does not prove that no request bytes reached the upstream. Missing transportPhase is unknown, not never-sent. The synthetic 429/reset-refused class must remain distinct from a genuine upstream quota 429. Failures across two hosts do not by themselves locate the cause at the client, relay or provider.
    • Likewise, the global websockets=false setting alone is not an observed per-request upstream transport record; upstream WebSocket eligibility has its own provider/request selection. Please retain the actual selected transport when available rather than inferring it from that flag.
    • I do not have a verified first-bad version or a verified stable rollback target from these reports. The landed fix(logs): preserve replay refusal attribution on WebSocket ingress #7830/fix(responses): keep replay-refusal attribution through combo failures #7864 provenance work improves diagnosis; it does not resolve the underlying reset or authorize an ambiguous replay. A safe automatic pre-send retry needs transport-owned evidence that dispatch could not have sent bytes, not an elapsed-time or missing-field heuristic.

    For a further redacted sample, the useful bounded fields are build/version, selected HTTP/WS transport, public destination hostname, elapsed duration, terminal code/class/source, send count, any explicit transport phase and recovery trail; absent fields should stay absent. Please omit headers, credentials, account identifiers and request content. Keeping this issue open for the transport root cause and a reproducible phase distinction.
13. github-actions commented on Nov 2, 2026

    

    github-actionsBot

    on Nov 2, 2026 – with GitHub Actions

    Contributor

    More actions

    _{Automated translation bookkeeping — detected language: English.}
14. lidge-jun commented on Nov 10, 2026

    

    lidge-jun

    on Nov 10, 2026

    Owner

    More actions

    Follow-up: #8486 merged into dev as c225873. It does not resolve the underlying reset, so this issue stays open. It changes two things relevant here:

    • When the native Responses lane refuses to resend after an ambiguous reset, the service log now says why: [upstream-retry] ambiguous reset not replaced (<host>, <stage>): ambiguous-no-policy | ambiguous-request-not-replayable | ambiguous-allowance-spent. The line carries only the host and stage, never request content or credentials. On a canonical openai row without retryOnReset, expect ambiguous-no-policy, which means the refusal is the fail-closed default, not a judgment about the turn.
    • Operators who opt into retryOnReset now get the replacement on chained Codex turns too. Previously a previous_response_id the proxy had already expanded locally made every chained turn ambiguous-request-not-replayable ([Bug]: retryOnReset replacements never spent for key-auth openai-responses providers — every pre-header reset answers synthetic 429 (v2.99.0) #8453).

    For a future occurrence, the most useful data is that log line next to the matching usage.jsonl row (status, errorCode, failureStage, sendCount, duration). That pins down which stage reset and whether the opt-in applied. The transport root cause, including the recurring ~15.7 s per send reported in #8453 and here, is still unexplained.
15. lidge-jun commented on Nov 15, 2026

    

    lidge-jun

    on Nov 15, 2026

    Owner

    More actions

    On the next occurrence, please post the [upstream-retry] ambiguous reset not replaced (...) log line (added by #8486) together with the matching usage.jsonl row (status, errorCode, failureStage, sendCount, duration).
16. lidge-jun

    added

    needs-infoWaiting on reporter for a concrete spec or reproductionWaiting on reporter for a concrete spec or reproduction

    on Nov 15, 2026