Skip to content

fix(ai): ChatGPT Responses retries errors it classified as non-retryable#110655

Merged
steipete merged 6 commits into
openclaw:mainfrom
Yigtwxx:fix/chatgpt-responses-non-retryable-retry
Jul 18, 2026
Merged

fix(ai): ChatGPT Responses retries errors it classified as non-retryable#110655
steipete merged 6 commits into
openclaw:mainfrom
Yigtwxx:fix/chatgpt-responses-non-retryable-retry

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Jul 18, 2026

Copy link
Copy Markdown
Contributor
Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

Fixes an issue where users running a ChatGPT/Codex model with an invalid or unauthorized credential would wait roughly 7 seconds before seeing the error, when the failure is known immediately.

The SSE retry loop in the ChatGPT Responses transport classifies each non-OK response with isRetryableError. When that check says the failure is not retryable, the loop builds the friendly error message and throws it — but the throw sits inside the same try block whose catch treats any error as a retryable network failure. The throw is swallowed and the request is sent again.

An invalid API key, a revoked or unauthorized account, or a malformed request is therefore sent maxRetries extra times (4 requests total by default) with 1s + 2s + 4s of backoff in between, before the original message finally surfaces.

Two signals show this was not intentional:

  • The catch carries an ad-hoc !lastError.message.includes("usage limit") string check. That only makes sense if friendly errors were known to leak into the network-retry path — one case was patched, the remaining non-retryable statuses were not.
  • The existing bounded-error-body test (openai-chatgpt-responses.test.ts) asserts toHaveBeenCalledTimes(1) on a 400 response, but it has to use a body starting with "usage limit " for that assertion to hold. Any other body would have retried.

Why This Change Was Made

The classification already exists and is correct; only the control flow discards it. The already-classified failure is now thrown as a private NonRetryableCodexResponseError, and the catch rethrows that type unchanged before its network-retry handling.

Deliberately out of scope: the "usage limit" string check is left in place. It also guards the path where fetch itself rejects, and four existing tests depend on it. Removing it would turn this into a behavior-changing refactor.

User Impact

Authentication and bad-request failures against ChatGPT/Codex models surface immediately instead of after ~7 seconds of pointless backoff, and the provider receives one request instead of four. The error message itself is unchanged, and genuinely retryable failures (429/5xx, rate-limit and overload bodies) still retry exactly as before.

Evidence

Runtime before/after against a real local HTTP server that answers 401 on every request. No fetch stubbing — real sockets, real responses. origin/main was materialized side by side with this branch's HEAD and both were driven through the public streamOpenAICodexResponses export:

Local provider stub listening on 127.0.0.1:60835, always answering 401.

main   | HTTP requests sent: 4 | elapsed: 7093ms | stopReason: error
       | errorMessage: Invalid credentials
HEAD   | HTTP requests sent: 1 | elapsed: 3ms | stopReason: error
       | errorMessage: Invalid credentials

Same surfaced error, one request instead of four, 7093ms down to 3ms.

Proof harness
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";

import { streamOpenAICodexResponses as headStream } from "./packages/ai/src/providers/openai-chatgpt-responses.js";
import { streamOpenAICodexResponses as mainStream } from "./packages/ai/src/providers/openai-chatgpt-responses.mainproof.js";

let requestCount = 0;
const server = createServer((req, res) => {
  requestCount += 1;
  res.writeHead(401, { "content-type": "application/json" });
  res.end(JSON.stringify({ error: { message: "Invalid credentials" } }));
  void req.resume();
});

await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const { port } = server.address() as AddressInfo;

function createJwt(payload: Record<string, unknown>): string {
  const header = Buffer.from(JSON.stringify({ alg: "none", typ: "JWT" })).toString("base64url");
  const body = Buffer.from(JSON.stringify(payload)).toString("base64url");
  return `${header}.${body}.signature`;
}

const model = {
  id: "gpt-5.5",
  name: "GPT-5.5",
  api: "openai-chatgpt-responses",
  provider: "openai",
  baseUrl: `http://127.0.0.1:${port}/backend-api`,
  reasoning: true,
  input: ["text"],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 128_000,
  maxTokens: 16_000,
};

const context = { messages: [{ role: "user", content: "hi", timestamp: 1 }] };
const apiKey = createJwt({ "https://api.openai.com/auth": { chatgpt_account_id: "acct-1" } });

async function measure(label: string, stream: typeof headStream): Promise<void> {
  requestCount = 0;
  const startedAt = Date.now();
  const result = await stream(model as any, context as any, { apiKey, transport: "sse" }).result();
  const elapsedMs = Date.now() - startedAt;
  console.log(
    `${label.padEnd(6)} | HTTP requests sent: ${requestCount} | elapsed: ${elapsedMs}ms | stopReason: ${result.stopReason}`,
  );
  console.log(`${" ".repeat(7)}| errorMessage: ${result.errorMessage}`);
}

await measure("main", mainStream as typeof headStream);
await measure("HEAD", headStream);
server.close();

The .mainproof.ts copy was produced with git show origin/main:packages/ai/src/providers/openai-chatgpt-responses.ts and removed afterwards.

Regression coverage

Four tests added to openai-chatgpt-responses.test.ts:

  • 401, 403 and 400 each send exactly one request and never schedule a backoff timer.
  • A 503 followed by a 401 still retries once, then stops — confirming the retryable path is untouched.

Against origin/main these fail with expected "vi.fn()" to be called 1 times, but got 4 times, each case taking ~7s. With the fix the full file passes 38/38, and its runtime drops from 22.98s to 2.05s.

Local checks: run-vitest 38/38 pass, run-tsgo exit 0, check-max-lines-ratchet OK.

The retry loop builds its friendly error and throws it from inside the same
try block whose catch treats unknown failures as retryable network errors.
Authentication and bad-request failures were therefore resent up to
maxRetries times, adding 7s of backoff before surfacing the same message.

Tag the already-classified failure with a private error type and rethrow it
unchanged from the catch, leaving the retryable paths untouched.
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 18, 2026
@clawsweeper

clawsweeper Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 18, 2026, 5:44 PM ET / 21:44 UTC.

Summary
The PR separates ChatGPT Responses HTTP retry handling from transport exceptions and adds regression coverage that non-retryable 400/401/403 responses do not back off while retryable 503 responses still do.

PR surface: Source +4, Tests +118. Total +122 across 2 files.

Reproducibility: yes. at source level: a non-retryable HTTP response previously reached the transport retry catch, and the supplied local-server trace demonstrates the four-request baseline versus one-request fixed behavior. The review did not independently execute the harness.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: packages/ai/src/providers/openai-chatgpt-responses.retry.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Let the assigned reviewer confirm the final merge result and wait for the current required checks to complete.

Risk before merge

  • [P1] The PR is behind main and its broader required-check fan-out is still running; merge should use the final clean three-way result and completed checks for the reviewed head.

Maintainer options:

  1. Decide the mitigation before merge
    Land the focused HTTP-versus-transport retry separation once the assigned reviewer confirms the clean merge result and current-head required checks remain green.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P2] No mechanical repair is identified; the active, assigned PR needs its normal final owner review and current-head merge/check confirmation.

Maintainer decision needed

  • Question: Should the assigned reviewer merge this focused retry-boundary fix after the final merge result and current-head checks are confirmed?
  • Rationale: The remaining action is ordinary owner judgment on an active, assigned provider-runtime change rather than a mechanical contributor repair.
  • Likely owner: steipete — The timeline assigns this PR to steipete and the final retry-boundary refactor is authored by steipete.
  • Options:
    • Review and merge after checks (recommended): Confirm the clean merge result preserves the tested non-retryable and retryable paths, then merge the focused fix.
    • Request a narrower revision: Request changes only if final-base review reveals a concrete regression in the HTTP or transport retry contract.

Security
Cleared: The reviewed changes are confined to provider retry control flow and tests; no dependency, workflow, permission, secret, or supply-chain change is indicated.

Review details

Best possible solution:

Land the focused HTTP-versus-transport retry separation once the assigned reviewer confirms the clean merge result and current-head required checks remain green.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: a non-retryable HTTP response previously reached the transport retry catch, and the supplied local-server trace demonstrates the four-request baseline versus one-request fixed behavior. The review did not independently execute the harness.

Is this the best way to solve the issue?

Yes. Separating classified HTTP response failures from fetch/transport exceptions is narrower and more maintainable than adding more message-based exclusions to the generic retry catch.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against fa9ae68c3423.

Label changes

Label justifications:

  • P2: This fixes unnecessary provider retries and delay for invalid ChatGPT Responses requests, with a bounded but real user-facing impact.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes after-fix terminal output from a real local HTTP server, comparing the public stream export on the baseline and final branch and showing the same 401 error after one request instead of four.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal output from a real local HTTP server, comparing the public stream export on the baseline and final branch and showing the same 401 error after one request instead of four.
Evidence reviewed

PR surface:

Source +4, Tests +118. Total +122 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 52 48 +4
Tests 1 118 0 +118
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 170 48 +122

What I checked:

Likely related people:

  • steipete: Authored the PR's later retry-boundary refactor and is assigned on the PR timeline, connecting them directly to the final implementation shape. (role: recent area contributor and assigned reviewer; confidence: high; commits: 94065b965fb0, 4839dae60e8c; files: packages/ai/src/providers/openai-chatgpt-responses.ts, packages/ai/src/providers/openai-chatgpt-responses.retry.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (5 earlier review cycles)
  • reviewed 2026-07-18T11:28:14.821Z sha 1e1c2f8 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T11:47:39.066Z sha 44651a2 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T20:13:40.965Z sha 85ec538 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T20:53:13.062Z sha 94065b9 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-18T21:25:27.033Z sha 4839dae :: needs maintainer review before merge. :: none

The provider test file was already near the 1000-line oxlint max-lines
budget, so the new cases pushed it over. Split them out following the
existing <module>.<topic>.test.ts convention.
@knoal

knoal commented Jul 18, 2026

Copy link
Copy Markdown

Real correctness fix, surgical implementation. The NonRetryableCodexResponseError class is the right design — clearly distinguishes retry-classified errors from network errors without changing the catch's general retry logic.

Two test-coverage concerns worth addressing:

  1. What if parseErrorResponse(fakeResponse) itself throws? The new if (error instanceof NonRetryableCodexResponseError) { throw error; } re-throws only when the thrown error is the new class. If parseErrorResponse throws a regular Error (e.g., a malformed JSON body that fails parsing), the catch treats it as a network error and retries — but the underlying status was already classified as non-retryable. The 4-second wait could happen for a status that should have surfaced immediately.

    The fix would be to wrap the throw more defensively — e.g., check response.status after parseErrorResponse throws and decide retryability from the status code, not just from the error type. Or move the throw new NonRetryableCodexResponseError(...) outside the inner try/catch entirely so a parse failure still surfaces with the right retry classification.

  2. Test for the 5xx-retry-still-works path. Tests cover 401, 403, 400 (non-retryable) and assert toHaveBeenCalledTimes(1). The retryable path (500, 502, 503, 504) is presumably tested elsewhere in openai-chatgpt-responses.test.ts, but after this change the catch has a new re-throw branch. Worth a regression test asserting that a 500 still triggers fetchMock.toHaveBeenCalledTimes(2) (or maxRetries + 1) with the new code path. If the new re-throw branch ever widens (e.g., to also re-throw on network errors), the 5xx test catches the regression.

The fix itself is correct for the documented case. These are hardening asks, not blockers.

@Yigtwxx

Yigtwxx commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

@knoal Checked both. Neither is reachable as described, so no change here — details below in case I'm missing something.

1. parseErrorResponse cannot throw on a malformed body.

The JSON.parse is entirely inside a swallowing catch:

async function parseErrorResponse(
  response: Response,
): Promise<{ message: string; friendlyMessage?: string }> {
  const raw = await readChatGptResponsesErrorTextLimited(response);
  let message = raw || response.statusText || "Request failed";
  let friendlyMessage: string | undefined;

  try {
    const parsed = JSON.parse(raw) as { ... };
    ...
  } catch {}

  return { message, friendlyMessage };
}

A malformed JSON body returns { message: raw }. The only statement outside that try is readChatGptResponsesErrorTextLimited, and it's reading new Response(errorText, ...) — an in-memory string body, whose stream doesn't reject.

The control-flow shape you describe is right (the throw is inside the try whose catch is the retry catch), there's just no way to trigger it from this path.

2. The 5xx retry test is already in this PR.

You looked at openai-chatgpt-responses.test.ts, which only has 200/400/429. The test you're asking for is in the file this PR adds, openai-chatgpt-responses.retry.test.ts:75:

  it("still retries retryable ChatGPT responses", async () => {
    const fetchMock = vi
      .fn<typeof fetch>()
      .mockResolvedValueOnce(new Response("overloaded", { status: 503 }))
      .mockResolvedValueOnce(
        new Response(JSON.stringify({ error: { message: "Invalid credentials" } }), {
          status: 401,
        }),
      );
    ...
    expect(fetchMock).toHaveBeenCalledTimes(2);
  });

That's exactly the "new re-throw branch didn't widen" regression guard — a 503 still retries, then the following 401 stops it at 2 calls.

One real instance of the shape you're pointing at, but it predates this PR.

Two lines above parseErrorResponse:

          const fakeResponse = new Response(errorText, {
            status: response.status,
            ...

new Response(body, { status }) throws TypeError for null-body statuses (204/205/304). A 304 is !response.ok and non-retryable, so it reaches this line, throws before NonRetryableCodexResponseError is constructed, and gets retried as a network error — the exact mis-classification you describe, just at a different line. It's on main today and independent of this change, so I'd rather not pull it into this diff. Happy to file it separately if a maintainer wants it.

@steipete steipete self-assigned this Jul 18, 2026
@steipete
steipete merged commit 19d8871 into openclaw:main Jul 18, 2026
100 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

@Yigtwxx
Yigtwxx deleted the fix/chatgpt-responses-non-retryable-retry branch July 18, 2026 22:13
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 19, 2026
…ble (openclaw#110655)

* fix(ai): ChatGPT Responses retries errors it classified as non-retryable

The retry loop builds its friendly error and throws it from inside the same
try block whose catch treats unknown failures as retryable network errors.
Authentication and bad-request failures were therefore resent up to
maxRetries times, adding 7s of backoff before surfacing the same message.

Tag the already-classified failure with a private error type and rethrow it
unchanged from the catch, leaving the retryable paths untouched.

* test(ai): move retry classification cases to a dedicated file

The provider test file was already near the 1000-line oxlint max-lines
budget, so the new cases pushed it over. Split them out following the
existing <module>.<topic>.test.ts convention.

* refactor(ai): separate HTTP and transport retries

Co-authored-by: Yigtwxx <[email protected]>

* test(ai): clarify synthetic JWT fixture

Co-authored-by: Yigtwxx <[email protected]>

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants