Skip to content

fix(agents): classify account-restricted model 400s as model_not_found#104878

Merged
altaywtf merged 2 commits into
openclaw:mainfrom
ObliviateRickLin:fix/codex-account-model-unsupported-104490
Jul 13, 2026
Merged

fix(agents): classify account-restricted model 400s as model_not_found#104878
altaywtf merged 2 commits into
openclaw:mainfrom
ObliviateRickLin:fix/codex-account-model-unsupported-104490

Conversation

@ObliviateRickLin

@ObliviateRickLin ObliviateRickLin commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

When an embedded Codex run fails because the configured model is not available on the account/plan (HTTP 400 invalid_request_error: "The 'gpt-5.5-pro' model is not supported when using Codex with a ChatGPT account."), Telegram users get the generic "Something went wrong… try again, or use /new" copy (#104490). Retrying or /new cannot fix a model/account mismatch — the actionable cause was only visible in gateway logs.

Why This Change Was Made

Root cause is a classification gap, not missing copy. The pipeline for this failure already exists end to end:

  1. The ambiguous-400 branch in classifyFailoverSignal (src/agents/embedded-agent-helpers/errors.ts) explicitly inspects the payload first "so … model-not-found errors … are not collapsed into generic format failures" — but isModelNotFoundErrorMessage (src/agents/live-model-errors.ts) had no pattern for the account-restricted rejection sentence, so the error fell through to format.
  2. With a model_not_found reason, the assistant-failover layer already does the right things: tries configured fallback models, and when surfacing throws a typed FailoverError.
  3. classifyProviderRequestError already maps that typed reason to PROVIDER_MODEL_UNAVAILABLE_USER_MESSAGE — copy that matches the issue's expectation ("needs a config update…; retrying or starting a new session won't fix it").

The fix is one pattern in the classifier that owns model-error text: model … is not supported when using …. The "when using" qualifier keeps capability rejections ("This model is not supported for tool calling.") out of the class — the existing negative tests still pass. No change to classifyProviderRequestError: its "typed reason only, no text matching" contract from #97611 is preserved (the message-level classification happens in the failover layer that owns it).

The issue's admin-notification request (route sanitized diagnostics to an operator) is a separate product surface and intentionally not part of this fix.

User Impact

  • Telegram (and every channel using the shared failure-reply path) now shows: "⚠️ The configured model is unavailable from the provider — it may have been renamed, retired, or is not offered on this account. This needs a config update (agents.defaults.model); retrying or starting a new session won't fix it." instead of the misleading retry//new suggestion.
  • With fallback models configured, the run now fails over to the next model instead of dying on a format-classified error.

Evidence

Executed real-module before/after (tsx over the actual source chain — failover classification → surfaced FailoverError → user-copy classification — using the exact payload from the issue's logs):

Before (main):

stage1 failover reason: "format"
stage2 classification code: (none)
USER SEES: ⚠️ Something went wrong while processing your request. Please try again, or use /new to start a fresh session.

After (this PR):

stage1 failover reason: "model_not_found"
stage2 classification code: provider_model_unavailable
USER SEES: ⚠️ The configured model is unavailable from the provider — it may have been renamed, retired, or is not offered on this account. This needs a config update (agents.defaults.model); retrying or starting a new session won't fix it.

Both runs include the capability-rejection guard: "This model is not supported for tool calling." stays out of model_not_found before and after.

Regression tests (node scripts/run-vitest.mjs run src/agents/live-model-errors.test.ts src/agents/failover-error.test.ts src/auto-reply/reply/provider-request-error-classifier.test.ts → 122/122):

  • live-model-errors.test.ts: the issue sentence and its raw JSON envelope both classify as model-not-found; existing negatives (capability/feature rejections) unchanged.
  • failover-error.test.ts: status: 400 + the Codex payload resolves to model_not_found; capability 400s do not.
  • Reverting only live-model-errors.ts fails both new tests (2 failed | 97 passed), confirming they gate the fix.

Codex-side provenance: the sibling openai/codex checkout confirms this rejection is generated by the OpenAI backend and forwarded verbatim (codex-rs/core/tests/suite/compact.rs mocks it as an HTTP 400 invalid_request_error; codex-rs/protocol/src/error.rs UnexpectedResponseError carries the body through), so message-level classification on the OpenClaw side is the right seam.

Fixes #104490

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Jul 12, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P0 Emergency: data loss, security bypass, crash loop, or unusable core runtime. labels Jul 12, 2026
@clawsweeper

clawsweeper Bot commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 12, 2026, 1:30 AM ET / 05:30 UTC.

Summary
The PR recognizes account-restricted unsupported-model responses as model_not_found and adds classifier and failover regression coverage.

PR surface: Source +8, Tests +41. Total +49 across 3 files.

Reproducibility: yes. Codex preserves the exact account-restriction response body, and current main's classifier demonstrably sends that unmatched HTTP 400 through the generic format-failure path.

Review metrics: none identified.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #104490
Summary: The linked issue is canonical; this PR supplies its focused raw-response classification fix, while the earlier merged PR provides the downstream typed reply behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Mantis proof suggestion
A Telegram Desktop capture would add direct channel-visible confirmation of the actionable replacement response, although the existing terminal evidence is already sufficient. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram desktop proof: reproduce the account-restricted model failure and capture the actionable model-unavailable reply replacing retry or /new guidance, with private details redacted.

Next step before merge

  • A maintainer should review and land the exact head through normal merge gates; no automated repair is needed.

Security
Cleared: The focused classifier and regression-test changes introduce no concrete security or supply-chain concern.

Review details

Best possible solution:

Merge the focused shared-classifier match with its positive and negative regression cases, preserving the existing typed failover, fallback, and user-reply layers.

Do we have a high-confidence way to reproduce the issue?

Yes. Codex preserves the exact account-restriction response body, and current main's classifier demonstrably sends that unmatched HTTP 400 through the generic format-failure path.

Is this the best way to solve the issue?

Yes. The shared failover message classifier is the narrowest established owner, and the PR reuses the existing typed fallback and user-message pipeline rather than duplicating channel-specific behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f696430fd8b4.

Label changes

Label justifications:

  • P0: An account/provider setup mismatch leaves a non-technical chat user unable to complete an agent turn, while recovery requires an operator to inspect logs and change model configuration.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR provides terminal before/after output through the real failover-to-user-copy chain and confirms the capability-rejection negative case remains excluded.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR provides terminal before/after output through the real failover-to-user-copy chain and confirms the capability-rejection negative case remains excluded.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The corrected recovery response is directly visible in Telegram chat and can be demonstrated in a short native recording.
Evidence reviewed

PR surface:

Source +8, Tests +41. Total +49 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 8 0 +8
Tests 2 41 0 +41
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 49 0 +49

What I checked:

Likely related people:

  • RomneyDa: Authored and merged the typed model-unavailable reply and runtime-model recovery hint that this classifier fix completes. (role: introduced adjacent behavior; confidence: high; commits: a436ebf8af7f, 586912ae1b85; files: src/auto-reply/reply/provider-request-error-classifier.ts, src/agents/embedded-agent-runner/model.ts)
  • Vincent Koc: Introduced the current extracted live-model-errors.ts classifier module on main and most recently maintained this exact source surface. (role: recent area contributor; confidence: high; commits: ddbec923d4ee; files: src/agents/live-model-errors.ts)
  • steipete: Git history shows earlier work on live model-not-found handling and the central agent error-classification paths. (role: historical area contributor; confidence: medium; commits: f66cc886d372; files: src/agents/live-model-errors.ts, src/agents/embedded-agent-helpers/errors.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-07-12T02:03:14.362Z sha 052b45f :: needs maintainer review before merge. :: none
  • reviewed 2026-07-12T04:05:00.247Z sha 6933e9d :: needs maintainer review before merge. :: none

@ObliviateRickLin
ObliviateRickLin force-pushed the fix/codex-account-model-unsupported-104490 branch from 052b45f to 6933e9d Compare July 12, 2026 03:50
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 12, 2026
@ObliviateRickLin
fix(agents): classify account-restricted model 400s as model_not_found
7abbb6f 
OpenAI-backed runtimes reject plan/account-restricted models with HTTP
400 invalid_request_error ("The '<model>' model is not supported when
using Codex with a ChatGPT account."). Without a model_not_found match
the ambiguous-400 branch collapses this into a format failure, so users
get the generic retry//new copy for a config-only problem (openclaw#104490).
The 'when using' qualifier keeps capability rejections ("not supported
for tool calling") out of the class, preserving the openclaw#97611 contract.

Fixes openclaw#104490
@ObliviateRickLin
ObliviateRickLin force-pushed the fix/codex-account-model-unsupported-104490 branch from 6933e9d to 7abbb6f Compare July 12, 2026 05:16
@altaywtf altaywtf self-assigned this Jul 13, 2026
@altaywtf

altaywtf commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Live verification refreshed after the maintainer review and preparation pass on exact PR head 25bf4390497792861870dc65a89118556c0db7b6.

The review found one important false-positive boundary in the original pattern: This model is not supported when using tool calling. would also have been classified as an unavailable model. The prepared follow-up narrows the match to the observed account/runtime form by requiring the with a ChatGPT account suffix and adds regression coverage at both the direct classifier and failover boundary.

Personal Cloudflare Crabbox proof on the byte-identical prepared tree:

  • Exact package, lockfile, and changed-file hashes matched the prepared tree.
  • oxfmt --check passed all 3 changed files.
  • Focused Vitest passed 3 files, 122 tests, 0 failures.
  • Production-shaped positive control still reaches model_not_found and the existing provider_model_unavailable configuration guidance.
  • Same-wording negative control (This model is not supported when using tool calling.) remains outside model_not_found at both tested boundaries.
  • No provider credentials were required or forwarded.

The guarded push verified that the remote PR head tree exactly matches the Cloudflare-validated local tree, and the PR is mergeable against current main without conflicts.

Hosted-check note: 65 checks have passed. Two non-required shards failed outside this PR's three-file agent-classifier scope: the plugin-SDK surface budget observed 210 wildcard re-exports against its pinned 209 baseline, and an unrelated TUI PTY E2E timed out after its mock provider response returned. Four platform scans remain queued/running. I did not blindly rerun the deterministic plugin-SDK failure.

Review finding resolved and preparation complete. /merge-pr should adjudicate the two unrelated hosted failures and the remaining platform scans.

@altaywtf
altaywtf merged commit 21c3778 into openclaw:main Jul 13, 2026
109 of 113 checks passed
@altaywtf

altaywtf commented Jul 13, 2026

Copy link
Copy Markdown
Member

Merged via squash.

wm0018 pushed a commit to wm0018/openclaw that referenced this pull request Jul 14, 2026
@ObliviateRickLin @altaywtf @wm0018
fix(agents): classify account-restricted model 400s as model_not_found (
5f6269b 
openclaw#104878)

* fix(agents): classify account-restricted model 400s as model_not_found

OpenAI-backed runtimes reject plan/account-restricted models with HTTP
400 invalid_request_error ("The '<model>' model is not supported when
using Codex with a ChatGPT account."). Without a model_not_found match
the ambiguous-400 branch collapses this into a format failure, so users
get the generic retry//new copy for a config-only problem (openclaw#104490).
The 'when using' qualifier keeps capability rejections ("not supported
for tool calling") out of the class, preserving the openclaw#97611 contract.

Fixes openclaw#104490

* fix(agents): narrow account model rejection match

---------

Co-authored-by: Altay <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 14, 2026
@ObliviateRickLin @altaywtf
fix(agents): classify account-restricted model 400s as model_not_found (
7020984 
openclaw#104878)

* fix(agents): classify account-restricted model 400s as model_not_found

OpenAI-backed runtimes reject plan/account-restricted models with HTTP
400 invalid_request_error ("The '<model>' model is not supported when
using Codex with a ChatGPT account."). Without a model_not_found match
the ambiguous-400 branch collapses this into a format failure, so users
get the generic retry//new copy for a config-only problem (openclaw#104490).
The 'when using' qualifier keeps capability rejections ("not supported
for tool calling") out of the class, preserving the openclaw#97611 contract.

Fixes openclaw#104490

* fix(agents): narrow account model rejection match

---------

Co-authored-by: Altay <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@altaywtf altaywtf

Labels

agents Agent runtime and tooling mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P0 Emergency: data loss, security bypass, crash loop, or unusable core runtime. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Generic Telegram failure suggests /new for non-user-recoverable model/config errors

2 participants

@ObliviateRickLin @altaywtf