Skip to content

fix: classify provider code-only failover errors#95729

Merged
steipete merged 5 commits into
openclaw:mainfrom
Pick-cat:fix-provider-code-failover-reasons
Jul 15, 2026
Merged

fix: classify provider code-only failover errors#95729
steipete merged 5 commits into
openclaw:mainfrom
Pick-cat:fix-provider-code-failover-reasons

Conversation

@Pick-cat

@Pick-cat Pick-cat commented Jun 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Problem This Solves

Structured assistant errors that only carry an error code (no HTTP status, no body message) fell through to null in the failover reason classifier and were reported as empty_response, so model failover/cooldown never ran. Provider-native codes like OpenAI SERVER_ERROR/INSUFFICIENT_QUOTA, Google UNAVAILABLE/DEADLINE_EXCEEDED/INTERNAL, and Anthropic RATE_LIMIT_ERROR/API_ERROR were not recognized.

Why This Change Was Made

Provider-native code→reason mappings now live in each provider plugin's classifyFailoverReason hook (OpenAI, the Google family covering google/google-vertex/google-antigravity, and Anthropic), keeping core classifyFailoverReasonFromCode generic (cross-provider codes only). classifyFailoverSignal already dispatches code-only structured signals to the matching provider hook, so the previous core PROVIDER_CODE_REASON_MAP was redundant duplication of plugin-owned policy that could drift from each provider as it evolves (AGENTS.md provider-boundary policy). The generic guard — message-less structured-code signals must not be reported as empty_response — stays in core where it belongs.

User Impact

Code-only provider errors correctly trigger model failover. When the primary provider returns a structured code with no message (e.g. OpenAI SERVER_ERROR, Google DEADLINE_EXCEEDED, Anthropic API_ERROR), the classifier maps it to the right failover reason and attempts a fallback model instead of treating the turn as an empty response.

Evidence

Boundary fix: removed the 13-entry PROVIDER_CODE_REASON_MAP from core src/agents/embedded-agent-helpers/errors.ts (now generic); added classifyFailoverReason hooks to extensions/openai/openai-provider.ts, extensions/google/provider-hooks.ts (shared GOOGLE_GEMINI_PROVIDER_HOOKS), and extensions/anthropic/register.runtime.ts. Provider-specific assertions moved to each plugin's test; the core test keeps the empty_response regression.

Typecheck + lint (clean):

$ pnpm tsgo:core          # exit 0
$ pnpm tsgo:extensions    # exit 0
$ node scripts/run-oxlint.mjs <touched files>   # exit 0

Core + sibling failover tests:

$ node scripts/run-vitest.mjs \
    src/agents/embedded-agent-helpers.isbillingerrormessage.test.ts \
    src/agents/embedded-agent-helpers/errors-provider-structured-signals.test.ts \
    src/agents/embedded-agent-helpers/provider-error-patterns.test.ts
 Tests  149 passed + 44 passed

Provider hook tests (each plugin owns its codes):

$ node scripts/run-vitest.mjs \
    extensions/openai/openai-provider.test.ts \
    extensions/anthropic/index.test.ts \
    extensions/google/provider-hooks.test.ts
 Tests  40 passed + 37 passed
  • openai: SERVER_ERROR→server_error, INSUFFICIENT_QUOTA→billing, API_ERROR→undefined (not OpenAI's)
  • google-family: UNAVAILABLE→overloaded, DEADLINE_EXCEEDED→timeout, INTERNAL→server_error
  • anthropic: RATE_LIMIT_ERROR→rate_limit, API_ERROR→timeout, INSUFFICIENT_QUOTA→undefined

Rebased on latest origin/main before push.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Jun 22, 2026
@Pick-cat
Pick-cat force-pushed the fix-provider-code-failover-reasons branch from a61efbd to e9974d1 Compare June 22, 2026 10:16
@Pick-cat
Pick-cat force-pushed the fix-provider-code-failover-reasons branch 3 times, most recently from 4ad2b4a to de94cb1 Compare June 22, 2026 11:49
@clawsweeper

clawsweeper Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 10, 2026, 1:50 PM ET / 17:50 UTC.

Summary
The PR adds provider-owned classification for code-only OpenAI, Google-family, and Anthropic failures, removes vendor-specific meanings from the generic core classifier, and prevents unknown structured failures from being reported as empty responses.

PR surface: Source +53, Tests +186. Total +239 across 9 files.

Reproducibility: yes. Maintainer Crabbox validation exercised a message-less Anthropic structured error through openclaw agent --local and observed the expected cross-model fallback.

Review metrics: none identified.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🦞 diamond lobster
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Rebase the PR onto current main and refresh the review against the resulting head.
  • Post redacted exact-head CLI or runtime output covering the provider-native fallback path and final Anthropic/non-owner semantics.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: Strong live output exists for the preceding head, but the external PR needs redacted runtime proof for final head 31f75d7; after adding it, update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

  • [P1] The posted end-to-end validation covers df4e8234, while the final head changes Anthropic API_ERROR semantics and cross-provider isolation; those final-head behaviors have test coverage but no posted real runtime proof.
  • [P1] The PR is behind its recorded base and cannot be directly merge-base-compared with the checked-out current main; maintainers should refresh the branch and review the resulting three-way diff before landing.

Maintainer options:

  1. Refresh head and runtime proof (recommended)
    Rebase onto current main and rerun the controlled provider-error scenario on the resulting exact head before merge.
  2. Pause until the base is comparable
    Leave the PR open without landing if rebasing materially changes the reviewed provider-runtime path or cannot produce a reviewable merge result.

Next step before merge

  • [P1] Maintainers should refresh the branch against current main and require final-head real-behavior proof; no narrow code defect remains for an automated repair worker.

Security
Cleared: The patch adds typed provider classification logic and tests without changing dependencies, permissions, secrets, workflows, package resolution, or downloaded code.

Review details

Best possible solution:

Rebase onto current main, preserve the provider-local mappings and generic core guard, then attach redacted exact-head runtime output showing provider-native fallback plus the final Anthropic server-error and non-owner behavior.

Do we have a high-confidence way to reproduce the issue?

Yes. Maintainer Crabbox validation exercised a message-less Anthropic structured error through openclaw agent --local and observed the expected cross-model fallback.

Is this the best way to solve the issue?

Yes. Provider-native meanings belong in the owning provider hooks, while core should retain only portable codes and generic structured-signal handling; the final implementation follows that boundary without duplicating policy.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 61893247ec7d.

Label changes

Label changes:

  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦞 diamond lobster.
  • remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

  • P2: Code-only provider failures can suppress expected model fallback, but the affected provider-routing path has a bounded blast radius.
  • merge-risk: 🚨 auth-provider: The changed mappings determine provider failure reasons, cooldowns, retries, and fallback-model routing.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦞 diamond lobster.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Strong live output exists for the preceding head, but the external PR needs redacted runtime proof for final head 31f75d7; after adding it, update the PR body to trigger review or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +53, Tests +186. Total +239 across 9 files.

View PR surface stats
Area Files Added Removed Net
Source 4 54 1 +53
Tests 5 187 1 +186
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 9 241 2 +239

What I checked:

  • Prior blockers resolved: The final commit removes provider-native codes from the generic core classifier, changes Anthropic API_ERROR from timeout to server error, and adds non-owner isolation coverage. (src/agents/embedded-agent-helpers/errors.ts:910, 31f75d7864f0)
  • Provider-first runtime path: The exact-head classifier consults the resolved provider hook before applying generic code and error-type fallbacks, preserving provider ownership of native descriptors. (src/agents/embedded-agent-helpers/errors.ts:1209, 31f75d7864f0)
  • Unknown structured failures remain distinguishable: Core now reserves empty_response for signals without a message, status, code, or error type; unowned structured descriptors instead remain unclassified. (src/agents/embedded-agent-helpers/errors.ts:1249, 31f75d7864f0)
  • Provider-family mappings: The final head maps OpenAI, Google-family, and Anthropic native descriptors through their existing provider hooks, with owner and non-owner regression coverage. (extensions/google/provider-hooks.ts:10, 31f75d7864f0)
  • Live proof applies to the preceding head: Maintainer validation exercised openclaw agent --local at df4e8234 with a message-less Anthropic rate_limit_error, observed reason=rate_limit, cross-model fallback, and a successful fallback response; the current head was created afterward. (df4e8234ec27)
  • Current-main refresh needed: GitHub reports the PR as mergeable but behind, and the checked-out current main has no local merge base with the PR head, so the merged result requires a refreshed review rather than assuming stale-base behavior. (61893247ec7d)

Likely related people:

  • altaywtf: Assigned to the PR, authored the final ownership and semantics correction, and performed live validation of the affected provider-failover path. (role: recent area contributor and validator; confidence: high; commits: 31f75d7864f0; files: src/agents/embedded-agent-helpers/errors.ts, extensions/anthropic/register.runtime.ts, src/agents/embedded-agent-helpers/errors-provider-structured-signals.test.ts)
  • fuller-stack-dev: Commit 89975eea introduced structured provider error signals into the hook path that this PR extends. (role: introduced structured provider signals; confidence: high; commits: 89975eea24a5; files: src/agents/embedded-agent-helpers/provider-error-patterns.ts, src/plugins/provider-runtime.ts)
  • steipete: Recent merged work substantially touched the central error classifier, provider runtime, and the affected provider implementations. (role: recent core and provider-runtime contributor; confidence: medium; commits: 5ea80c8d80fd, 062f88e3e3af, 827402243d7f; files: src/agents/embedded-agent-helpers/errors.ts, src/plugins/provider-runtime.ts, extensions/anthropic/register.runtime.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (4 earlier review cycles)
  • reviewed 2026-07-02T11:43:07.450Z sha 7925484 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-06T11:45:31.419Z sha dba3773 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-10T14:41:47.848Z sha 36f61d1 :: needs real behavior proof before merge. :: [P1] Keep vendor-native codes out of the generic classifier | [P1] Classify Anthropic API errors as server failures
  • reviewed 2026-07-10T15:16:17.775Z sha df4e823 :: needs real behavior proof before merge. :: [P1] Keep vendor-native codes out of the generic classifier | [P1] Classify Anthropic API errors as server failures

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels Jun 23, 2026
@Pick-cat

Pick-cat commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Added real behavior proof:

  1. node --import tsx production function proof: Called classifyAssistantFailoverReason and classifyProviderRuntimeFailureKind from production source, showing 9 provider-owned code-only error codes correctly classified and 3 non-owned provider guards correctly returning null.

  2. Before/after regression test proof: 14 new tests FAIL on main (errors.ts reverted), all 160 pass on the fix branch.

  3. Full Root Cause analysis with 3-layer causal chain tracing from symptom through classifyFailoverReasonFromCode -> classifyFailoverSignal -> classifyProviderRuntimeFailureKind.

  4. Compatibility matrix confirming no regression for gateway, openai-compatible, or message-based classification paths.

Real behavior proof CI check passed.

@clawsweeper

clawsweeper Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 23, 2026
@Pick-cat

Pick-cat commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Fixed P1: moved provider-native code mappings (OpenAI/Google/Anthropic) from core classifyFailoverReasonFromCode into each provider's classifyFailoverReason hook. Core now keeps only generic classification. 196 tests passed.

@clawsweeper

clawsweeper Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 24, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 24, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jul 4, 2026
Open
@altaywtf altaywtf assigned steipete and altaywtf and unassigned steipete Jul 6, 2026
@altaywtf
altaywtf force-pushed the fix-provider-code-failover-reasons branch from dba3773 to 36f61d1 Compare July 10, 2026 14:24
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 10, 2026
@altaywtf
altaywtf force-pushed the fix-provider-code-failover-reasons branch from 36f61d1 to df4e823 Compare July 10, 2026 14:57
@altaywtf

altaywtf commented Jul 10, 2026

Copy link
Copy Markdown
Member

Maintainer live validation

Validated PR head df4e8234 in Cloudflare Crabbox through the user-facing openclaw agent --local entrypoint.

Deterministic structured-error proof

  • The Anthropic transport received a message-less structured error: error.type = rate_limit_error.
  • The HTTP response was deliberately 400 so the result depended on preserving the provider-native structured signal rather than generic HTTP 429 handling.
  • Runtime decision: fallback_model, reason=rate_limit, from anthropic/claude-sonnet-4-6 to openai/gpt-5.4-mini.
  • Exactly one request reached each controlled provider endpoint.
  • The fallback completed successfully and returned the expected marker.

Genuine upstream Anthropic exhaustion proof

  • A bounded low-output request burst exhausted the real Anthropic requests/minute bucket; Anthropic reported requests-remaining: 0 and returned genuine HTTP 429 responses with error.type = rate_limit_error.
  • OpenClaw itself received two of those upstream 429s.
  • Its execution trace recorded both as same_model_rate_limit with reason=rate_limit.
  • Anthropic accepted the third internal retry after the bucket began refilling, so this run recovered on the primary provider before cross-model fallback was needed.

Result

The exact provider-structured classification path and the model-fallback path pass end to end. Separately, a real Anthropic rate-limit event is correctly surfaced and classified by the runtime. The upstream run did not independently demonstrate cross-model fallback because the primary recovered during its bounded same-model retry sequence.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 10, 2026
@steipete steipete self-assigned this Jul 15, 2026
@steipete steipete mentioned this pull request Jul 15, 2026
Merged
@steipete

steipete commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Maintainer landing proof for 23824a2840ed123986ae5bc564fbe664fefb754f:

  • Deep review: ready. Provider-native meanings stay in the owning Anthropic, Google-family, and OpenAI hooks; core retains only portable classification and the generic structured-signal guard. Official provider error contracts and SDK types were checked. Fresh full-branch autoreview found no accepted/actionable findings.
  • Focused remote proof: AWS Crabbox run run_fbe7ba9a6a9c passed 325 provider/runtime/billing/structured-signal assertions.
  • Live behavior: the real openclaw agent --local path received a message-less Anthropic rate_limit_error, made exactly one request to an OpenAI fallback, selected the fallback, and returned FALLBACK-OK. The same Testbox session completed a full build: https://github.com/openclaw/openclaw/actions/runs/29388637896
  • Exact-head aggregate CI is green against current main: https://github.com/openclaw/openclaw/actions/runs/29410139609
  • The prior check-guards ENOBUFS and macOS dead-code failures were unrelated long-lived-PR/mainline failures. Both pass on this exact head after their current-main fixes.
  • The final commit only removes the PR-local changelog entry under the release-owned changelog policy; runtime and test files are unchanged from reviewed head 4af30a2afc880ad3cba47ae9c04ca144f6bd8c0d.

No remaining proof gap found. Strong related-item search found no duplicate for these specific provider-native code mappings.

@steipete
steipete merged commit 1c2bcbb into openclaw:main Jul 15, 2026
113 checks passed
@steipete

steipete commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Jul 15, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 16, 2026
@Pick-cat @altaywtf @steipete
fix: classify provider code-only failover errors (openclaw#95729)
ebff2b7 
* fix(agents): classify provider-native failover errors

Co-authored-by: Altay <[email protected]>

* fix(agents): scope provider failover codes

* docs(changelog): credit provider failover fix

* style(anthropic): format failover classifier

* docs(changelog): defer release-owned entry

---------

Co-authored-by: Pick-cat <[email protected]>
Co-authored-by: Altay <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

@altaywtf altaywtf

Labels

agents Agent runtime and tooling extensions: anthropic extensions: google extensions: openai merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Pick-cat @altaywtf @steipete