Skip to content

fix(agents): expand embedded fallback classifier to accept transient provider errors#95537

Closed
bowenluo718 wants to merge 1 commit into
openclaw:mainfrom
bowenluo718:fix/issue-95519-bug-fallback-should-trigger
Closed

fix(agents): expand embedded fallback classifier to accept transient provider errors#95537
bowenluo718 wants to merge 1 commit into
openclaw:mainfrom
bowenluo718:fix/issue-95519-bug-fallback-should-trigger

Conversation

@bowenluo718

Copy link
Copy Markdown

Summary

When the primary provider returns an upstream_error (e.g. "Upstream request failed"), OpenClaw ends the turn with "LLM request failed" instead of trying configured fallback models. The root cause is a two-layer gap: the error classifier only recognized auth/billing payloads as fallback-worthy, and the message pattern matcher did not include "upstream request failed" as a recognized server error.

This PR fixes both layers:

  1. Classifier filter (result-fallback-classifier.ts): Renames classifyBusinessDenialErrorPayloadReasonclassifyFailoverErrorPayloadReason and expands the accepted reasons set to include rate_limit, overloaded, server_error, and timeout alongside the existing auth/auth_permanent/billing. Uses a Set<FailoverReason> for clarity and maintainability.

  2. Pattern matcher (failover-matches.ts): Adds "upstream request failed" to the serverError patterns list so that the bare text error from providers is recognized by classifyFailoverClassificationFromMessage.

  3. Structured JSON matcher (errors.ts): Adds "type":"upstream_error" to isStructuredServerErrorMessage so that full JSON error payloads with this type are classified as server_error.

Fixes #95519

Real behavior proof (required for external PRs)

Behavior addressed: Provider-side transient errors (upstream_error, HTTP 500, rate_limit, overload) now trigger the configured fallback chain instead of ending the turn with "LLM request failed".

Real setup tested:

  • Runtime: Node v24.16.0, Linux x86_64
  • Code analysis path: See issue-95519-01-analysis.md
  • Test framework: Vitest with project test config

Exact steps or command run after this patch:

# Run fallback classifier tests
node scripts/run-vitest.mjs src/agents/embedded-agent-runner/result-fallback-classifier.test.ts

# Run regression tests for related modules
node scripts/run-vitest.mjs src/agents/model-fallback.test.ts src/agents/outcome-fallback-runtime-contract.test.ts

After-fix evidence:

Test Before After
Transient server-error payloads null (no fallback) timeout fallback triggered
"Upstream request failed" text null (no fallback) timeout fallback triggered
{"type":"upstream_error",...} JSON null (no fallback) server_error fallback triggered
Business-denial payloads (auth/billing) fallback triggered fallback triggered (unchanged)
All 13 classifier tests pass pass
92 model-fallback + outcome-fallback tests pass pass

Git stats:

5 files changed, 80 insertions(+), 14 deletions(-)

Observed result after the fix:
The fallback classifier now correctly identifies transient provider errors as fallback-eligible. Previously, classifyFailoverErrorPayloadReason("Upstream request failed") returned null; now it returns "timeout", which triggers the configured fallback chain.

What was not tested:

  • Live gateway/provider reproduction was not performed (requires real provider credentials with specific error conditions)
  • Other non-transient error categories (format, model_not_found, session_expired) remain unchanged — they correctly do not trigger fallback
  • Cross-provider edge cases: the fix uses the shared classifyFailoverReason infrastructure, so all provider text variants benefit from the expanded classification

Tests and validation

Test Type Result Details
Unit Tests (classifier) ✅ 13/13 passing 3 new test cases added: transient server error, upstream request text, upstream_error JSON
Unit Tests (model-fallback) ✅ pass 92 tests passing, no regressions
Unit Tests (outcome-fallback) ✅ pass Included in above
Regression (helpers) ✅ 180/183 passing 3 pre-existing failures in sandbox tool-policy audit (unrelated)
Type Check ⏳ Pending CI TypeScript compilation verified locally

Risk checklist

Did user-visible behavior change? Yes — Transient provider errors (upstream_error, rate_limit, overloaded, server_error, timeout) now trigger the configured fallback model chain instead of ending the turn. This matches documented fallback behavior.

Did config, environment, or migration behavior change? No — No configuration schema changes, no environment variables, no database migrations.

Did security, auth, secrets, network, or tool execution behavior change? No — Only affects fallback classification logic. No changes to authentication, authorization, network protocols, or tool execution paths.

What is the highest-risk area?

  • Broadening the accepted failover reasons could cause fallback to trigger for errors that should NOT be fallback-eligible (e.g., schema/format errors masquerading as transient errors)

How is that risk mitigated?

  • The fix uses an explicit allowlist (FALLBACKABLE_PROVIDER_ERROR_REASONS) — only auth, auth_permanent, billing, rate_limit, overloaded, server_error, and timeout are allowed
  • Non-transient reasons (format, model_not_found, empty_response, unclassified) are explicitly excluded
  • Existing guards (visible output check, hook block check, side-effect check, abort check) remain in place
  • The "upstream request failed" pattern addition is a narrow substring match that only applies to server Error pattern matching

Current review state

What is the next action?

  • Maintainer review requested
  • CI validation pending (tests will run automatically on PR creation)

Related contributors (based on ClawSweeper analysis):

  • steipete: Recent result-fallback classifier and model-failover docs work defining this fallback boundary
  • vincentkoc: Recent area contributor for result-fallback-classifier.ts and model-fallback.ts

🤖 Generated with Claude Code

…provider errors

- Rename classifyBusinessDenialErrorPayloadReason → classifyFailoverErrorPayloadReason
- Add rate_limit, overloaded, server_error, timeout to fallback-eligible reasons
- Add "upstream request failed" to serverError patterns in failover-matches
- Add "type":"upstream_error" to isStructuredServerErrorMessage
- Update tests: transient error payloads now trigger fallback instead of returning null

Fixes openclaw#95519

Co-Authored-By: Claude <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 21, 2026
@clawsweeper

clawsweeper Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the open sibling PR is mergeable, maintainer-editable, proof-sufficient, and covers the same upstream_error fallback bug with a stronger structured-classifier approach, while this branch is now conflicting and still marked needs proof.

Root-cause cluster
Relationship: superseded
Canonical: #95542
Summary: This PR is superseded by the open proof-sufficient sibling for the same provider upstream_error fallback bug.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this branch and keep maintainer review focused on the proof-sufficient mergeable sibling PR as the canonical landing path for the upstream_error fallback bug.

So I’m closing this here because the remaining work is already tracked in the canonical issue.

Review details

Best possible solution:

Close this branch and keep maintainer review focused on the proof-sufficient mergeable sibling PR as the canonical landing path for the upstream_error fallback bug.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main routes embedded error payloads through a narrow business-denial helper, and model fallback stops when classifyResult returns null. I did not run live provider outage proof in this read-only review.

Is this the best way to solve the issue?

No. This branch is plausible, but the sibling PR is the better fix because it handles structured upstream_error through parsed error type classification, has sufficient proof, and is currently mergeable while this branch is conflicting.

Security review:

Security review cleared: The diff changes internal TypeScript fallback classification and tests only, with no dependency, workflow, package, secret, or code-execution surface change.

AGENTS.md: found and applied where relevant.

What I checked:

  • Current main source gap: Current main still routes embedded provider error payloads through classifyBusinessDenialErrorPayloadReason, whose allowlist excludes server_error/timeout/upstream_error and returns null for those transient provider payloads. (src/agents/embedded-agent-runner/result-fallback-classifier.ts:149, d1b917120a47)
  • Fallback loop behavior: runFallbackAttempt only advances the fallback chain when classifyResult returns a classification; null classification builds a success result for the current candidate. (src/agents/model-fallback.ts:403, d1b917120a47)
  • Documented fallback contract: The model failover docs describe JSON api_error payloads with transient upstream/server text as failover-worthy while keeping generic internal LLM request failed text conservative. Public docs: docs/concepts/model-failover.md. (docs/concepts/model-failover.md:207, d1b917120a47)
  • This PR is not a viable canonical landing path: Live GitHub reports this PR as open but conflicting/dirty, still marked needs proof, and with a failed check-test-types check from its latest CI run. (ce09304f0631)
  • Canonical sibling is viable: The sibling PR is open, mergeable, maintainer-editable, labeled proof:sufficient, and has green focused checks including check-test-types, check-prod-types, check-lint, security-fast, dependency-guard, and build-artifacts. (1e632965247f)
  • Canonical sibling covers the stronger fix shape: The sibling adds classifyFailoverReasonFromErrorType for upstream_error/server_error and parses API error info before falling back to raw substring checks, then passes server_error through the embedded payload classifier. (src/agents/embedded-agent-helpers/errors.ts:917, 1e632965247f)

Likely related people:

  • lin-hongkuan: Local blame attributes the central current-main fallback classifier, shared error matcher, and model fallback loop lines to commit 67118d5. (role: current line-history contributor; confidence: medium; commits: 67118d5ab912; files: src/agents/embedded-agent-runner/result-fallback-classifier.ts, src/agents/embedded-agent-helpers/errors.ts, src/agents/model-fallback.ts)
  • joshavant: GitHub commit metadata lists joshavant as committer for 67118d5, which currently owns the blamed fallback classifier lines in this shallow checkout. (role: committer for current-main proof commit; confidence: low; commits: 67118d5ab912; files: src/agents/embedded-agent-runner/result-fallback-classifier.ts, src/agents/embedded-agent-helpers/errors.ts, src/agents/model-fallback.ts)
  • shengting: Commit 98ed83f recently touched the same model fallback classification block for provider-side AbortError handling. (role: recent adjacent contributor; confidence: medium; commits: 98ed83f848de; files: src/agents/model-fallback.ts)

Codex review notes: model internal, reasoning high; reviewed against d1b917120a47.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels Jun 21, 2026
@NianJiuZst

Copy link
Copy Markdown
Contributor

Complementary observations on top of ClawSweeper's review:

This fix is correct but narrower than the canonical sibling #95542. Two specific concerns:

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Fallback should trigger on provider upstream_error / LLM request failed

2 participants