Skip to content

fix(agents): fallback on embedded upstream errors#96260

Closed
harjothkhara wants to merge 1 commit into
openclaw:mainfrom
harjothkhara:codex/95519-upstream-error-fallback
Closed

fix(agents): fallback on embedded upstream errors#96260
harjothkhara wants to merge 1 commit into
openclaw:mainfrom
harjothkhara:codex/95519-upstream-error-fallback

Conversation

@harjothkhara

Copy link
Copy Markdown
Contributor

Closes #95519

What Problem This Solves

Embedded agent runs could end immediately on provider-side transient failures even when agents.defaults.model.fallbacks was configured. In particular, structured provider payloads such as {"type":"upstream_error","message":"Upstream request failed"} were surfaced as LLM request failed instead of advancing to the next configured fallback model.

Why This Change Was Made

The embedded result fallback classifier already delegates provider payload text to the shared failover matcher, but it only accepted auth, billing, and rate-limit reasons. This patch lets the embedded path also treat shared transient provider reasons (overloaded, server_error, and timeout) as fallback-safe, while preserving the existing guards for visible output, hook blocks, replay-invalid state, committed delivery evidence, and unclassified application errors.

The shared matcher also now recognizes structured server_error / upstream_error JSON fields even when the JSON has spaces around : rather than the exact compact string.

User Impact

Users with configured model fallbacks should now get the next fallback candidate for transient upstream provider failures instead of a failed turn after the primary provider returns an upstream error.

Evidence

Behavior addressed: Embedded provider error payloads with upstream_error, HTTP 500 server errors, or existing shared transient failover classifications now produce embedded_error_payload fallback classifications so runWithModelFallback can advance to the next configured candidate.

Real environment tested: Local OpenClaw checkout on macOS, Node 24.16.0, using the repo vitest wrapper in this Codex worktree.

Exact steps run after the patch:

PATH=/Users/harjothk/.nvm/versions/node/v24.16.0/bin:$PATH node scripts/run-vitest.mjs src/agents/embedded-agent-runner/result-fallback-classifier.test.ts src/agents/embedded-agent-helpers/failover-matches.test.ts
PATH=/Users/harjothk/.nvm/versions/node/v24.16.0/bin:$PATH node scripts/run-vitest.mjs src/agents/outcome-fallback-runtime-contract.test.ts src/agents/model-fallback.test.ts
PATH=/Users/harjothk/.nvm/versions/node/v24.16.0/bin:$PATH node_modules/.bin/oxfmt --check src/agents/embedded-agent-helpers/errors.ts src/agents/embedded-agent-helpers/failover-matches.test.ts src/agents/embedded-agent-runner/result-fallback-classifier.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.ts
PATH=/Users/harjothk/.nvm/versions/node/v24.16.0/bin:$PATH node scripts/run-oxlint.mjs src/agents/embedded-agent-helpers/errors.ts src/agents/embedded-agent-helpers/failover-matches.test.ts src/agents/embedded-agent-runner/result-fallback-classifier.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.ts
git diff --check
PATH=/Users/harjothk/.nvm/versions/node/v24.16.0/bin:/Users/harjothk/.cache/codex-runtimes/codex-primary-runtime/dependencies/bin:$PATH /Users/harjothk/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --prompt "Review scope: fix openclaw/openclaw#95519. This branch should only make embedded-agent provider error payloads with shared transient failover classifications (upstream_error/server_error/timeout/overloaded) trigger configured model fallback, while preserving guards for visible output, hook blocks, replay/side-effect safety, and unclassified application errors."

Evidence after fix:

  • result-fallback-classifier.test.ts + failover-matches.test.ts: 49 tests passed.
  • outcome-fallback-runtime-contract.test.ts + model-fallback.test.ts: 92 tests passed.
  • oxfmt --check: all matched files use the correct format.
  • run-oxlint.mjs: passed for the touched files.
  • git diff --check: passed.
  • autoreview: clean, no accepted/actionable findings reported.

Observed result: Structured upstream_error payloads classify as server_error, embedded provider payloads classify as embedded_error_payload, and existing fallback-loop tests continue to pass.

What was not tested: Live gateway/provider run against an actual failing upstream service. The issue is covered with source-level classifier proof and fallback-loop unit coverage.

AI-assisted: yes. This PR was implemented with Codex assistance; I reviewed the code paths, focused tests, and autoreview result before opening it.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 24, 2026
@harjothkhara

Copy link
Copy Markdown
Contributor Author

Closing as a duplicate — standing down in favor of the existing open fixes for #95519.

Patch-level comparison of the open PRs:

This PR adds nothing those don't already cover, and broadening the gate further is wider than the reported bug (higher merge-risk on an auth-provider / message-delivery path), so I'm closing to avoid piling on an over-subscribed P2. I left one review finding on #95542 (its overloaded_error → overloaded matcher mapping isn't accepted at the embedded gate). Thanks all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Fallback should trigger on provider upstream_error / LLM request failed

1 participant