fix: run fallback models on upstream provider errors#96196
Conversation
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Close: this PR is a plausible and proof-backed candidate, but the same remaining bug is better owned by the open, mergeable, proof-positive canonical PR at #95542. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Canonical path: Close this PR and let #95542 be the canonical landing path for the linked upstream_error fallback bug. So I’m closing this here and keeping the remaining discussion on #95542. Review detailsBest possible solution: Close this PR and let #95542 be the canonical landing path for the linked upstream_error fallback bug. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main can be traced from embedded provider error payload classification to a null classifier result, which runWithModelFallback treats as a completed primary attempt instead of advancing fallback. Is this the best way to solve the issue? No, not as the best landing path: this patch is plausible, but #95542 is the stronger canonical fix because it handles structured upstream_error normalization at the shared boundary and is open, mergeable, and proof-positive. Security review: Security review cleared: Security review cleared: the diff changes internal TypeScript fallback classification and tests only, with no dependency, workflow, package, secret, or code-execution surface change. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against ae06d846faf3. |
clawsweeper
Bot
added
rating: 🧂 unranked krab
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
clawsweeper
Bot
added
proof: sufficient
|
ClawSweeper applied the proposed close for this PR.
|
1 participant
Fixes #95519.
What Problem This Solves
Fixes an issue where users with configured fallback models could still see a primary provider failure instead of trying the next model when the primary provider returned a transient upstream error payload such as
upstream_error/LLM request failed.Why This Change Was Made
Embedded agent result fallback already treats auth, billing, and rate-limit error payloads as safe fallback triggers when no user-visible output was delivered. This extends that same guarded path to transient provider failures (
overloaded,server_error, andtimeout) and teaches the shared matcher the canonicalupstream_errortoken, while leaving schema/tool-payload rejections non-retryable.User Impact
Users with model fallbacks configured should get an automatic fallback attempt for transient upstream provider failures instead of a failed turn from the first provider. Request-shape errors and already-delivered user-visible output remain unchanged.
Evidence
Real behavior proof: A local OpenClaw production-classifier/runtime proof command imported the production
classifyEmbeddedAgentRunResultForModelFallbackandrunWithModelFallbackmodules, fed the provider error payload shape from #95519, and verified the runtime advanced from the primary model to the configured fallback model. The proof used a temporary redactedOPENCLAW_STATE_DIR; no credentials or external provider calls were used.Redacted terminal output from the proof command:
{ "proof": "OpenClaw #96196 production fallback classifier and fallback runtime", "tempState": "<redacted temp OPENCLAW_STATE_DIR>", "upstreamPayloadText": "{\"error\":{\"message\":\"Upstream request failed\",\"type\":\"upstream_error\",\"param\":\"\",\"code\":null}}", "productionClassification": { "message": "openai/gpt-5.5 ended with a provider error: {\"error\":{\"message\":\"Upstream request failed\",\"type\":\"upstream_error\",\"param\":\"\",\"code\":null}}", "reason": "timeout", "code": "embedded_error_payload", "rawError": "{\"error\":{\"message\":\"Upstream request failed\",\"type\":\"upstream_error\",\"param\":\"\",\"code\":null}}" }, "schemaRejectionClassification": null, "runtime": { "outcome": "completed", "provider": "anthropic", "model": "claude-sonnet-4-6", "attempts": [ { "provider": "openai", "model": "gpt-5.5", "error": "{\"error\":{\"message\":\"Upstream request failed\",\"type\":\"upstream_error\",\"param\":\"\",\"code\":null}}", "reason": "timeout", "code": "embedded_error_payload" } ], "callSequence": [ { "provider": "openai", "model": "gpt-5.5", "isFinalFallbackAttempt": false }, { "provider": "anthropic", "model": "claude-sonnet-4-6", "isFinalFallbackAttempt": true } ], "fallbackPayloadText": "fallback model completed the turn" }, "assertions": { "upstreamErrorClassifiedAsFallback": true, "schemaRejectionNotRetried": true, "fallbackAttempted": true, "completedWithFallback": true } }Focused validation:
node scripts/run-vitest.mjs src/agents/embedded-agent-runner/result-fallback-classifier.test.tsnode scripts/run-vitest.mjs src/agents/embedded-agent-runner/result-fallback-classifier.test.ts src/agents/outcome-fallback-runtime-contract.test.ts src/agents/model-fallback.test.ts src/agents/embedded-agent-helpers.isbillingerrormessage.test.tspnpm exec oxfmt --check src/agents/embedded-agent-helpers/failover-matches.ts src/agents/embedded-agent-runner/result-fallback-classifier.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.tsnode scripts/run-oxlint.mjs src/agents/embedded-agent-helpers/failover-matches.ts src/agents/embedded-agent-runner/result-fallback-classifier.ts src/agents/embedded-agent-runner/result-fallback-classifier.test.tspnpm tsgo:core:testgit diff --checkcodex review --base origin/main-> no discrete correctness regressions foundAI-assisted contribution disclosure: I used Codex to inspect the fallback/classifier path, implement the patch, run focused validation, produce the production proof above, and prepare this PR. I reviewed the diff and validation output before submission.