What Problem This Solves

Fixes #95519. When a primary model's provider returns an error with type: "upstream_error" (the OpenAI-compatible "the upstream backend failed" shape, e.g. {"error":{"message":"Upstream request failed","type":"upstream_error","param":"","code":null}}), OpenClaw ends the turn with LLM request failed instead of trying the configured agents.defaults.model.fallbacks. The trajectory shows only the primary model, no model.fallback_step event, and no fallback attempt.

Root cause is a classification gap. The provider error surfaces as an assistant message with stopReason: "error" and errorType: "upstream_error". The embedded runner's assistant-stage failover gate (run.ts calls isFailoverAssistantError / classifyAssistantFailoverReason) is what decides whether to throw a FailoverError into the model-fallback loop. classifyAssistantFailoverReason runs classifyFailoverSignal, and nothing mapped upstream_error to a failover reason:

• upstream_error is lowercase snake_case, so it was not recognized as a transient provider type, and
• the message text Upstream request failed matches no transient classifier, and
• there is no recognized HTTP status in this payload to fall back on.

So the gate returned null ("not a failover error"), the assistant stage took the continue_normal path instead of throwing, and the run returned a terminal error payload (non_deliverable_terminal_turn) that the model-fallback loop treats as a successful result — the fallback chain never runs.

Why This Change Was Made

The fix classifies upstream_error as a transient, fallbackable reason at the point the signal is built, so the existing assistant-stage failover machinery throws a FailoverError and the model-fallback loop advances to the next configured model — the same path already used for Codex server_error (run.codex-server-error-fallback.test.ts).

Three changes in src/agents/embedded-agent-helpers/errors.ts and src/agents/failover-error.ts:

1. New classifyFailoverReasonFromErrorType in errors.ts, run early in classifyFailoverSignal (before message/status classification): upstream_error → timeout, server_error → server_error; unknown types return null and fall through. This is the change that fixes [Bug]: Fallback should trigger on provider upstream_error / LLM request failed #95519, because buildAssistantFailoverSignal passes the assistant message's errorType straight into classifyFailoverSignal, so the gate now recognizes upstream_error.
2. isStableProviderErrorType in failover-error.ts now also accepts lowercase_snake_case (in addition to SCREAMING_SNAKE_CASE). This only affects readDirectErrorType, used by the thrown-error coercion path (resolveFailoverReasonFromError), so a thrown upstream_error payload is also extractable. The generic-type exclusion list (api_error, server_error, etc.) is unchanged.
3. API_ERROR_TRANSIENT_SIGNALS_RE matches upstream.error, so both upstream error and upstream_error are caught in the message-text path used for api_error-typed payloads (defense-in-depth).

Non-transient types (auth, context overflow, validation) are unaffected and continue to return null.

User Impact

With agents.defaults.model.fallbacks configured, a primary-model upstream_error is now treated as a fallbackable failure: the run throws a FailoverError, the model-fallback loop skips the failed candidate and tries the next model, and the trajectory records a fallback step. If all candidates fail, the user gets a FallbackSummaryError with per-attempt detail instead of the generic LLM request failed. No behavior change when no fallback is configured, and no change for non-transient errors.

Evidence

Focused regression coverage in src/agents/failover-error.test.ts, including a test that drives the exact production gate the runtime uses (classifyAssistantFailoverReason / isFailoverAssistantError) with the #95519 assistant-message shape (stopReason: "error", errorType: "upstream_error", message Upstream request failed, no HTTP status). This message returns null from the gate without the fix (verified: message-only classifyFailoverSignal is null) and timeout with it, so it is a true regression test of the runtime path — not just an isolated signal-classifier call.

Command:

pnpm test src/agents/failover-error.test.ts

Result:

 Test Files  1 passed (1)
      Tests  97 passed (97)
   Duration  565ms

New/relevant cases:

✓ upstream_error type triggers fallback > classifies upstream_error errorType as timeout (fallbackable)
✓ upstream_error type triggers fallback > classifies server_error errorType as server_error (fallbackable)
✓ upstream_error type triggers fallback > resolves upstream_error via resolveFailoverReasonFromError with nested payload
✓ upstream_error type triggers fallback > does not classify unknown errorType as fallbackable
✓ upstream_error type triggers fallback > classifies upstream_error via JSON api_error transient signals (regex)
✓ upstream_error type triggers fallback > treats an upstream_error assistant message as fallbackable at the production gate

No-regression check on the adjacent failover surfaces the change touches:

pnpm test src/agents/embedded-agent-runner/result-fallback-classifier.test.ts src/agents/embedded-agent-helpers/failover-matches.test.ts
 Test Files  2 passed (2)
      Tests  47 passed (47)

The transport layer already extracts a nested provider type into errorType (proven for the identical code path by the rate_limit_error case in src/agents/openai-transport-stream.test.ts), which is how errorType: "upstream_error" reaches the gate above.