-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Bug]: Provider refusals (Anthropic refusal / OpenAI content_filter) never trigger the model fallback chain — turn dies with generic 'LLM request failed.' #98976
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.ClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:needs-security-reviewClawSweeper marked this issue as needing security-sensitive review.ClawSweeper marked this issue as needing security-sensitive review.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.impact:securitySecurity boundary, credential, authz, sandbox, or sensitive-data risk.Security boundary, credential, authz, sandbox, or sensitive-data risk.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
What happened
When a provider returns a refusal (e.g. Anthropic's
refusalstop reason from its safety classifiers), the turn ends with a hardLLM request failed.error surfaced to the user. The configured model fallback chain (agents.defaults.model.fallbacks/ per-agentmodel.fallbacks) is never consulted, even though the user's prompt was not processed.From the user's perspective a refusal is indistinguishable from any other provider failure — the prompt did not get processed — so it should be treated as failover-eligible and retried on the next candidate in the chain.
Evidence (production logs, 2026-07-02)
Three failed runs in one morning on
claude-fable-5, all with fallbacks configured:Note
failoverReason: null— no failover classification ever happened. Two of the three were classifier false positives (a music track titled "Species Filter" tripping thebiocategory). Anthropic's own error copy explicitly recommends falling back to another model.Why it happens (source analysis)
refusal.src/shared/anthropic-refusal.ts(applyAnthropicRefusal, called fromsrc/agents/anthropic-transport-stream.ts,src/llm/providers/anthropic.ts, andsrc/plugin-sdk/provider-stream-shared.ts) converts it into an assistant message withstopReason: "error",errorMessage: "Anthropic refusal…", and aprovider_refusaldiagnostic.src/agents/model-fallback.ts) only walks the candidate chain when a request-level error is classified into aFailoverReason(src/agents/embedded-agent-helpers/failover-matches.ts,errors.ts). Since the request "succeeded", no classification occurs and the chain is skipped.failover-matches.tsnorfailover-policy.tsmention refusals anywhere; there are no refusal-related failover tests.The OpenAI transport has the same shape:
finish_reason: content_filteris mapped tostopReason: "error"(src/agents/openai-transport-stream.ts) without failover.Expected behavior
A provider refusal should be classified as failover-eligible (e.g.
FailoverReason: "refusal") and the run should continue down the configured fallback chain, same as rate limits / 5xx / timeouts. If all candidates refuse, surface the final refusal message.Acceptance criteria
refusalstop reason triggers the model fallback chain when fallbacks are configuredcontent_filterfinish reason does the samefailoverReasonis populated (e.g."refusal") inembedded_run_agent_enddiagnostics/logsLLM request failed.)Environment
Related: #48104 (model safety blocking authorized tasks — adjacent but about policy, not failover routing). We plan to submit a PR for this issue.