fix(router): fail over to a sibling deployment on non-retryable 4xx errors - #46003

TLDR

Problem this solves:

• A 404 or 400 from one deployment fails the whole request
• Healthy sibling deployments in the same model group are never tried
• Cooldown benches the broken deployment only after callers already saw errors

How it solves it:

• Non-retryable errors now retry on a deployment that has not refused yet
• Retries stop once every deployment has refused, so nothing repeats in place
• Single-deployment groups, retry policies and cooldown behave as before
• Guardrail rejections are never forwarded to another deployment

User Flow

Before: a developer calling a model group with one broken and one healthy deployment gets a 404 even though the group could have answered

1. The proxy admin configures gpt-failover with a deployment whose model id OpenAI no longer serves, plus a healthy gpt-5.4-mini deployment
2. The developer sends POST http://localhost:4000/v1/chat/completions with "model": "gpt-failover"
3. The request lands on the broken deployment and comes back as a 404 saying No fallback was attempted.
4. The next requests succeed only because the broken deployment is now cooling down, and the 404 comes back every time the cooldown expires

After: the same first request succeeds on the healthy deployment

1. The proxy admin configures gpt-failover with a deployment whose model id OpenAI no longer serves, plus a healthy gpt-5.4-mini deployment
2. The developer sends POST http://localhost:4000/v1/chat/completions with "model": "gpt-failover"
3. The request lands on the broken deployment, is retried on the healthy one and comes back 200 with x-litellm-attempted-retries: 1
4. The broken deployment still goes into cooldown, so later requests go straight to the healthy one

Relevant issues

Fixes #46002

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs hit the real OpenAI API through a local proxy on port 4000 running 2 workers that share one dedicated Redis, started fresh for each run with docker run -d --rm --name litellm-qa-redis -p 127.0.0.1:6391:6379 redis:7.2.1. The native router extension was rebuilt at each SHA with uv pip install -e .. The config is a copy of litellm/proxy/dev_config.yaml with the block below added. broken-leg points at a model id OpenAI answers 404 for and carries most of the weight, so the first request lands on it

model_list:
  - model_name: gpt-failover
    litellm_params:
      model: openai/gpt-5.4-mini-retired
      api_key: os.environ/OPENAI_API_KEY
      weight: 9
    model_info:
      id: broken-leg
  - model_name: gpt-failover
    litellm_params:
      model: openai/gpt-5.4-mini
      api_key: os.environ/OPENAI_API_KEY
      weight: 1
    model_info:
      id: healthy-leg

router_settings:
  num_retries: 2
  redis_host: 127.0.0.1
  redis_port: 6391

general_settings:
  coordination_redis:
    host: 127.0.0.1
    port: 6391

Before (1432e23)

1. Start the proxy

python litellm/proxy/proxy_cli.py --config qa_config.yaml --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log

The log shows two workers serving, Started server process [75866] and Started server process [75867]

2. Send a request to the group

curl -s -D - localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "gpt-failover", "messages": [{"role": "user", "content": "Say ok"}], "max_tokens": 20}'

3. The caller gets the 404 from the broken deployment

HTTP/1.1 404 Not Found
x-litellm-model-id: broken-leg
{"error":{"message":"litellm.NotFoundError: OpenAIException - The model `gpt-5.4-mini-retired` does not exist or you do not have access to it.\n\nLiteLLM: model group 'gpt-failover' failed with the error above. No fallback was attempted.","type":"invalid_request_error","param":null,"code":"404"}}

4. Redis holds deployment:broken-leg:cooldown, so only later callers are routed around it

After (5ee17bd)

1. Start the proxy the same way. The log shows Started server process [75669] and Started server process [75670]

2. Send the same request

3. The caller is failed over to the healthy deployment inside the same request

HTTP/1.1 200 OK
x-litellm-model-id: healthy-leg
x-litellm-attempted-retries: 1
x-litellm-attempted-fallbacks: 0
{"id":"chatcmpl-...","model":"gpt-failover","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Ok","role":"assistant",...}}],...}

4. Redis still holds deployment:broken-leg:cooldown, so later requests go straight to the healthy deployment

Type

🐛 Bug Fix

Changes

Upstream moved the router onto the Rust core (#50178), so the fix lives in litellm-rust/crates/router-core. should_retry in retry.rs used to refuse every non-retryable status other than 401 and 403, and every NotFoundError, so a 404 or 400 from one deployment never got a second try in its group. RetryContext now carries has_untried_deployment, and those failures are retried only while it is true

In call.rs the retry loop already pushed each refusing deployment onto skipped_ids so the next pick avoids it. It now records that id before deciding, and sets has_untried_deployment when a healthy deployment outside skipped_ids is left. Once every deployment has refused, the loop stops and returns the last error. Streams and the Python should_retry_this_error member pass false, so they behave as before

A guardrail rejection is also a 400, so the loop now returns a guardrail intervention raised by a retry as is, the same way the first attempt already did. Without that, a prompt a guardrail rejected on a retry could be forwarded to an unguarded sibling

Rust tests in retry.rs cover the 400, 404 and 422 decision with and without an untried deployment. tests/call_loop.rs covers a 400 retried on the group's other deployment, a 400 from every deployment trying each once and then taking the fallback (this replaces upstream's a_bad_request_falls_back_without_retrying, which pinned the old behavior), and a guardrail rejection on a retry not reaching the third deployment

The Python tests in tests/unit/test_router/test_router.py drive real HTTP through respx against the native core. They cover 404 and 400 failing over to a sibling, the 404 deployment still landing in cooldown, a three-deployment group failing over past two refusing deployments, a single-deployment group making exactly one call, and a request every deployment rejects making one call per deployment and raising the last error. tests/unit/test_router/core/test_retries_cooldowns.py expects a 400, context window or content policy error to try each deployment of a two-deployment group once

Mutation check on the Rust change. Never reporting an untried deployment turns 4 tests red, ignoring which deployments already refused turns 2 red, and dropping the guardrail check in the loop turns 2 red

Caveats (if any)

Medium

• Failover only happens within the retry budget
  • With num_retries: 0 the first 4xx still reaches the caller
  • Router defaults num_retries to 2 when unset
  • A group with more broken deployments than num_retries + 1 can still fail

Low

• A request every provider rejects now costs one call per deployment
  • Bounded by num_retries, and single-deployment groups still make one call
• 401 and 403 keep their older multi-deployment retry rule, unchanged here

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR