Skip to content

[Bug]: Model fallback chain doesn't reach cross-provider fallbacks when provider-wide cooldown is active during session override #26680

Description

@gbballpack

Summary

When a session model override is active and the Anthropic provider enters rate-limit cooldown, the fallback chain terminates after trying only same-provider models and never attempts configured cross-provider fallbacks.

Steps to reproduce

  1. Set agents.defaults.model.primary to anthropic/claude-haiku-4-5 with fallbacks: ["openai/gpt-5.1-codex"]
  2. Confirm openai/gpt-5.1-codex works by manually switching to it
  3. Set a session model override to anthropic/claude-sonnet-4-6 via session_status(model="sonnet")
  4. Trigger Anthropic provider-wide rate-limit cooldown (e.g., heavy concurrent usage across agents)
  5. Send a message during cooldown and observe the error

Expected behavior

The fallback candidate chain should be: session override → primary → fallbacks, producing 3 candidates: anthropic/claude-sonnet-4-6anthropic/claude-haiku-4-5openai/gpt-5.1-codex. When both Anthropic models are skipped due to provider cooldown, openai/gpt-5.1-codex should be attempted and succeed.

Actual behavior

Error shows only 2 models attempted — the cross-provider fallback was never tried:

All models failed (2): anthropic/claude-sonnet-4-6: Provider anthropic is in cooldown
(all profiles unavailable) (rate_limit) | anthropic/claude-haiku-4-5: Provider anthropic
is in cooldown (all profiles unavailable) (rate_limit)

OpenClaw version

2026.2.25 (81ce45a)

Operating system

Ubuntu 24.04 host

Install method

docker

Logs, screenshots, and evidence

**Error output during cooldown with session override active:**


All models failed (2): anthropic/claude-sonnet-4-6: Provider anthropic is in cooldown
(all profiles unavailable) (rate_limit) | anthropic/claude-haiku-4-5: Provider anthropic
is in cooldown (all profiles unavailable) (rate_limit)


**Config (redacted):**


"model": {
    "primary": "anthropic/claude-haiku-4-5",
    "fallbacks": ["openai/gpt-5.1-codex"]
}


**Verification that OpenAI fallback is functional:**

Manually switching to `openai/gpt-5.1-codex` works correctly — the issue is not an auth or config problem.

Impact and severity

  • Affected users/systems/channels: Any user with cross-provider fallbacks configured who also uses session model overrides. Likely affects all channels (Telegram, Discord, etc.).
  • Severity: Blocks workflow. The agent becomes completely unresponsive during provider cooldowns despite having a healthy fallback provider configured.
  • Frequency: Deterministic — occurs every time the override model's provider enters cooldown.
  • Consequence: Missed messages, agent downtime, and the configured fallback chain provides zero resilience. Users must manually intervene (gateway restart or config change) to recover. This defeats the purpose of configuring cross-provider fallbacks.

Additional information

Root cause hypothesis: The runWithModelFallback candidate list builder (in src/agents/pi-embedded-runner/run.ts) appears to construct only [session_override, primary] when a session override is present, omitting the fallbacks array. The (2) in the error (rather than (3)) is the key evidence — if fallbacks were in the list, they would appear even if skipped.

An alternative hypothesis is that the candidate loop exits early when all candidates so far share a provider that is in cooldown, rather than continuing to check remaining candidates from other providers.

Related issues (same systemic pattern):

Suggested fix direction:

  1. Ensure the candidate list always includes configured fallbacks regardless of whether a session override is active
  2. Implement per-model cooldown tracking — a rate limit on one model should not block models from other providers
  3. Never exit the candidate loop early on provider cooldown — always iterate through all candidates

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions