Skip to content

Rate-limited profiles cause immediate failure instead of waiting for cooldown expiry #24158

Description

@YoungMoneyInvestments

Problem

When all auth profiles for a provider enter cooldown due to rate limiting, runWithModelFallback immediately throws "All models failed" — even if the cooldown expires in just a few seconds. This surfaces as a hard error to end users on messaging channels (Discord, Telegram, etc.) for what is a brief, transient rate limit.

Error example

⚠ Agent failed before reply: All models failed (2):
anthropic/claude-sonnet-4-6: No available auth profile for anthropic (all in cooldown or unavailable). (rate_limit)
| anthropic/claude-sonnet-4-5: Provider anthropic is in cooldown (all profiles unavailable) (rate_limit)

Root cause

In src/agents/model-fallback.ts, runWithModelFallback() iterates over model candidates. When all profiles for a candidate are in cooldown, it skips that candidate immediately. If every candidate is skipped, the function throws without ever waiting for a cooldown to expire.

The existing shouldProbePrimaryDuringCooldown mechanism only probes the primary model (and only when fallback candidates exist). It does not cover the case where ALL candidates across ALL providers have all profiles in cooldown.

Contributing factors

  • Cooldown escalation is aggressive: calculateAuthProfileCooldownMs uses min(1hr, 1min × 5^(n-1)) — after just 3 errors, the cooldown is 25 minutes. After 4+, it's 1 hour.
  • Stale error counts compound the problem: If a profile accumulates errors (e.g. from an expired token) and the failureWindowHours hasn't elapsed, the next transient rate limit immediately escalates to a long cooldown. The clearExpiredCooldowns function addresses this when cooldowns expire naturally, but doesn't help when all profiles are simultaneously in cooldown.
  • No "wait for recovery" path: The function has no mechanism to sleep until the soonest cooldown expires when the wait would be short (e.g. under 90 seconds).

Expected behavior

When all model candidates fail due to cooldowns, and the soonest cooldown expiry is within a reasonable window (e.g. 90 seconds), the function should wait for the cooldown to expire, clear it, and retry once before throwing.

Proposed fix

PR #24157 implements this: after exhausting all candidates, if all failures are cooldown-related and the soonest expiry is ≤90s away, sleep until expiry + clear expired cooldowns + retry once.

Environment

  • OpenClaw version: 2026.2.21-2
  • Auth: Multiple OAuth token profiles on the same provider
  • Channels: Discord (most visible impact)

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions