Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

Automatic model failover does not work when the primary model hits rate limits. Instead of progressing through the configured fallback chain, the system enters an infinite loop attempting to revert to the failed primary model before each fallback can be tested. This requires manual intervention via the web portal.

Environment

• OpenClaw Version: 2026.3.28 (f9b1079)
• Channel: stable
• Installation Method: pnpm
• OS: Linux 6.12.73+deb13-amd64 (x64)
• Node: v22.22.1
• Gateway Mode: local
• Session Type: agent:main:main (direct, Telegram channel)
• Session ID: d47c1076-1f13-4712-9f9b-0dedf35753dd

Configuration

Model Configuration (openclaw.json)

"agents": {
  "defaults": {
    "model": {
      "primary": "google-gemini-cli/gemini-3-pro-preview",
      "fallbacks": [
        "google/gemini-2.5-flash",
        "anthropic/claude-haiku-4-5",
        "anthropic/claude-sonnet-4-5",
        "openai/gpt-5.1-codex",
        "google/gemini-3-pro-preview"
      ]
    }
  }
}

Auth Profile

"auth": {
  "profiles": {
    "google-gemini-cli:[[email protected]](mailto:[email protected])": {
      "provider": "google-gemini-cli",
      "mode": "oauth",
      "email": "[email protected]"
    }
  }
}

Expected Behavior

When the primary model (google-gemini-cli/gemini-3-pro-preview) returns HTTP 429 rate limit errors:

1. System should detect the rate limit failure 2. Mark the provider/profile as in cooldown 3. Attempt first fallback: google/gemini-2.5-flash 4. If that succeeds, remain on the working fallback until primary recovers 5. If fallback fails, progress to next in chain: anthropic/claude-haiku-4-5 6. Continue until a working model is found 7. Circuit breaker should prevent retrying rate-limited provider during cooldown window

User should experience: Brief delay (<30s) followed by automatic recovery on working fallback model.

Actual Behavior

System enters an infinite failover loop and never settles on a working model:

1. Primary model fails with 429 (rate limit) 2. System attempts to switch to first fallback 3. Before the fallback attempt executes, session reverts to primary model 4. Primary fails again (still rate-limited) 5. System tries next fallback 6. Loop repeats for all fallbacks in chain 7. No model ever successfully handles the request 8. Manual intervention required via web portal to force model selection

User experiences: Complete system unresponsiveness requiring manual fix.

Reproduction Steps

1. Configure a primary model that will hit rate limits (e.g., Gemini with heavy usage) 2. Configure a fallback chain with known-working models 3. Trigger enough API requests to exhaust primary model quota 4. Observe system behavior when rate limit is hit during an active session

Trigger condition: Primary model returns HTTP 429 during an active user request in a persistent session.

Evidence: Log Analysis

Timeline (2026-03-30 13:45-13:54 UTC)

Initial Rate Limit Detection

13:45:42 [agent/embedded] embedded run agent end: runId=6359ce69-be07-40df-97d3-3b152caf8f97 isError=true model=gemini-3-pro-preview provider=google-gemini-cli error=⚠️ API rate limit reached. Please try again later. rawError=Cloud Code Assist API error (429): You have exhausted your capacity on this model.

Cooldown Window Logged (But Not Enforced) ```

13:46:19 [agent/embedded] auth profile failure state updated: runId=6359ce69-be07-40df-97d3-3b152caf8f97 profile=sha256:4613a0bb7a76 provider=google-gemini-cli reason=rate_limit window=cooldown reused=false

13:46:19 [agent/embedded] embedded run failover decision: runId=6359ce69-be07-40df-97d3-3b152caf8f97 stage=assistant decision=fallback_model reason=rate_limit provider=google-gemini-cli/gemini-3-pro-preview profile=sha256:4613a0bb7a76

13:46:19 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=google-gemini-cli/gemini-3-pro-preview candidate=google-gemini-cli/gemini-3-pro-preview reason=rate_limit next=google/gemini-2.5-flash ```

Expected: System should now avoid google-gemini-cli during cooldown.

The Loop Begins (Smoking Gun)

13:46:20 [agent/embedded] live session model switch detected before attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: google/gemini-2.5-flash -> google-gemini-cli/gemini-3-pro-preview

13:46:20 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=google-gemini-cli/gemini-3-pro-preview candidate=google/gemini-2.5-flash reason=unknown next=anthropic/claude-haiku-4-5 ```

Critical Issue: "live session model switch detected before attempt" shows the session is being reset to the primary model before the fallback (`google/gemini-2.5-flash`) can even be tried.

#### Loop Continues Through All Fallbacks ```
13:46:20 [agent/embedded] live session model switch detected before attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: anthropic/claude-haiku-4-5 -> google-gemini-cli/gemini-3-pro-preview

13:46:20 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=google-gemini-cli/gemini-3-pro-preview candidate=anthropic/claude-haiku-4-5 reason=unknown next=anthropic/claude-sonnet-4-5

13:46:20 [agent/embedded] live session model switch detected before attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: anthropic/claude-sonnet-4-5 -> google-gemini-cli/gemini-3-pro-preview

13:46:20 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=google-gemini-cli/gemini-3-pro-preview candidate=anthropic/claude-sonnet-4-5 reason=unknown next=openai/gpt-5.1-codex

13:46:20 [agent/embedded] live session model switch detected before attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: openai/gpt-5.1-codex -> google-gemini-cli/gemini-3-pro-preview

13:46:20 [model-fallback/decision] model fallback decision: decision=candidate_failed requested=google-gemini-cli/gemini-3-pro-preview candidate=openai/gpt-5.1-codex reason=unknown next=google/gemini-3-pro-preview

13:46:20 [agent/embedded] live session model switch detected before attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: google/gemini-3-pro-preview -> google-gemini-cli/gemini-3-pro-preview

Pattern: Every fallback candidate is marked as candidate_failed with reason=unknown before it's attempted, because the session keeps reverting to the primary model.

Rate Limit Retry Spam (No Circuit Breaker) ```

13:46:20 [agent/embedded] google tool schema snapshot
13:46:28 [agent/embedded] embedded run agent end: runId=6359ce69-be07-40df-97d3-3b152caf8f97 isError=true model=gemini-3-pro-preview provider=google-gemini-cli error=⚠️ API rate limit reached. Please try again later.
13:46:37 [agent/embedded] embedded run agent end: runId=6359ce69-be07-40df-97d3-3b152caf8f97 isError=true model=gemini-3-pro-preview provider=google-gemini-cli error=⚠️ API rate limit reached. Please try again later.

This pattern repeats for 8 minutes until manual intervention.

#### Gateway Restart (Did Not Fix Loop)

13:51:16 [gateway] signal SIGTERM received
13:51:16 [gateway] received SIGTERM; shutting down
13:51:35 Started openclaw-gateway.service - OpenClaw Gateway (v2026.3.28).

After restart, the same loop behavior continued:

13:52:42 [agent/embedded] embedded run agent end: isError=true model=gemini-3-pro-preview provider=google-gemini-cli error=⚠️ API rate limit reached.
13:53:19 [agent/embedded] live session model switch detected after failed attempt for d47c1076-1f13-4712-9f9b-0dedf35753dd: google-gemini-cli/gemini-3-pro-preview -> anthropic/claude-sonnet-4-5 ```

Manual Resolution Required

User had to manually select anthropic/claude-sonnet-4-5 via web portal at ~13:53 UTC.

Root Cause Analysis

Primary Issue: Session State Overrides Failover Logic

The logs show a clear conflict between:

1. Failover decision engine (correctly identifying fallback candidates) 2. Live session model preference (forcing reversion to primary)

The "live session model switch detected before attempt" events indicate that session-level state is being restored/enforced in a way that prevents fallback models from ever being tested.

Contributing Factors

1. No circuit breaker on rate-limited providers

• Despite logging "auth profile failure state updated: window=cooldown"
• System continues attempting the rate-limited provider
• No enforcement of cooldown period

2. Retry logic doesn't respect failover state

• Each retry attempt resets to primary model
• Fallback chain is evaluated but never executed

3. Long retry delays

• 8-15 seconds between retry attempts
• 44+ seconds total per failover cycle
• Multiplied over 8 minutes = severe user impact

4. Gateway restart doesn't clear stuck state

• Same behavior after restart
• Session state appears to be persisted across restarts

Impact

User Experience

• 8 minutes of complete unresponsiveness (13:45-13:53 UTC)
• Multiple user queries went unanswered
• Required manual technical intervention via web portal
• Erosion of trust in automatic failover systems

System Reliability

• Core reliability feature (automatic failover) does not work as designed
• Rate limit handling is broken
• Manual intervention requirement defeats purpose of automation

Business Impact

• For production use cases (e.g., customer support bots), this would mean extended outages
• SLA violations if relying on automatic failover for uptime guarantees

Workarounds

Temporary Mitigation Applied

Manually selected anthropic/claude-sonnet-4-5 via web portal.

Preventive Workarounds

1. Make Sonnet the primary model:

   openclaw config set agents.defaults.model.primary anthropic/claude-sonnet-4-5

   Avoids rate limit exposure until bug is fixed.

2. Reduce usage of rate-limited providers:

• Switch cron jobs to less expensive models
• Use Gemini Flash instead of Pro for routine tasks

3. External monitoring:

• Implement external health checks
• Alert on consecutive failures
• Don't rely solely on internal failover

Requested Fix

Immediate (Patch Release)

1. Honor cooldown windows

• When a provider is marked window=cooldown, do NOT retry it
• Implement circuit breaker with exponential backoff

2. Fix session model reversion bug

• Prevent "live session model switch" from overriding active failover decisions
• Session preferences should be suspended during failover recovery

3. Let fallbacks actually execute

• Ensure each fallback candidate gets at least one real attempt
• Log when a fallback is actually used (not just considered)

Short-term (Next Release)

1. Add failover success logging

• Clear log entries when failover succeeds
• Example: [model-fallback/success] Successfully failed over to anthropic/claude-sonnet-4-5

2. Faster failover progression

• Don't retry failed primary model during failover cycle
• Move to next fallback within 2-3 seconds max

3. User-facing failover status

• Expose in Control UI when system is in failover mode
• Show which fallback is currently active

Long-term (Future Enhancement)

1. Configurable failover behavior

• Option: "sticky" failover (stay on working fallback)
• Option: "retry primary" with configurable timeout
• Option: "round-robin" across all working models

2. Rate limit prediction

• Track usage against quota
• Proactively switch before hitting limits
• Warn user when approaching quota

3. Per-session failover state

• Store which model is actually working for a session
• Don't reset on gateway restart
• Graceful migration back to primary when it recovers

Testing Recommendations

Regression Test Case

1. Configure primary + fallback chain
2. Mock rate limit response (429) from primary provider 3. Send user request 4. Verify: System progresses to first fallback within 5 seconds 5. Verify: Request completes successfully on fallback 6. Verify: No retries of rate-limited provider during cooldown 7. Verify: Logs show [model-failover/success] event

Integration Test Case

1. Configure real Gemini API with low quota 2. Exhaust quota with automated requests 3. Send user request via Telegram 4. Verify: Automatic failover to working model 5. Verify: User receives response within 30 seconds 6. Verify: No manual intervention required

Additional Context

Related Issues

• Session persistence across restarts may be related
• Similar to issues reported with model switching in sub-agents?

User Profile

• Solo production deployment
• Multiple Telegram groups (6+ agents)
• Heavy cron usage (10 jobs in 21h)
• Relies on automatic failover for reliability

Attachments

Full Log Excerpt

See attached: openclaw-gateway-failover-loop-2026-03-30.log (13:45-13:54 UTC)

Configuration

See attached: openclaw.json (redacted auth tokens)

Session Details

• Session ID: d47c1076-1f13-4712-9f9b-0dedf35753dd
• Agent: main
• Channel: telegram
• Chat ID: 6318186767

Contact

Reporter: Peter Gialames (via Herman assistant agent)
Environment: Production solo deployment
Availability: Available for follow-up testing

Priority Justification

This is a HIGH severity bug because:

1. ✅ Core reliability feature (automatic failover) completely broken 2. ✅ Affects production deployments relying on uptime guarantees 3. ✅ Requires manual technical intervention (not acceptable for automation) 4. ✅ Reproducible with clear evidence 5. ✅ Impacts all users with fallback chains configured 6. ✅ Workarounds exist but defeat the purpose of the feature

Request: Prioritize for next patch release (2026.3.29 or 2026.4.x).

Steps to reproduce

1. Configure google-gemini-cli/gemini-3-pro-preview as the primary model and a fallback chain (as shown in the configuration above).
2. Ensure there's enough activity or concurrent requests to trigger a rate limit (HTTP 429) on the primary Gemini model.
3. 3. Observe the OpenClaw agent's behavior during an active session when the rate limit is hit.

Trigger condition: Primary model returns HTTP 429 during an active user request in a persistent session.

Expected behavior

When the primary model (google-gemini-cli/gemini-3-pro-preview) returns HTTP 429 rate limit errors:

1. System should detect the rate limit failure.
2. Mark the provider/profile as in cooldown.
3. Attempt first fallback: google/gemini-2.5-flash.
4. If that succeeds, remain on the working fallback until primary recovers.
5. If fallback fails, progress to next in chain: anthropic/claude-haiku-4-5.
6. Continue until a working model is found.
7. A circuit breaker should prevent retrying the rate-limited provider during its cooldown window.

User should experience: Brief delay (<30s) followed by automatic recovery on a working fallback model.

Actual behavior

System enters an infinite failover loop and never settles on a working model:

1. Primary model fails with 429 (rate limit).
2. System attempts to switch to first fallback.
3. Before the fallback attempt executes, the session reverts to the primary model.
4. Primary fails again (still rate-limited).
5. System tries next fallback.
6. This loop repeats for all fallbacks in the chain.
7. No model ever successfully handles the request.
8. Manual intervention required via the web portal to force model selection.

User experiences: Complete system unresponsiveness requiring manual fix.

OpenClaw version

v2026.3.28 (stable channel)

Operating system

Linux 6.12.73+deb13-amd64 (x64)

Install method

pnpm

Model

google-gemini-cli/gemini-3-pro-preview

Provider / routing chain

openclaw -> google-gemini-cli -> google/gemini-3-pro-preview

Additional provider/model setup details

No response

Logs, screenshots, and evidence

Impact and severity

No response

Additional information

No response