Skip to content

Fix: live session model switch no longer blocks failover (Resolves #58466)#58589

Merged
fabianwilliams merged 2 commits into
openclaw:mainfrom
Mlightsnow:fix/58466-failover-live-model-switch-loop
Apr 1, 2026
Merged

Fix: live session model switch no longer blocks failover (Resolves #58466)#58589
fabianwilliams merged 2 commits into
openclaw:mainfrom
Mlightsnow:fix/58466-failover-live-model-switch-loop

Conversation

@Mlightsnow

Copy link
Copy Markdown
Contributor

Fixes #58466.

Problem

When a model returns overloaded_error (503), runWithModelFallback attempts to switch to a fallback model. However, the "live session model switch" guard inside runEmbeddedPiAgent detects the model difference between the fallback candidate and the session store, throwing LiveSessionModelSwitchError. When this happens on the last fallback candidate, the error is re-thrown by runWithModelFallback, causing the outer retry loop (cron/auto-reply) to restart with the original overloaded model — creating an infinite retry loop with no circuit breaker.

Fix

Primary fix (model-fallback.ts): Intercept LiveSessionModelSwitchError in the fallback loop and convert it to a FailoverError. This allows the fallback chain to continue to the next candidate instead of re-throwing the error. When all candidates are exhausted, a proper fallback failure summary is surfaced instead of the raw LiveSessionModelSwitchError.

Circuit breaker (cron/isolated-agent/run.ts): Add a max retry limit (2) for LiveSessionModelSwitchError in the cron retry loop, matching the existing guard in the auto-reply path (added in #58348). This prevents infinite loops even if LiveSessionModelSwitchError somehow escapes the fallback handler.

Test plan

  • New test: LiveSessionModelSwitchError on last candidate surfaces as FailoverError, not LiveSessionModelSwitchError
  • New test: fallback chain continues past LiveSessionModelSwitchError to next candidate
  • New test: cron retry loop aborts after exceeding retry limit
  • Existing model-fallback and cron live-session-model-switch tests pass
  • pnpm check passes

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Mar 31, 2026
@greptile-apps

greptile-apps Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a subtle infinite-retry loop (issue #58466) where a LiveSessionModelSwitchError thrown during model fallback would escape runWithModelFallback and cause the outer cron/auto-reply loop to restart indefinitely with the original overloaded model.

Changes:

  • model-fallback.ts: Intercepts LiveSessionModelSwitchError inside the fallback candidate loop and converts it to a FailoverError so the chain continues to the next candidate. When all candidates are exhausted, a proper fallback failure summary is surfaced instead of the raw LiveSessionModelSwitchError.
  • cron/isolated-agent/run.ts: Adds a MAX_MODEL_SWITCH_RETRIES = 2 circuit breaker to the LiveSessionModelSwitchError retry loop, matching the existing guard in the auto-reply path from Session death loop on Anthropic overloaded_error — no fallback, no retry limit, zombie state #58348.
  • Tests: Three new tests cover the critical scenarios — last-candidate conversion, chain continuation, and circuit-breaker abort.

The fix is well-reasoned and the two-layer defense (primary conversion + circuit breaker safety net) is sound. One minor observability note: the new FailoverError that wraps a LiveSessionModelSwitchError hardcodes reason: "overloaded", which is semantically inaccurate since the error reflects a session model conflict rather than infrastructure overload — "unknown" would be a more accurate choice for logging/metrics purposes.

Confidence Score: 5/5

Safe to merge — fixes a real infinite-retry regression with no functional regressions introduced.

All remaining findings are P2 (observability/semantic accuracy of the logged reason). The core bug fix and circuit breaker are correct, well-tested, and consistent with existing patterns. No data integrity, security, or reliability concerns.

No files require special attention.

Important Files Changed

Filename Overview
src/agents/model-fallback.ts Intercepts LiveSessionModelSwitchError in the fallback loop and converts it to a FailoverError so the chain continues to the next candidate. Logic is correct; minor semantic concern with reason: "overloaded".
src/agents/model-fallback.test.ts Adds two new tests: one verifying LiveSessionModelSwitchError on the last candidate surfaces as FailoverError, and one verifying the fallback chain continues past the error to the next candidate. Good coverage of the new behavior.
src/cron/isolated-agent/run.ts Adds a MAX_MODEL_SWITCH_RETRIES = 2 circuit breaker to the LiveSessionModelSwitchError retry loop, preventing infinite loops if the error somehow escapes the primary fix. Implementation is correct and consistent with the auto-reply guard from #58348.
src/cron/isolated-agent/run.live-session-model-switch.test.ts Adds a test verifying the circuit breaker aborts after 3 total attempts (initial + 2 retries) and logs the expected warning. Test is well-scoped and correctly validates the max-retry boundary condition.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 793-797

Comment:
**Hardcoded `reason: "overloaded"` is semantically inaccurate**

`LiveSessionModelSwitchError` signals a session-stored model conflict, not an infrastructure overload. Using `"overloaded"` here will appear in the `attempts` array and the final `FailoverError` summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.

Looking at `failover-policy.ts`, valid `FailoverReason` values include `"unknown"`, which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:

```suggestion
        const switchNormalized = new FailoverError(switchMsg, {
          reason: "unknown",
          provider: candidate.provider,
          model: candidate.model,
        });
```

This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the `reason` value.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix: prevent infinite retry loop when li..." | Re-trigger Greptile

Comment on lines +793 to +797
const switchNormalized = new FailoverError(switchMsg, {
reason: "overloaded",
provider: candidate.provider,
model: candidate.model,
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Hardcoded reason: "overloaded" is semantically inaccurate

LiveSessionModelSwitchError signals a session-stored model conflict, not an infrastructure overload. Using "overloaded" here will appear in the attempts array and the final FailoverError summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.

Looking at failover-policy.ts, valid FailoverReason values include "unknown", which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:

Suggested change
const switchNormalized = new FailoverError(switchMsg, {
reason: "overloaded",
provider: candidate.provider,
model: candidate.model,
});
const switchNormalized = new FailoverError(switchMsg, {
reason: "unknown",
provider: candidate.provider,
model: candidate.model,
});

This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the reason value.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 793-797

Comment:
**Hardcoded `reason: "overloaded"` is semantically inaccurate**

`LiveSessionModelSwitchError` signals a session-stored model conflict, not an infrastructure overload. Using `"overloaded"` here will appear in the `attempts` array and the final `FailoverError` summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.

Looking at `failover-policy.ts`, valid `FailoverReason` values include `"unknown"`, which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:

```suggestion
        const switchNormalized = new FailoverError(switchMsg, {
          reason: "unknown",
          provider: candidate.provider,
          model: candidate.model,
        });
```

This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the `reason` value.

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e7014c342

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +791 to +795
if (err instanceof LiveSessionModelSwitchError) {
const switchMsg = err.message;
const switchNormalized = new FailoverError(switchMsg, {
reason: "overloaded",
provider: candidate.provider,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve live-switch error on sole fallback candidate

This branch converts every LiveSessionModelSwitchError into a generic FailoverError and continues, including when the current candidate is the last/only candidate. That removes the signal that outer retry loops depend on (runCronIsolatedAgentTurn and runAgentTurnWithFallback only retry on LiveSessionModelSwitchError), so runs with no remaining fallbacks (for example fallbacksOverride: [] or an empty fallback config) can no longer apply the requested live model/auth-profile switch and now fail immediately instead of retrying with the switched selection. The conversion should be limited to cases where another candidate remains, while preserving the original error on the final attempt.

Useful? React with 👍 / 👎.

@Mlightsnow
Mlightsnow force-pushed the fix/58466-failover-live-model-switch-loop branch from 5e7014c to 0147aa0 Compare April 1, 2026 00:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0147aa0d40

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +791 to +793
if (err instanceof LiveSessionModelSwitchError) {
const switchMsg = err.message;
const switchNormalized = new FailoverError(switchMsg, {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Re-throw live-switch error on final candidate

Handling LiveSessionModelSwitchError unconditionally as a FailoverError here means the fallback loop never re-throws the original switch signal on the last/only candidate. In single-candidate runs (for example fallbacksOverride: [] or configs without fallbacks), this now surfaces as a generic failover and bypasses outer retry handlers that only react to LiveSessionModelSwitchError (src/cron/isolated-agent/run.ts:604 and src/auto-reply/reply/agent-runner-execution.ts:624), so the requested provider/model/auth-profile switch is not applied and the run fails immediately. Fresh evidence from this commit is that cron now has its own retry cap (src/cron/isolated-agent/run.ts:597-611), so preserving the original switch error on the final candidate no longer implies an unbounded loop.

Useful? React with 👍 / 👎.

@fabianwilliams fabianwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — reviewed code, tests, and approach. Clean implementation.

@fabianwilliams
fabianwilliams merged commit 5471548 into openclaw:main Apr 1, 2026
40 checks passed
@Mlightsnow
Mlightsnow deleted the fix/58466-failover-live-model-switch-loop branch April 1, 2026 02:46
steipete pushed a commit to Mlightsnow/openclaw that referenced this pull request Apr 1, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
Nachx639 pushed a commit to Nachx639/clawdbot that referenced this pull request Jun 17, 2026
…enclaw#58466) (openclaw#58589)

* fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466)

* fix: remove unused resolveOllamaBaseUrlForRun import after rebase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: live session model switch blocks failover on overloaded_error — infinite retry loop

2 participants