Fix: live session model switch no longer blocks failover (Resolves #58466)#58589
Conversation
Greptile SummaryThis PR fixes a subtle infinite-retry loop (issue #58466) where a Changes:
The fix is well-reasoned and the two-layer defense (primary conversion + circuit breaker safety net) is sound. One minor observability note: the new Confidence Score: 5/5Safe to merge — fixes a real infinite-retry regression with no functional regressions introduced. All remaining findings are P2 (observability/semantic accuracy of the logged reason). The core bug fix and circuit breaker are correct, well-tested, and consistent with existing patterns. No data integrity, security, or reliability concerns. No files require special attention.
|
| Filename | Overview |
|---|---|
| src/agents/model-fallback.ts | Intercepts LiveSessionModelSwitchError in the fallback loop and converts it to a FailoverError so the chain continues to the next candidate. Logic is correct; minor semantic concern with reason: "overloaded". |
| src/agents/model-fallback.test.ts | Adds two new tests: one verifying LiveSessionModelSwitchError on the last candidate surfaces as FailoverError, and one verifying the fallback chain continues past the error to the next candidate. Good coverage of the new behavior. |
| src/cron/isolated-agent/run.ts | Adds a MAX_MODEL_SWITCH_RETRIES = 2 circuit breaker to the LiveSessionModelSwitchError retry loop, preventing infinite loops if the error somehow escapes the primary fix. Implementation is correct and consistent with the auto-reply guard from #58348. |
| src/cron/isolated-agent/run.live-session-model-switch.test.ts | Adds a test verifying the circuit breaker aborts after 3 total attempts (initial + 2 retries) and logs the expected warning. Test is well-scoped and correctly validates the max-retry boundary condition. |
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 793-797
Comment:
**Hardcoded `reason: "overloaded"` is semantically inaccurate**
`LiveSessionModelSwitchError` signals a session-stored model conflict, not an infrastructure overload. Using `"overloaded"` here will appear in the `attempts` array and the final `FailoverError` summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.
Looking at `failover-policy.ts`, valid `FailoverReason` values include `"unknown"`, which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:
```suggestion
const switchNormalized = new FailoverError(switchMsg, {
reason: "unknown",
provider: candidate.provider,
model: candidate.model,
});
```
This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the `reason` value.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "fix: prevent infinite retry loop when li..." | Re-trigger Greptile
| const switchNormalized = new FailoverError(switchMsg, { | ||
| reason: "overloaded", | ||
| provider: candidate.provider, | ||
| model: candidate.model, | ||
| }); |
There was a problem hiding this comment.
Hardcoded
reason: "overloaded" is semantically inaccurate
LiveSessionModelSwitchError signals a session-stored model conflict, not an infrastructure overload. Using "overloaded" here will appear in the attempts array and the final FailoverError summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.
Looking at failover-policy.ts, valid FailoverReason values include "unknown", which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:
| const switchNormalized = new FailoverError(switchMsg, { | |
| reason: "overloaded", | |
| provider: candidate.provider, | |
| model: candidate.model, | |
| }); | |
| const switchNormalized = new FailoverError(switchMsg, { | |
| reason: "unknown", | |
| provider: candidate.provider, | |
| model: candidate.model, | |
| }); |
This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the reason value.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/model-fallback.ts
Line: 793-797
Comment:
**Hardcoded `reason: "overloaded"` is semantically inaccurate**
`LiveSessionModelSwitchError` signals a session-stored model conflict, not an infrastructure overload. Using `"overloaded"` here will appear in the `attempts` array and the final `FailoverError` summary logged when all candidates are exhausted, making it harder to distinguish a genuine overload from a session state mismatch.
Looking at `failover-policy.ts`, valid `FailoverReason` values include `"unknown"`, which is a much more accurate stand-in since the session mismatch may or may not have been caused by overload on the original model:
```suggestion
const switchNormalized = new FailoverError(switchMsg, {
reason: "unknown",
provider: candidate.provider,
model: candidate.model,
});
```
This is purely an observability concern — the core fix (preventing re-throw and infinite retry) is unaffected by the `reason` value.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e7014c342
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (err instanceof LiveSessionModelSwitchError) { | ||
| const switchMsg = err.message; | ||
| const switchNormalized = new FailoverError(switchMsg, { | ||
| reason: "overloaded", | ||
| provider: candidate.provider, |
There was a problem hiding this comment.
Preserve live-switch error on sole fallback candidate
This branch converts every LiveSessionModelSwitchError into a generic FailoverError and continues, including when the current candidate is the last/only candidate. That removes the signal that outer retry loops depend on (runCronIsolatedAgentTurn and runAgentTurnWithFallback only retry on LiveSessionModelSwitchError), so runs with no remaining fallbacks (for example fallbacksOverride: [] or an empty fallback config) can no longer apply the requested live model/auth-profile switch and now fail immediately instead of retrying with the switched selection. The conversion should be limited to cases where another candidate remains, while preserving the original error on the final attempt.
Useful? React with 👍 / 👎.
5e7014c to
0147aa0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0147aa0d40
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (err instanceof LiveSessionModelSwitchError) { | ||
| const switchMsg = err.message; | ||
| const switchNormalized = new FailoverError(switchMsg, { |
There was a problem hiding this comment.
Re-throw live-switch error on final candidate
Handling LiveSessionModelSwitchError unconditionally as a FailoverError here means the fallback loop never re-throws the original switch signal on the last/only candidate. In single-candidate runs (for example fallbacksOverride: [] or configs without fallbacks), this now surfaces as a generic failover and bypasses outer retry handlers that only react to LiveSessionModelSwitchError (src/cron/isolated-agent/run.ts:604 and src/auto-reply/reply/agent-runner-execution.ts:624), so the requested provider/model/auth-profile switch is not applied and the run fails immediately. Fresh evidence from this commit is that cron now has its own retry cap (src/cron/isolated-agent/run.ts:597-611), so preserving the original switch error on the final candidate no longer implies an unbounded loop.
Useful? React with 👍 / 👎.
fabianwilliams
left a comment
There was a problem hiding this comment.
LGTM — reviewed code, tests, and approach. Clean implementation.
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
…enclaw#58466) (openclaw#58589) * fix: prevent infinite retry loop when live session model switch blocks failover (openclaw#58466) * fix: remove unused resolveOllamaBaseUrlForRun import after rebase
Fixes #58466.
Problem
When a model returns
overloaded_error(503),runWithModelFallbackattempts to switch to a fallback model. However, the "live session model switch" guard insiderunEmbeddedPiAgentdetects the model difference between the fallback candidate and the session store, throwingLiveSessionModelSwitchError. When this happens on the last fallback candidate, the error is re-thrown byrunWithModelFallback, causing the outer retry loop (cron/auto-reply) to restart with the original overloaded model — creating an infinite retry loop with no circuit breaker.Fix
Primary fix (
model-fallback.ts): InterceptLiveSessionModelSwitchErrorin the fallback loop and convert it to aFailoverError. This allows the fallback chain to continue to the next candidate instead of re-throwing the error. When all candidates are exhausted, a proper fallback failure summary is surfaced instead of the rawLiveSessionModelSwitchError.Circuit breaker (
cron/isolated-agent/run.ts): Add a max retry limit (2) forLiveSessionModelSwitchErrorin the cron retry loop, matching the existing guard in the auto-reply path (added in #58348). This prevents infinite loops even ifLiveSessionModelSwitchErrorsomehow escapes the fallback handler.Test plan
LiveSessionModelSwitchErroron last candidate surfaces asFailoverError, notLiveSessionModelSwitchErrorLiveSessionModelSwitchErrorto next candidatepnpm checkpasses