fix: distinguish user-initiated model switches from system fallbacks in LiveSessionModelSwitchError#60266
Conversation
Greptile SummaryThis PR replaces the ambiguous in-memory model-switch detection ( The two new helpers — Confidence Score: 5/5Safe to merge — the fix is logically sound and all remaining findings are minor P2 observations. The root-cause fix (disk-flag instead of in-memory model comparison) is correct and precisely scoped. The two P2 findings (redundant disk reads in No files require special attention.
|
| const storePath = resolveStorePath(cfg.session?.store, { | ||
| agentId: params.agentId?.trim(), | ||
| }); | ||
| const entry = loadSessionStore(storePath, { skipCache: true })[sessionKey]; | ||
| if (!entry?.liveModelSwitchPending) { | ||
| return undefined; | ||
| } | ||
| const persisted = resolveLiveSessionModelSelection({ | ||
| cfg, | ||
| sessionKey, | ||
| agentId: params.agentId, | ||
| defaultProvider: params.defaultProvider, | ||
| defaultModel: params.defaultModel, | ||
| }); | ||
| if ( | ||
| !hasDifferentLiveSessionModelSelection( | ||
| { | ||
| provider: params.currentProvider, | ||
| model: params.currentModel, | ||
| authProfileId: params.currentAuthProfileId, | ||
| authProfileIdSource: params.currentAuthProfileIdSource, | ||
| }, | ||
| persisted, | ||
| ) | ||
| ) { | ||
| return undefined; | ||
| } | ||
| return persisted ?? undefined; |
There was a problem hiding this comment.
Two separate disk reads for the same session entry
shouldSwitchToLiveModel reads the session store twice with skipCache: true: once here to check liveModelSwitchPending, and a second time inside resolveLiveSessionModelSelection (which also calls loadSessionStore). In practice the window is tiny, but a write arriving between the two reads could cause liveModelSwitchPending to be cleared on disk while resolveLiveSessionModelSelection picks up a different snapshot. Passing the already-loaded entry directly to avoid the redundant read would close this gap, though it would require small refactoring of resolveLiveSessionModelSelection.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/live-model-switch.ts
Line: 156-183
Comment:
**Two separate disk reads for the same session entry**
`shouldSwitchToLiveModel` reads the session store twice with `skipCache: true`: once here to check `liveModelSwitchPending`, and a second time inside `resolveLiveSessionModelSelection` (which also calls `loadSessionStore`). In practice the window is tiny, but a write arriving between the two reads could cause `liveModelSwitchPending` to be cleared on disk while `resolveLiveSessionModelSelection` picks up a different snapshot. Passing the already-loaded `entry` directly to avoid the redundant read would close this gap, though it would require small refactoring of `resolveLiveSessionModelSelection`.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Valid observation. The two disk reads are: (1) loadSessionStore to check liveModelSwitchPending, then (2) resolveLiveSessionModelSelection which calls loadSessionStore again internally. The race window is negligible (microseconds, same process, synchronous calls) and fixing it would require refactoring resolveLiveSessionModelSelection to accept a pre-loaded entry, which touches other callers. Keeping this as-is for now to minimize the surface area of the change.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93facbcca6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ) { | ||
| return undefined; |
There was a problem hiding this comment.
Consume pending live-switch flag when selection already matches
shouldSwitchToLiveModel exits early when the persisted selection equals the current model, but it leaves liveModelSwitchPending set. That means a /model change made while idle (or during a turn that cannot restart) can leave a stale flag in sessions.json; later, any unrelated fallback attempt that differs from the persisted model can be treated as a fresh live-switch request and throw LiveSessionModelSwitchError, short-circuiting normal retry/compaction behavior. This branch should consume/clear the pending flag once the requested selection is already active.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — addressed in 440731475d. shouldSwitchToLiveModel now calls void clearLiveModelSwitchPending(...) (fire-and-forget) in the models-match branch before returning undefined. A dedicated test ("clears the stale liveModelSwitchPending flag when models already match") verifies the flag is removed from the session entry.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 440731475d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| void clearLiveModelSwitchPending({ | ||
| cfg, | ||
| sessionKey, | ||
| agentId: params.agentId, | ||
| }); |
There was a problem hiding this comment.
Catch fire-and-forget stale-flag clear failures
The stale-flag branch launches clearLiveModelSwitchPending(...) with void and no rejection handler, but that function awaits updateSessionStore, which can throw on write/lock/fs errors. Under Node 22, an unhandled rejection is fatal, so a transient session-store write failure here can terminate the process even though this path is only trying to clean up state. Please await this call or attach an explicit .catch(...) handler.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 31492d8 — replaced void clearLiveModelSwitchPending(...) with clearLiveModelSwitchPending(...).catch(() => { /* best-effort */ }). This prevents fatal unhandled rejections under Node 22+ while keeping the fire-and-forget semantics (the stale-flag clear is non-critical — the worst case if it fails is one extra no-op re-evaluation on the next retry iteration).
4407314 to
31492d8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 31492d8881
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| await clearLiveModelSwitchPending({ | ||
| cfg: params.config, | ||
| sessionKey: params.sessionKey, | ||
| agentId: params.agentId, | ||
| }); |
There was a problem hiding this comment.
Guard pending-flag clear against newer /model updates
requestedSelection is computed from a snapshot (shouldSwitchToLiveModel) and then clearLiveModelSwitchPending deletes liveModelSwitchPending unconditionally before throwing. Because /model writes that same flag in handleDirectiveOnly (sessionEntry.liveModelSwitchPending = true) and persists via updateSessionStore, a second /model command that lands between the read and this clear can have its new pending switch erased, while the runner still throws using the older selection. This drops the most recent user intent and can switch to the wrong model under rapid consecutive /model changes.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid theoretical concern. The race window is extremely narrow: shouldSwitchToLiveModel reads the flag and clearLiveModelSwitchPending clears it within the same synchronous call site in run.ts:621-641 — the only async gap is the updateSessionStore write itself. A second /model command arriving in that exact window would need to complete its own updateSessionStore write between our read and our clear.
In practice this requires sub-millisecond /model command timing during an active retry attempt, which is not a realistic user interaction pattern. A compare-and-swap guard (e.g., checking a generation counter before clearing) would add complexity disproportionate to the risk.
Noted as a potential follow-up if rapid /model switching becomes a supported UX pattern.
31492d8 to
ec05597
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec05597da2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const persisted = resolveLiveSessionModelSelection({ | ||
| cfg, | ||
| sessionKey, | ||
| agentId: params.agentId, | ||
| defaultProvider: params.defaultProvider, |
There was a problem hiding this comment.
Resolve pending switch from selected override, not runtime model
shouldSwitchToLiveModel derives persisted via resolveLiveSessionModelSelection, which prefers runtime modelProvider/model over providerOverride/modelOverride. When /model is issued during a turn that cannot restart, liveModelSwitchPending remains set, and the run-completion path rewrites runtime model fields to the old model (updateSessionStoreAfterAgentRun in src/agents/command/session-store.ts). On the next turn, this code can interpret the old runtime model as the requested switch, so the pending flag is never consumed (or, on a retryable empty attempt, it throws LiveSessionModelSwitchError back to the old model), effectively dropping the user’s latest model choice.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This concern is about the behavior of resolveLiveSessionModelSelection (an existing function we don't modify), not the new flag mechanism. The function's preference order (runtime fields > override fields) is the existing upstream contract — changing it would be a separate refactor with its own risk surface.
In the specific scenario described: /model during an active run sets providerOverride/modelOverride AND liveModelSwitchPending = true. When the run completes and updateSessionStoreAfterAgentRun writes runtime fields, the override fields still reflect the user's choice. On the next turn, resolveLiveSessionModelSelection reads runtime fields first — but these were just written by the previous run's completion, and the override fields still carry the user's /model selection. The pending flag ensures the switch fires.
If the runtime fields happen to match the old model, the switch still works because shouldSwitchToLiveModel compares the resolved selection against the current candidate — they'll differ, and the switch throws correctly.
The edge case would require runtime fields to be overwritten to match the new model (making the switch look like a no-op) while overrides are cleared — but updateSessionStoreAfterAgentRun doesn't clear overrides, so this doesn't happen in practice.
ec05597 to
d059147
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0591476db
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const persisted = resolveLiveSessionModelSelection({ | ||
| cfg, | ||
| sessionKey, | ||
| agentId: params.agentId, | ||
| defaultProvider: params.defaultProvider, |
There was a problem hiding this comment.
Resolve pending switch from persisted override, not runtime
shouldSwitchToLiveModel computes the pending target via resolveLiveSessionModelSelection, which internally prefers modelProvider/model runtime fields over providerOverride/modelOverride. In the /model-during-active-run path, liveModelSwitchPending can carry into the next turn, and updateSessionStoreAfterAgentRun repopulates runtime fields with the old model from the previous run; this makes the pending switch resolve to the stale runtime model and can throw LiveSessionModelSwitchError back to the old selection, dropping the user’s requested switch. Fresh evidence in this commit is that pending-switch handling now routes through this new call site while the new flag is persisted from directive handling, so the stale runtime-vs-override conflict is now on the critical path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Duplicate of the comment above — same concern about resolveLiveSessionModelSelection preferring runtime over override fields. See reply there. The resolution priority is the existing upstream contract; our PR doesn't modify it.
|
@openclaw-bot please assign to a maintainer and get this merged to the upstream main. |
Replace the ambiguous comparison-based approach (hasDifferentLiveSessionModelSelection + in-memory map EMBEDDED_RUN_MODEL_SWITCH_REQUESTS) with a persisted `liveModelSwitchPending` flag on SessionEntry. The root cause: the in-memory map was never populated in production because requestLiveSessionModelSwitch() was removed in commit 622b91d and replaced with refreshQueuedFollowupSession(). This left the comparison-based detection as the only path, which could not distinguish user-initiated model switches (via /model command) from system-initiated fallback rotations. The fix: - Add `liveModelSwitchPending?: boolean` to SessionEntry (persisted) - Set the flag to true ONLY when /model command applies a model override - New `shouldSwitchToLiveModel()` checks the flag + model mismatch together - New `clearLiveModelSwitchPending()` resets the flag after consumption - Replace throw-site logic in run.ts to use the new flag-based functions - Remove orphaned resolveCurrentLiveSelection helper Only the /model command sets this flag, so system-initiated fallback rotations are never mistaken for user-initiated model switches. This restores the live-switch-during-active-run feature that was accidentally broken. Fixes openclaw#57857, openclaw#57760, openclaw#58137
When the liveModelSwitchPending flag is set but the current model already matches the persisted selection (e.g. the switch was applied as an override and the current attempt is already using the new model), the flag is now consumed eagerly via a fire-and-forget clearLiveModelSwitchPending() call. Without this, the stale flag could persist across fallback iterations and later cause a spurious LiveSessionModelSwitchError when the model rotates to a fallback candidate that differs from the persisted selection. Also expands JSDoc on shouldSwitchToLiveModel to document the stale-flag clearing and deferral semantics.
d059147 to
32fb15d
Compare
|
Landed via temp rebase onto main.
Thanks @kiranvk-2011! |
|
@steipete humbled and honored 🙏 |
Problem
LiveSessionModelSwitchError(introduced in v2026.3.28) detects mid-conversation model switches by comparing the current retry candidate against the session's persisted model selection. This comparison cannot distinguish between:/model claude-sonnet-4-6during an active retryrunWithModelFallbacktries the next candidate in the fallback chainIn scenarios 2 and 3, every fallback/override candidate differs from the persisted primary →
hasDifferentLiveSessionModelSelection()returnstrue→ every candidate throwsLiveSessionModelSwitchErrorbefore making any API call → entire fallback chain fails → user sees "All models failed."This completely negates the stepped backoff and per-model cooldown improvements from PR #49834.
Note: The cron/subagent variant of this bug was fixed by @vincentkoc in v2026.4.1. This PR addresses the remaining scenarios.
Fixes #57857
Fixes #57760
Fixes #58137
Solution
Replace the ambiguous comparison-based detection with an explicit user-action signal:
liveModelSwitchPendingflag to the session entry/modelcommand handler — the only place where a user explicitly requests a model switchhasDifferentLiveSessionModelSelection()withshouldSwitchToLiveModel()that checks the flag instead of comparing model namesWhy this approach?
isFallbackAttemptparameter, this doesn't require modifying call signatures throughout the retry chainChanges
src/config/sessions/types.tsliveModelSwitchPending?: booleantoSessionEntrysrc/agents/live-model-switch.tsshouldSwitchToLiveModel()andclearLiveModelSwitchPending()src/agents/pi-embedded-runner/run.tshasDifferentLiveSessionModelSelection+consumeLiveSessionModelSwitchwith flag-based functions; remove orphanedresolveCurrentLiveSelectionhelpersrc/auto-reply/reply/directive-handling.impl.tsliveModelSwitchPending = truewhen/modelcommand applies a model overridesrc/agents/live-model-switch.test.tsshouldSwitchToLiveModel()andclearLiveModelSwitchPending()Test Plan
shouldSwitchToLiveModel()(4 tests) andclearLiveModelSwitchPending()(3 tests)LiveSessionModelSwitchErrorinmodel-fallback.test.tsstill passlive-model-switch.test.tspassCascade failure sequence (before fix)
For reference, here's the exact failure sequence when the primary model hits a 429:
runWithModelFallbackstarts with candidates:[primary/opus, fallback/glm-5, fallback/minimax, ...]primary/opus→ API call → 429 quota exceeded →markAuthProfileFailure()→ 30s cooldownfallback/glm-5→ BEFORE API call:hasDifferentLiveSessionModelSelection()comparesfallback/glm-5vs persistedprimary/opus→ DIFFERENT →throw LiveSessionModelSwitchError→ NO API CALL MADEfallback/minimax→ same check → DIFFERENT →throw→ NO API CALL MADEAfter this fix, step 3+ would check
shouldSwitchToLiveModel()which returnsnull(no user-initiated switch pending) → fallback candidates proceed to make their API calls normally.