Skip to content

Commit b098e77

Browse files
steipetesteipete
committed
fix(agents): probe primary after auto fallback pin
1 parent 1426112 commit b098e77

19 files changed

Lines changed: 1990 additions & 66 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ Docs: https://docs.openclaw.ai
8989
- Providers/catalogs: reject malformed successful LM Studio, GitHub Copilot, DeepInfra, Vercel AI Gateway, and Kilocode model-list responses with provider-owned errors instead of raw parser/type failures or silent fallback catalogs.
9090
- Providers/polling: reject array, null, or scalar successful operation status responses with provider-owned malformed JSON errors instead of waiting until timeout.
9191
- ACPX/Codex: reap plugin-local Codex ACP adapter orphans on startup after wrapper crashes while keeping direct adapter commands out of launch-lease injection. Fixes #82364. (#82459) Thanks @joshavant.
92+
- Agents/model fallback: periodically probe the configured primary for auto-pinned fallback sessions and clear the pin when it recovers, preventing sessions from staying on a fallback model indefinitely. Fixes #82544. Thanks @crpol.
9293
- Telegram: send presentation-only payloads by rendering fallback text and inline buttons instead of treating them as empty. Fixes #82404. (#82449) Thanks @joshavant.
9394
- Providers/Kimi: preserve Kimi Coding `reasoning_content` replay and backfill assistant tool-call placeholders when thinking is enabled, so `kimi-for-coding` follow-up tool turns no longer fail after prior tool use. Fixes #82161. Thanks @amknight.
9495
- Providers/search tools: reject malformed successful xAI, Gemini, and Kimi web/code search responses with provider-owned errors instead of silent `No response` payloads or ungrounded fallback state.

‎docs/concepts/model-failover.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ OpenClaw separates the selected provider/model from why it was selected. That so
6060

6161
- **Configured default**: `agents.defaults.model.primary` uses `agents.defaults.model.fallbacks`.
6262
- **Agent primary**: `agents.list[].model` is strict unless that agent model object includes its own `fallbacks`. Use `fallbacks: []` to make the strict behavior explicit, or provide a non-empty list to opt that agent into model fallback.
63-
- **Auto fallback override**: a runtime fallback writes `providerOverride`, `modelOverride`, `modelOverrideSource: "auto"`, and the selected origin model before retrying. That auto override can keep walking the configured fallback chain and is cleared by `/new`, `/reset`, and `sessions.reset`. Heartbeat runs without an explicit `heartbeat.model` also clear a direct auto override when its origin no longer matches the current configured default.
63+
- **Auto fallback override**: a runtime fallback writes `providerOverride`, `modelOverride`, `modelOverrideSource: "auto"`, and the selected origin model before retrying. That auto override can keep walking the configured fallback chain without probing the primary on every message, but OpenClaw periodically probes the configured origin again and clears the auto override when it recovers. `/new`, `/reset`, and `sessions.reset` also clear auto-sourced overrides. Heartbeat runs without an explicit `heartbeat.model` clear direct auto overrides when their origin no longer matches the current configured default.
6464
- **User session override**: `/model`, the model picker, `session_status(model=...)`, and `sessions.patch` write `modelOverrideSource: "user"`. That is an exact session selection. If the selected provider/model fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated configured fallback.
6565
- **Legacy session override**: older session entries may have `modelOverride` without `modelOverrideSource`. OpenClaw treats those as user overrides so an explicit old selection is not silently converted into fallback behavior.
6666
- **Cron payload model**: a cron job `payload.model` / `--model` is a job primary, not a user session override. It uses configured fallbacks unless the job provides `payload.fallbacks`; `payload.fallbacks: []` makes the cron run strict.
@@ -305,7 +305,7 @@ That means fallback retries have to coordinate with live model switching:
305305
- System-driven model changes such as fallback rotation, heartbeat overrides, or compaction never mark a pending live switch on their own.
306306
- User-driven model overrides are treated as exact selections for fallback policy, so an unreachable selected provider surfaces as a failure instead of being masked by `agents.defaults.model.fallbacks`.
307307
- Before a fallback retry starts, the reply runner persists the selected fallback override fields to the session entry.
308-
- Auto fallback overrides remain selected on subsequent turns so OpenClaw does not probe a known-bad primary on every message. `/new`, `/reset`, and `sessions.reset` clear auto-sourced overrides and return the session to the configured default.
308+
- Auto fallback overrides remain selected on subsequent turns so OpenClaw does not probe a known-bad primary on every message. OpenClaw periodically probes the configured origin again and clears the auto override when it recovers; `/new`, `/reset`, and `sessions.reset` clear auto-sourced overrides immediately.
309309
- `/status` shows the selected model and, when fallback state differs, the active fallback model and reason.
310310
- Live-session reconciliation prefers persisted session overrides over stale runtime model fields.
311311
- If a live-switch error points at a later candidate in the active fallback chain, OpenClaw jumps directly to that selected model instead of walking unrelated candidates first.

‎docs/concepts/models.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ OpenClaw selects models in this order:
5959
The same `provider/model` can mean different things depending on where it came from:
6060

6161
- Configured defaults (`agents.defaults.model.primary` and agent-specific primaries) are the normal starting point and use `agents.defaults.model.fallbacks`.
62-
- Auto fallback selections are temporary recovery state. They are stored with `modelOverrideSource: "auto"` so later turns can keep using the fallback chain without probing a known-bad primary first.
62+
- Auto fallback selections are temporary recovery state. They are stored with `modelOverrideSource: "auto"` so later turns can keep using the fallback chain without probing a known-bad primary every time; OpenClaw periodically probes the original primary again and clears the auto selection when it recovers.
6363
- User session selections are exact. `/model`, the model picker, `session_status(model=...)`, and `sessions.patch` store `modelOverrideSource: "user"`; if that selected provider/model is unreachable, OpenClaw fails visibly instead of falling through to another configured model.
6464
- Cron `--model` / payload `model` is a per-job primary. It still uses configured fallbacks unless the job supplies explicit payload `fallbacks` (use `fallbacks: []` for a strict cron run).
6565
- CLI default-model and allowlist pickers respect `models.mode: "replace"` by listing explicit `models.providers.*.models` instead of loading the full built-in catalog.

‎src/agents/agent-command.live-model-switch.test.ts‎

Lines changed: 130 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
2+
import type { SessionEntry } from "../config/sessions.js";
23
import { INTERNAL_RUNTIME_CONTEXT_BEGIN, INTERNAL_RUNTIME_CONTEXT_END } from "./internal-events.js";
34
import { LiveSessionModelSwitchError } from "./live-model-switch-error.js";
45

@@ -33,6 +34,7 @@ const state = vi.hoisted(() => ({
3334
deliverAgentCommandResultMock: vi.fn(),
3435
trajectoryRecordEventMock: vi.fn(),
3536
trajectoryFlushMock: vi.fn(async () => undefined),
37+
persistSessionEntryMock: vi.fn(async (..._args: unknown[]): Promise<unknown> => undefined),
3638
clearSessionAuthProfileOverrideMock: vi.fn(),
3739
isThinkingLevelSupportedMock: vi.fn((_args: unknown) => true),
3840
resolveThinkingDefaultMock: vi.fn((_args: unknown) => "low"),
@@ -46,6 +48,7 @@ const state = vi.hoisted(() => ({
4648
authProfileStoreMock: { profiles: {} } as { profiles: Record<string, unknown> },
4749
sessionEntryMock: undefined as unknown,
4850
sessionStoreMock: undefined as unknown,
51+
storePathMock: undefined as string | undefined,
4952
}));
5053

5154
vi.mock("./model-fallback.js", () => ({
@@ -66,6 +69,16 @@ vi.mock("./command/attempt-execution.runtime.js", () => ({
6669
sessionFileHasContent: vi.fn(async () => false),
6770
}));
6871

72+
vi.mock("./command/attempt-execution.shared.js", async () => {
73+
const actual = await vi.importActual<typeof import("./command/attempt-execution.shared.js")>(
74+
"./command/attempt-execution.shared.js",
75+
);
76+
return {
77+
...actual,
78+
persistSessionEntry: (...args: unknown[]) => state.persistSessionEntryMock(...args),
79+
};
80+
});
81+
6982
vi.mock("./command/delivery.runtime.js", () => ({
7083
deliverAgentCommandResult: (...args: unknown[]) => state.deliverAgentCommandResultMock(...args),
7184
}));
@@ -99,7 +112,7 @@ vi.mock("./command/session.js", () => ({
99112
skillsSnapshot: { prompt: "", skills: [], version: 0 },
100113
},
101114
sessionStore: state.sessionStoreMock,
102-
storePath: undefined,
115+
storePath: state.storePathMock,
103116
isNewSession: false,
104117
persistedThinking: undefined,
105118
persistedVerbose: undefined,
@@ -108,6 +121,10 @@ vi.mock("./command/session.js", () => ({
108121

109122
vi.mock("./command/types.js", () => ({}));
110123

124+
vi.mock("./harness/runtime-plugin.js", () => ({
125+
ensureSelectedAgentHarnessPlugin: vi.fn(async () => undefined),
126+
}));
127+
111128
vi.mock("../acp/policy.js", () => ({
112129
resolveAcpAgentPolicyError: (...args: unknown[]) => state.resolveAcpAgentPolicyErrorMock(...args),
113130
resolveAcpDispatchPolicyError: (...args: unknown[]) =>
@@ -266,9 +283,13 @@ vi.mock("../utils/message-channel.js", () => ({
266283
}));
267284

268285
vi.mock("./agent-scope.js", () => ({
286+
clearAutoFallbackPrimaryProbeSelection: vi.fn(),
287+
entryMatchesAutoFallbackPrimaryProbe: () => true,
269288
hasSessionAutoModelFallbackProvenance: () => false,
270289
listAgentEntries: () => [],
271290
listAgentIds: () => ["default"],
291+
markAutoFallbackPrimaryProbe: vi.fn(),
292+
resolveAutoFallbackPrimaryProbe: () => undefined,
272293
resolveAgentConfig: () => undefined,
273294
resolveAgentDir: () => "/tmp/agent",
274295
resolveEffectiveModelFallbacks: state.resolveEffectiveModelFallbacksMock,
@@ -735,6 +756,30 @@ describe("agentCommand – LiveSessionModelSwitchError retry", () => {
735756
state.authProfileStoreMock = { profiles: {} };
736757
state.sessionEntryMock = undefined;
737758
state.sessionStoreMock = undefined;
759+
state.storePathMock = undefined;
760+
state.persistSessionEntryMock.mockImplementation(async (...args: unknown[]) => {
761+
const params = args[0] as {
762+
sessionStore?: Record<string, unknown>;
763+
sessionKey?: string;
764+
entry?: unknown;
765+
shouldPersist?: (entry: unknown) => boolean;
766+
};
767+
const current =
768+
params.sessionStore && params.sessionKey
769+
? params.sessionStore[params.sessionKey]
770+
: undefined;
771+
if (params.shouldPersist && !params.shouldPersist(current)) {
772+
if (current === undefined && params.sessionStore && params.sessionKey) {
773+
delete params.sessionStore[params.sessionKey];
774+
}
775+
return current;
776+
}
777+
if (params.sessionStore && params.sessionKey && params.entry) {
778+
params.sessionStore[params.sessionKey] = params.entry;
779+
return params.entry;
780+
}
781+
return current;
782+
});
738783
state.buildWorkspaceSkillSnapshotMock.mockReturnValue({
739784
prompt: "",
740785
skills: [],
@@ -1033,6 +1078,90 @@ describe("agentCommand – LiveSessionModelSwitchError retry", () => {
10331078
expect(state.runWithModelFallbackMock).toHaveBeenCalledTimes(2);
10341079
});
10351080

1081+
it("does not persist a user live switch as an auto fallback probe result", async () => {
1082+
const sessionEntry: SessionEntry = {
1083+
sessionId: "session-1",
1084+
updatedAt: Date.now(),
1085+
providerOverride: "openai",
1086+
modelOverride: "claude",
1087+
modelOverrideSource: "auto",
1088+
modelOverrideFallbackOriginProvider: "anthropic",
1089+
modelOverrideFallbackOriginModel: "claude",
1090+
skillsSnapshot: { prompt: "", skills: [], version: 0 },
1091+
};
1092+
state.sessionEntryMock = sessionEntry;
1093+
const sessionStore: Record<string, SessionEntry> = { "agent:main": sessionEntry };
1094+
state.sessionStoreMock = sessionStore;
1095+
state.storePathMock = "/tmp/openclaw-session-store.json";
1096+
setupModelSwitchRetry({
1097+
provider: "openai",
1098+
model: "gpt-5.4",
1099+
authProfileId: "openai:primary",
1100+
authProfileIdSource: "user",
1101+
});
1102+
state.runAgentAttemptMock.mockResolvedValue(makeSuccessResult("openai", "gpt-5.4"));
1103+
1104+
await runBasicAgentCommand();
1105+
1106+
const autoPinnedSwitchWrites = state.persistSessionEntryMock.mock.calls.filter((call) => {
1107+
const entry = (call[0] as { entry?: Record<string, unknown> } | undefined)?.entry;
1108+
return (
1109+
entry?.providerOverride === "openai" &&
1110+
entry?.modelOverride === "gpt-5.4" &&
1111+
entry?.modelOverrideSource === "auto" &&
1112+
entry?.modelOverrideFallbackOriginProvider === "anthropic"
1113+
);
1114+
});
1115+
expect(autoPinnedSwitchWrites).toHaveLength(0);
1116+
expectRecordFields(mockCallArg(state.updateSessionStoreAfterAgentRunMock), {
1117+
fallbackProvider: "openai",
1118+
fallbackModel: "gpt-5.4",
1119+
});
1120+
});
1121+
1122+
it("does not overwrite a concurrent user model switch after a primary probe", async () => {
1123+
const sessionEntry: SessionEntry = {
1124+
sessionId: "session-1",
1125+
updatedAt: Date.now(),
1126+
providerOverride: "openai",
1127+
modelOverride: "claude",
1128+
modelOverrideSource: "auto",
1129+
modelOverrideFallbackOriginProvider: "anthropic",
1130+
modelOverrideFallbackOriginModel: "claude",
1131+
skillsSnapshot: { prompt: "", skills: [], version: 0 },
1132+
};
1133+
state.sessionEntryMock = sessionEntry;
1134+
const sessionStore: Record<string, SessionEntry> = { "agent:main": sessionEntry };
1135+
state.sessionStoreMock = sessionStore;
1136+
state.storePathMock = "/tmp/openclaw-session-store.json";
1137+
state.runWithModelFallbackMock.mockImplementationOnce(async (params: FallbackRunnerParams) => {
1138+
const result = await params.run(params.provider, params.model);
1139+
sessionStore["agent:main"] = {
1140+
sessionId: "session-1",
1141+
updatedAt: Date.now(),
1142+
providerOverride: "google",
1143+
modelOverride: "gemini-3-pro",
1144+
modelOverrideSource: "user",
1145+
skillsSnapshot: { prompt: "", skills: [], version: 0 },
1146+
};
1147+
return {
1148+
result,
1149+
provider: params.provider,
1150+
model: params.model,
1151+
attempts: [],
1152+
};
1153+
});
1154+
state.runAgentAttemptMock.mockResolvedValue(makeSuccessResult("anthropic", "claude"));
1155+
1156+
await runBasicAgentCommand();
1157+
1158+
expectRecordFields(sessionStore["agent:main"], {
1159+
providerOverride: "google",
1160+
modelOverride: "gemini-3-pro",
1161+
modelOverrideSource: "user",
1162+
});
1163+
});
1164+
10361165
it("keeps aliased session auth profiles for codex-cli runs", async () => {
10371166
let capturedAuthProfileProvider: string | undefined;
10381167
const sessionEntry = {

0 commit comments

Comments
 (0)