Skip to content

Commit 3001be3

Browse files
Jetjet
Jet
authored and
jet
committed
Add resilient fallback policy for user model overrides
Co-Authored-By: Cha <[email protected]> via OpenClaw
1 parent 800a0d3 commit 3001be3

14 files changed

Lines changed: 273 additions & 37 deletions

‎docs/concepts/model-failover.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ OpenClaw separates the selected provider/model from why it was selected. That so
6161
- **Configured default**: `agents.defaults.model.primary` uses `agents.defaults.model.fallbacks`.
6262
- **Agent primary**: `agents.list[].model` is strict unless that agent model object includes its own `fallbacks`. Use `fallbacks: []` to make the strict behavior explicit, or provide a non-empty list to opt that agent into model fallback.
6363
- **Auto fallback override**: a runtime fallback writes `providerOverride`, `modelOverride`, `modelOverrideSource: "auto"`, and the selected origin model before retrying. That auto override can keep walking the configured fallback chain without probing the primary on every message, but OpenClaw periodically probes the configured origin again and clears the auto override when it recovers. `/new`, `/reset`, and `sessions.reset` also clear auto-sourced overrides. Heartbeat runs without an explicit `heartbeat.model` clear direct auto overrides when their origin no longer matches the current configured default.
64-
- **User session override**: `/model`, the model picker, `session_status(model=...)`, and `sessions.patch` write `modelOverrideSource: "user"`. That is an exact session selection. If the selected provider/model fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated configured fallback.
64+
- **User session override**: `/model`, the model picker, `session_status(model=...)`, and `sessions.patch` write `modelOverrideSource: "user"`. That is an exact session selection by default. If the selected provider/model fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated configured fallback. Set `agents.defaults.model.userOverrideFallbackPolicy: "resilient"` when user-selected models should retry through the configured fallback chain before failing the turn. Agent model objects can set `userOverrideFallbackPolicy` to override the default for that agent.
6565
- **Legacy session override**: older session entries may have `modelOverride` without `modelOverrideSource`. OpenClaw treats those as user overrides so an explicit old selection is not silently converted into fallback behavior.
6666
- **Cron payload model**: a cron job `payload.model` / `--model` is a job primary, not a user session override. It uses configured fallbacks unless the job provides `payload.fallbacks`; `payload.fallbacks: []` makes the cron run strict.
6767

@@ -259,7 +259,7 @@ If all profiles for a provider fail, OpenClaw moves to the next model in `agents
259259

260260
Overloaded and rate-limit errors are handled more aggressively than billing cooldowns. By default, OpenClaw allows one same-provider auth-profile retry, then switches to the next configured model fallback without waiting. Provider-busy signals such as `ModelNotReadyException` land in that overloaded bucket. Tune this with `auth.cooldowns.overloadedProfileRotations`, `auth.cooldowns.overloadedBackoffMs`, and `auth.cooldowns.rateLimitedProfileRotations`.
261261

262-
When a run starts from the configured default primary, a cron job primary, an agent primary with explicit fallbacks, or an auto-selected fallback override, OpenClaw can walk the matching configured fallback chain. Agent primaries without explicit fallbacks and explicit user selections (for example `/model ollama/qwen3.5:27b`, the model picker, `sessions.patch`, or one-off CLI provider/model overrides) are strict: if that provider/model is unreachable or fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated fallback.
262+
When a run starts from the configured default primary, a cron job primary, an agent primary with explicit fallbacks, or an auto-selected fallback override, OpenClaw can walk the matching configured fallback chain. Agent primaries without explicit fallbacks and explicit user selections (for example `/model ollama/qwen3.5:27b`, the model picker, `sessions.patch`, or one-off CLI provider/model overrides) are strict by default: if that provider/model is unreachable or fails before producing a reply, OpenClaw reports the failure instead of answering from an unrelated fallback. To keep explicit user model choices resilient, set `agents.defaults.model.userOverrideFallbackPolicy: "resilient"` or set `userOverrideFallbackPolicy` on an agent's model object; per-agent model fallbacks and explicit `fallbacks: []` still control that agent's allowed fallback chain.
263263

264264
### Candidate chain rules
265265

‎docs/gateway/config-agents.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1080,7 +1080,7 @@ for provider examples and precedence.
10801080

10811081
- `id`: stable agent id (required).
10821082
- `default`: when multiple are set, first wins (warning logged). If none set, first list entry is default.
1083-
- `model`: string form sets a strict per-agent primary with no model fallback; object form `{ primary }` is also strict unless you add `fallbacks`. Use `{ primary, fallbacks: [...] }` to opt that agent into fallback, or `{ primary, fallbacks: [] }` to make strict behavior explicit. Cron jobs that only override `primary` still inherit default fallbacks unless you set `fallbacks: []`.
1083+
- `model`: string form sets a strict per-agent primary with no model fallback; object form `{ primary }` is also strict unless you add `fallbacks`. Use `{ primary, fallbacks: [...] }` to opt that agent into fallback, or `{ primary, fallbacks: [] }` to make strict behavior explicit. Cron jobs that only override `primary` still inherit default fallbacks unless you set `fallbacks: []`. User session model selections are strict by default; set `agents.defaults.model.userOverrideFallbackPolicy: "resilient"` or `agents.list[].model.userOverrideFallbackPolicy` when `/model` and model-picker selections should keep using configured fallbacks on provider failures.
10841084
- `params`: per-agent stream params merged over the selected model entry in `agents.defaults.models`. Use this for agent-specific overrides like `cacheRetention`, `temperature`, or `maxTokens` without duplicating the whole model catalog.
10851085
- `tts`: optional per-agent text-to-speech overrides. The block deep-merges over `messages.tts`, so keep shared provider credentials and fallback policy in `messages.tts` and set only persona-specific values such as provider, voice, model, style, or auto mode here.
10861086
- `skills`: optional per-agent skill allowlist. If omitted, the agent inherits `agents.defaults.skills` when set; an explicit list replaces defaults instead of merging, and `[]` means no skills.

‎src/agents/agent-scope.test.ts‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,116 @@ describe("resolveAgentConfig", () => {
381381
).toStrictEqual([]);
382382
});
383383

384+
it("allows user session model overrides to inherit configured fallbacks when opted in", () => {
385+
const cfg: OpenClawConfig = {
386+
agents: {
387+
defaults: {
388+
model: {
389+
primary: "openai/gpt-5.4",
390+
fallbacks: ["anthropic/claude-haiku-3-5", "google/gemini-2.5-flash"],
391+
userOverrideFallbackPolicy: "resilient",
392+
},
393+
},
394+
list: [{ id: "linus" }],
395+
},
396+
};
397+
398+
expect(
399+
resolveEffectiveModelFallbacks({
400+
cfg,
401+
agentId: "linus",
402+
hasSessionModelOverride: true,
403+
modelOverrideSource: "user",
404+
}),
405+
).toEqual(["anthropic/claude-haiku-3-5", "google/gemini-2.5-flash"]);
406+
expect(
407+
resolveEffectiveModelFallbacks({
408+
cfg,
409+
agentId: "linus",
410+
hasSessionModelOverride: true,
411+
}),
412+
).toEqual(["anthropic/claude-haiku-3-5", "google/gemini-2.5-flash"]);
413+
});
414+
415+
it("keeps explicit agent fallback overrides authoritative for resilient user selections", () => {
416+
const cfg: OpenClawConfig = {
417+
agents: {
418+
defaults: {
419+
model: {
420+
primary: "openai/gpt-5.4",
421+
fallbacks: ["google/gemini-2.5-flash"],
422+
userOverrideFallbackPolicy: "resilient",
423+
},
424+
},
425+
list: [
426+
{
427+
id: "linus",
428+
model: {
429+
primary: "anthropic/claude-sonnet-4-6",
430+
fallbacks: ["openrouter/meta-llama/llama-3.3-70b-instruct"],
431+
},
432+
},
433+
{
434+
id: "strict",
435+
model: {
436+
primary: "anthropic/claude-sonnet-4-6",
437+
},
438+
},
439+
],
440+
},
441+
};
442+
443+
expect(
444+
resolveEffectiveModelFallbacks({
445+
cfg,
446+
agentId: "linus",
447+
hasSessionModelOverride: true,
448+
modelOverrideSource: "user",
449+
}),
450+
).toEqual(["openrouter/meta-llama/llama-3.3-70b-instruct"]);
451+
expect(
452+
resolveEffectiveModelFallbacks({
453+
cfg,
454+
agentId: "strict",
455+
hasSessionModelOverride: true,
456+
modelOverrideSource: "user",
457+
}),
458+
).toStrictEqual([]);
459+
});
460+
461+
it("lets an agent override the default user fallback policy", () => {
462+
const cfg: OpenClawConfig = {
463+
agents: {
464+
defaults: {
465+
model: {
466+
primary: "openai/gpt-5.4",
467+
fallbacks: ["google/gemini-2.5-flash"],
468+
userOverrideFallbackPolicy: "strict",
469+
},
470+
},
471+
list: [
472+
{
473+
id: "linus",
474+
model: {
475+
primary: "anthropic/claude-sonnet-4-6",
476+
fallbacks: ["openrouter/meta-llama/llama-3.3-70b-instruct"],
477+
userOverrideFallbackPolicy: "resilient",
478+
},
479+
},
480+
],
481+
},
482+
};
483+
484+
expect(
485+
resolveEffectiveModelFallbacks({
486+
cfg,
487+
agentId: "linus",
488+
hasSessionModelOverride: true,
489+
modelOverrideSource: "user",
490+
}),
491+
).toEqual(["openrouter/meta-llama/llama-3.3-70b-instruct"]);
492+
});
493+
384494
it("updates the effective model primary at the winning config layer", () => {
385495
const cfg: OpenClawConfig = {
386496
agents: {

‎src/agents/agent-scope.ts‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,17 @@
11
import fs from "node:fs";
22
import path from "node:path";
3-
import { resolveAgentModelFallbackValues } from "../config/model-input.js";
3+
import {
4+
resolveAgentModelFallbackValues,
5+
resolveUserModelOverrideFallbackPolicy,
6+
} from "../config/model-input.js";
47
import { hasSessionAutoModelFallbackProvenance } from "../config/sessions/model-override-provenance.js";
58
export { hasSessionAutoModelFallbackProvenance } from "../config/sessions/model-override-provenance.js";
69
import type { SessionEntry } from "../config/sessions/types.js";
710
import type { AgentDefaultsConfig } from "../config/types.agent-defaults.js";
8-
import type { AgentModelConfig } from "../config/types.agents-shared.js";
11+
import type {
12+
AgentModelConfig,
13+
UserModelOverrideFallbackPolicy,
14+
} from "../config/types.agents-shared.js";
915
import type { AgentConfig } from "../config/types.agents.js";
1016
import type { OpenClawConfig } from "../config/types.js";
1117
import { isPathInside } from "../infra/path-guards.js";
@@ -497,6 +503,18 @@ export function hasConfiguredModelFallbacks(params: {
497503
return (fallbacksOverride ?? defaultFallbacks).length > 0;
498504
}
499505

506+
export function resolveAgentUserModelOverrideFallbackPolicy(
507+
cfg: OpenClawConfig,
508+
agentId: string,
509+
): UserModelOverrideFallbackPolicy {
510+
const agentPolicy = resolveUserModelOverrideFallbackPolicy(
511+
resolveAgentConfig(cfg, agentId)?.model,
512+
);
513+
return (
514+
agentPolicy ?? resolveUserModelOverrideFallbackPolicy(cfg.agents?.defaults?.model) ?? "strict"
515+
);
516+
}
517+
500518
export function resolveEffectiveModelFallbacks(params: {
501519
cfg: OpenClawConfig;
502520
agentId: string;
@@ -508,13 +526,15 @@ export function resolveEffectiveModelFallbacks(params: {
508526
if (!params.hasSessionModelOverride) {
509527
return agentFallbacksOverride;
510528
}
529+
const defaultFallbacks = resolveAgentModelFallbackValues(params.cfg.agents?.defaults?.model);
511530
const canUseConfiguredFallbacks =
512531
params.modelOverrideSource === "auto" ||
513532
(params.modelOverrideSource === undefined && params.hasAutoFallbackProvenance === true);
514533
if (!canUseConfiguredFallbacks) {
515-
return [];
534+
return resolveAgentUserModelOverrideFallbackPolicy(params.cfg, params.agentId) === "resilient"
535+
? (agentFallbacksOverride ?? defaultFallbacks)
536+
: [];
516537
}
517-
const defaultFallbacks = resolveAgentModelFallbackValues(params.cfg.agents?.defaults?.model);
518538
return agentFallbacksOverride ?? defaultFallbacks;
519539
}
520540

‎src/config/model-input.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,11 @@ import {
55
normalizeOptionalString,
66
resolvePrimaryStringValue,
77
} from "../shared/string-coerce.js";
8-
import type { AgentModelConfig } from "./types.agents-shared.js";
8+
import type {
9+
AgentChatModelConfig,
10+
AgentModelConfig,
11+
UserModelOverrideFallbackPolicy,
12+
} from "./types.agents-shared.js";
913

1014
type AgentModelListLike = {
1115
primary?: string;
@@ -55,6 +59,18 @@ export function resolveAgentModelTimeoutMsValue(model?: AgentModelConfig): numbe
5559
: undefined;
5660
}
5761

62+
export function resolveUserModelOverrideFallbackPolicy(
63+
model?: AgentChatModelConfig,
64+
): UserModelOverrideFallbackPolicy | undefined {
65+
if (!model || typeof model !== "object") {
66+
return undefined;
67+
}
68+
return model.userOverrideFallbackPolicy === "resilient" ||
69+
model.userOverrideFallbackPolicy === "strict"
70+
? model.userOverrideFallbackPolicy
71+
: undefined;
72+
}
73+
5874
export function toAgentModelListLike(model?: AgentModelConfig): AgentModelListLike | undefined {
5975
if (typeof model === "string") {
6076
const primary = normalizeOptionalString(model);

‎src/config/schema.help.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1372,6 +1372,10 @@ export const FIELD_HELP: Record<string, string> = {
13721372
"agents.defaults.model.primary": "Primary model (provider/model).",
13731373
"agents.defaults.model.fallbacks":
13741374
"Ordered fallback models (provider/model). Used when the primary model fails.",
1375+
"agents.defaults.model.userOverrideFallbackPolicy":
1376+
'Controls whether explicit user session model choices may use configured fallbacks. "strict" keeps current exact-selection behavior; "resilient" retries through configured fallbacks when the selected model fails.',
1377+
"agents.list.*.model.userOverrideFallbackPolicy":
1378+
'Overrides agents.defaults.model.userOverrideFallbackPolicy for one agent. Use "resilient" when explicit user model choices for this agent should retry through that agent model fallback chain.',
13751379
"agents.defaults.agentRuntime":
13761380
"Legacy whole-agent runtime policy. It is ignored by runtime selection; configure runtime policy on a provider or model instead. Run openclaw doctor --fix to remove stale values.",
13771381
"agents.defaults.agentRuntime.id":

‎src/config/schema.labels.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ export const FIELD_LABELS: Record<string, string> = {
103103
"agents.list[].contextLimits.toolResultMaxChars": "Agent Tool Result Max Chars",
104104
"agents.list[].contextLimits.postCompactionMaxChars": "Agent Post-compaction Max Chars",
105105
"agents.list.*.models": "Agent Model Overrides",
106+
"agents.list.*.model.userOverrideFallbackPolicy": "Agent User Override Fallback Policy",
106107
"agents.list.*.models.*.agentRuntime": "Agent Model Runtime",
107108
"agents.list.*.models.*.agentRuntime.id": "Agent Model Runtime ID",
108109
"agents.list.*.agentRuntime": "Legacy Agent Runtime",
@@ -637,6 +638,7 @@ export const FIELD_LABELS: Record<string, string> = {
637638
"agents.defaults.models.*.agentRuntime.id": "Default Agent Model Runtime ID",
638639
"agents.defaults.model.primary": "Primary Model",
639640
"agents.defaults.model.fallbacks": "Model Fallbacks",
641+
"agents.defaults.model.userOverrideFallbackPolicy": "User Override Fallback Policy",
640642
"agents.defaults.imageModel.primary": "Image Model",
641643
"agents.defaults.imageModel.fallbacks": "Image Model Fallbacks",
642644
"agents.defaults.imageGenerationModel.primary": "Image Generation Model",

‎src/config/types.agent-defaults.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import type { SilentReplyPolicyShape } from "../shared/silent-reply-policy.js";
22
import type {
33
AgentEmbeddedHarnessConfig,
4+
AgentChatModelConfig,
45
AgentModelConfig,
56
AgentRuntimePolicyConfig,
67
AgentSandboxConfig,
@@ -204,8 +205,8 @@ export type AgentDefaultsConfig = {
204205
agentRuntime?: AgentRuntimePolicyConfig;
205206
/** @deprecated Use agentRuntime. */
206207
embeddedHarness?: AgentEmbeddedHarnessConfig;
207-
/** Primary model and fallbacks (provider/model). Accepts string or {primary,fallbacks}. */
208-
model?: AgentModelConfig;
208+
/** Primary chat model and fallbacks (provider/model). Accepts string or {primary,fallbacks}. */
209+
model?: AgentChatModelConfig;
209210
/** Optional image-capable model and fallbacks (provider/model). Accepts string or {primary,fallbacks}. */
210211
imageModel?: AgentModelConfig;
211212
/** Optional image-generation model and fallbacks (provider/model). Accepts string or {primary,fallbacks}. */

‎src/config/types.agents-shared.ts‎

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,25 @@ import type {
55
SandboxSshSettings,
66
} from "./types.sandbox.js";
77

8-
export type AgentModelConfig =
8+
export type UserModelOverrideFallbackPolicy = "strict" | "resilient";
9+
10+
export type AgentModelObjectConfig = {
11+
/** Primary model (provider/model). */
12+
primary?: string;
13+
/** Per-agent model fallbacks (provider/model). */
14+
fallbacks?: string[];
15+
/** Optional provider request timeout in milliseconds for capabilities that support it. */
16+
timeoutMs?: number;
17+
};
18+
19+
export type AgentModelConfig = string | AgentModelObjectConfig;
20+
21+
export type AgentChatModelConfig =
922
| string
10-
| {
11-
/** Primary model (provider/model). */
12-
primary?: string;
13-
/** Per-agent model fallbacks (provider/model). */
14-
fallbacks?: string[];
15-
/** Optional provider request timeout in milliseconds for capabilities that support it. */
16-
timeoutMs?: number;
17-
};
23+
| (AgentModelObjectConfig & {
24+
/** Whether explicit user session model overrides may use configured fallbacks. */
25+
userOverrideFallbackPolicy?: UserModelOverrideFallbackPolicy;
26+
});
1827

1928
export type AgentEmbeddedHarnessConfig = {
2029
/** Agent runtime id. Omitted uses "pi"; "auto" opts into plugin harness auto-selection. */

‎src/config/types.agents.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import type {
88
} from "./types.agent-defaults.js";
99
import type {
1010
AgentEmbeddedHarnessConfig,
11+
AgentChatModelConfig,
1112
AgentModelConfig,
1213
AgentRuntimePolicyConfig,
1314
AgentSandboxConfig,
@@ -89,7 +90,8 @@ export type AgentConfig = {
8990
agentRuntime?: AgentRuntimePolicyConfig;
9091
/** @deprecated Use agentRuntime. */
9192
embeddedHarness?: AgentEmbeddedHarnessConfig;
92-
model?: AgentModelConfig;
93+
/** Per-agent chat model and fallback behavior. */
94+
model?: AgentChatModelConfig;
9395
/** Per-model metadata overrides for this agent. */
9496
models?: Record<string, AgentModelEntryConfig>;
9597
/** @deprecated Legacy per-agent compaction config is kept for raw doctor migration/repair. */

0 commit comments

Comments
 (0)