Skip to content

fix: add resilient fallback policy for user model overrides#80181

Open
jetd1 wants to merge 1 commit into
openclaw:mainfrom
jetd1:fix/user-override-fallback-policy
Open

fix: add resilient fallback policy for user model overrides#80181
jetd1 wants to merge 1 commit into
openclaw:mainfrom
jetd1:fix/user-override-fallback-policy

Conversation

@jetd1

@jetd1 jetd1 commented May 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Describe the problem and fix in 2–5 bullets:

  • Problem: session/user model overrides always disabled configured fallback chains.
  • Why it matters: operators could configure provider fallback behavior, but a user-selected model had no fallback safety net during transient provider/model failures.
  • What changed: adds opt-in userOverrideFallbackPolicy: "resilient" on agents.defaults.model and per-agent chat model config, with schema/types/help/labels/docs and fallback-resolution tests.
  • What did NOT change (scope boundary): default behavior remains strict; auto-selected model overrides and configured agent primaries keep their existing fallback behavior; non-chat model config schemas remain unchanged.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

  • Closes N/A
  • Related N/A
  • This PR fixes a bug or regression

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: a user/session model override can now inherit the configured fallback chain when the new policy is explicitly set to resilient, while the default strict behavior still returns no fallbacks.
  • Real environment tested: local OpenClaw checkout on pop-os, Node v25.8.1, branch fix/user-override-fallback-policy.
  • Exact steps or command run after this patch:
    pnpm exec tsx -e 'import { resolveEffectiveModelFallbacks } from "./src/agents/agent-scope.ts"; const cfg={agents:{defaults:{model:{primary:"openai/gpt-5.4",fallbacks:["anthropic/claude-haiku-3-5","google/gemini-2.5-flash"],userOverrideFallbackPolicy:"resilient"}},list:[{id:"linus"}]}}; console.log("user override resilient =>", JSON.stringify(resolveEffectiveModelFallbacks({cfg,agentId:"linus",hasSessionModelOverride:true,modelOverrideSource:"user"}))); const strict={agents:{defaults:{model:{primary:"openai/gpt-5.4",fallbacks:["anthropic/claude-haiku-3-5"]}},list:[{id:"linus"}]}}; console.log("user override strict =>", JSON.stringify(resolveEffectiveModelFallbacks({cfg:strict,agentId:"linus",hasSessionModelOverride:true,modelOverrideSource:"user"})));'
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
    user override resilient => ["anthropic/claude-haiku-3-5","google/gemini-2.5-flash"]
    user override strict => []
    
  • Observed result after fix: resilient user overrides receive the configured fallback list; strict user overrides receive an empty fallback list.
  • What was not tested: live provider failover against real external model APIs, because that would require changing a running gateway config. The changed resolver and config schema were tested locally.
  • Before evidence (optional but encouraged): covered by the existing and new regression tests: the pre-change resolver returned [] for user overrides whenever hasSessionModelOverride was true.

Root Cause (if applicable)

For bug fixes or regressions, explain why this happened, not just what changed. Otherwise write N/A. If the cause is unclear, write Unknown.

  • Root cause: resolveEffectiveModelFallbacks treated every non-auto session model override as authoritative and returned an empty fallback list without a config escape hatch.
  • Missing detection / guardrail: no regression coverage for user-selected models that should intentionally keep operator-configured fallback chains.
  • Contributing context (if known): auto model overrides and user/session overrides were represented in the same resolver path, but operators need separate fallback policy for user choices.

Regression Test Plan (if applicable)

For bug fixes or regressions, name the smallest reliable test coverage that should catch this. Otherwise write N/A.

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/agents/agent-scope.test.ts, src/config/zod-schema.agent-defaults.test.ts
  • Scenario the test should lock in: user model overrides remain strict by default, can opt into resilient fallback inheritance, and per-agent policy overrides defaults.
  • Why this is the smallest reliable guardrail: the behavior is selected in the config schema and resolved in the agent-scope fallback helper before provider execution.
  • Existing test that already covers this (if any): N/A
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

Users can set userOverrideFallbackPolicy: "resilient" under agents.defaults.model or an individual agent's chat model block. The default is still strict.

Diagram (if applicable)

Before:
[user override] -> [fallbacks disabled]

After:
[user override] -> [strict policy] -> [fallbacks disabled]
[user override] -> [resilient policy] -> [configured fallbacks preserved]

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: pop-os Linux
  • Runtime/container: local Node v25.8.1 / pnpm
  • Model/provider: N/A, pure config + fallback resolver change
  • Integration/channel (if any): N/A
  • Relevant config (redacted): agents.defaults.model.userOverrideFallbackPolicy: "resilient"

Steps

  1. Configure fallback models under agents.defaults.model.fallbacks.
  2. Set agents.defaults.model.userOverrideFallbackPolicy: "resilient".
  3. Resolve fallbacks for a session/user model override.

Expected

  • Resilient user overrides inherit configured fallback models.
  • Strict/default user overrides keep fallback models disabled.

Actual

  • Matches expected in the local resolver smoke and targeted tests.

Evidence

Attach at least one:

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Verification commands run:

codex review --base origin/main
pnpm check:changed
pnpm test src/agents/agent-scope.test.ts src/config/zod-schema.agent-defaults.test.ts
pnpm config:docs:check && pnpm config:schema:check
git diff --check origin/main...HEAD

codex review --base origin/main result:

The change consistently adds the new user override fallback policy across config types, schema validation, labels/help, docs, and fallback resolution. I did not find a discrete correctness issue in the modified paths.

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios: strict default, resilient defaults, per-agent policy override, explicit fallback override behavior, config schema acceptance/rejection, generated config docs/schema consistency.
  • Edge cases checked: explicit empty fallback list still disables fallback inheritance; non-chat model schemas still reject the chat-only policy field.
  • What you did not verify: live external provider outage failover through a running gateway.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

If a bot review conversation is addressed by this PR, resolve that conversation yourself. Do not leave bot review conversation cleanup for maintainers.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) Yes, optional new config field
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps: set userOverrideFallbackPolicy: "resilient" only when user-selected models should keep configured fallback chains.

Risks and Mitigations

  • Risk: operators may expect resilient behavior after adding fallbacks but forget the new policy.
    • Mitigation: docs and schema help call out the strict default and the explicit opt-in field.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime agents Agent runtime and tooling size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 10, 2026
@clawsweeper

clawsweeper Bot commented May 10, 2026

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed July 3, 2026, 11:45 PM ET / 03:45 UTC.

Summary
The branch adds an opt-in userOverrideFallbackPolicy to chat model config, updates fallback resolution/schema/help/docs, and adds resolver/schema tests.

PR surface: Source +57, Tests +179, Docs 0. Total +236 across 14 files.

Reproducibility: yes. for the source-level behavior: current main returns [] for user-sourced session model overrides, and current docs/tests encode strict selected-model behavior. A full live provider-outage recovery path was not reproduced in this read-only review.

Review metrics: 1 noteworthy metric.

  • Config fallback policy surface: 1 optional policy added in 2 chat-model config locations. New config/default surfaces that affect provider/model fallback routing need explicit compatibility and upgrade review before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #84865
Summary: This PR is a competing opt-in candidate for the canonical user-selected model fallback policy issue; another open PR proposes a direct behavior change, and neither has merged.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Get maintainer approval for the opt-in fallback policy and exact default contract.
  • Rebase over current main while preserving subagent fallback handling and the current chat/tool model schema split.
  • Align sibling docs and legacy override behavior with the accepted policy.

Risk before merge

  • [P2] This adds a new operator-facing config/default surface for user-selected model fallback behavior, so maintainers need to approve the public product/API contract before merge.
  • [P1] With resilient enabled, a failed explicit user-selected model can retry through another configured provider or auth profile, changing cost, latency, privacy, and credential-routing expectations.
  • [P2] The branch is currently conflicting against main; a refresh must preserve the current chat/tool model schema split, subagent fallback handling, legacy override semantics, and sibling docs.

Maintainer options:

  1. Approve Policy Then Rebase (recommended)
    Have a maintainer approve the strict-default opt-in contract, then rebase while preserving current schema splits, subagent fallback behavior, docs, and legacy-selection semantics.
  2. Pause For Canonical Direction
    Pause this PR until maintainers choose between this opt-in policy and the direct fallback fix tracked by user-switched model has no fallback chain, causing session deadlock on provider outage #84865.
  3. Accept Operator-Owned Routing
    Maintainers can intentionally accept the provider/auth routing risk if final docs clearly state that resilient user selections may answer from another configured fallback provider/model.

Next step before merge

  • [P2] Manual review is needed because the unresolved work is product/API contract approval and rebase preservation rather than a mechanical repair.

Security
Cleared: No supply-chain, workflow, dependency, secrets, or new code-execution surface was introduced; the opt-in provider-routing concern is tracked as merge risk.

Review findings

  • [P2] Preserve the chat/tool model schema split — src/config/zod-schema.agent-model.ts:21
  • [P2] Update the sibling model-selection docs — docs/concepts/model-failover.md:64
  • [P2] Keep legacy overrides strict unless maintainers opt in — src/agents/agent-scope.ts:534-536
Review details

Best possible solution:

Maintainers should choose one user-selected model fallback contract, then land one rebased implementation with resolver behavior, schemas, docs, tests, and runtime proof aligned to that decision.

Do we have a high-confidence way to reproduce the issue?

Yes for the source-level behavior: current main returns [] for user-sourced session model overrides, and current docs/tests encode strict selected-model behavior. A full live provider-outage recovery path was not reproduced in this read-only review.

Is this the best way to solve the issue?

Unclear as-is: the strict-default opt-in policy is a plausible safer shape than changing every user selection, but it is not the best merge path until maintainers approve the product/API contract and the branch is rebased against current main.

Full review comments:

  • [P2] Preserve the chat/tool model schema split — src/config/zod-schema.agent-model.ts:21
    Current main keeps timeoutMs on tool/media model schemas only, but this PR's new chat schema also accepts it here. During rebase, add the fallback policy without re-allowing chat model timeoutMs.
    Confidence: 0.84
  • [P2] Update the sibling model-selection docs — docs/concepts/model-failover.md:64
    This PR updates the failover concept page, but docs/concepts/models.md still says user session selections never fall through to another configured model. If maintainers accept resilient, update that sibling concept page too.
    Confidence: 0.86
  • [P2] Keep legacy overrides strict unless maintainers opt in — src/agents/agent-scope.ts:534-536
    This branch also applies resilient to session overrides with no modelOverrideSource, while current docs treat those legacy entries as exact user selections. Restrict fallback inheritance to explicit user-sourced entries or update the legacy contract and tests with maintainer approval.
    Confidence: 0.81

Overall correctness: patch is incorrect
Overall confidence: 0.87

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9d68f877ac3e.

Label changes

Label justifications:

  • P2: This is a normal-priority provider/model fallback improvement with limited blast radius but non-trivial config and routing semantics.
  • merge-risk: 🚨 compatibility: The PR adds a user-facing config policy and changes how explicit user model selections can use configured fallback chains.
  • merge-risk: 🚨 auth-provider: The new policy directly affects provider/model and credential routing after failures of a user-selected model.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body includes copied terminal output from running the actual resolver after the patch, which is sufficient for resolver-level behavior; live provider outage failover remains untested.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied terminal output from running the actual resolver after the patch, which is sufficient for resolver-level behavior; live provider outage failover remains untested.
Evidence reviewed

PR surface:

Source +57, Tests +179, Docs 0. Total +236 across 14 files.

View PR surface stats
Area Files Added Removed Net
Source 10 89 32 +57
Tests 2 181 2 +179
Docs 2 3 3 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 14 273 37 +236

What I checked:

  • Root and scoped policy read: Root AGENTS.md and scoped docs/agents guides were read; root policy treats config/default additions and fallback behavior as compatibility-sensitive review surfaces. (AGENTS.md:13, 9d68f877ac3e)
  • Live PR state: Live GitHub reports head 3001be3 and mergeable CONFLICTING / mergeStateStatus DIRTY. (3001be3b71b9)
  • PR resolver behavior: The PR returns configured fallbacks for non-auto session overrides when the new policy resolves to resilient. (src/agents/agent-scope.ts:534, 3001be3b71b9)
  • Current main strict resolver: Current main returns an empty fallback list for session model overrides unless they are auto-sourced or recovered auto provenance. (src/agents/agent-scope.ts:567, 9d68f877ac3e)
  • Caller path: The agent command path forwards explicit run overrides as modelOverrideSource: "user" into resolveEffectiveModelFallbacks, so this resolver controls fallback candidates for user model choices. (src/agents/agent-command.ts:1832, 9d68f877ac3e)
  • Fallback runner contract: runWithModelFallback() treats a defined fallback override as authoritative; [] disables defaults while undefined allows default fallbacks. (src/agents/model-fallback.ts:1009, 9d68f877ac3e)

Likely related people:

  • steipete: Git history shows commits keeping user model switches and agent primaries strict and adding the current model schema split. (role: strict fallback policy author and config/docs contributor; confidence: high; commits: d2320e4d4b42, 12962dd883eb, e96428b008a9; files: src/agents/agent-scope.ts, docs/concepts/model-failover.md, docs/concepts/models.md)
  • neeravmakwana: Commit 711ab45 changed legacy auto fallback pin handling across the same session override and provenance boundary. (role: recent adjacent fallback-state contributor; confidence: medium; commits: 711ab45025a2; files: src/agents/agent-scope.ts, src/auto-reply/reply/model-selection.ts, src/commands/agent.test.ts)
  • vincentkoc: Recent commits changed model fallback and failover error-normalization paths used after the resolver chooses fallback candidates. (role: recent model fallback contributor; confidence: medium; commits: 80805ad7a583, aa3797c8d0d7, 160aad6fb359; files: src/agents/model-fallback.ts, src/agents/embedded-agent-runner/run/failover-policy.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@jetd1
jetd1 force-pushed the fix/user-override-fallback-policy branch from c323428 to dbd835c Compare May 12, 2026 18:09
@jetd1
jetd1 requested a review from a team as a code owner May 12, 2026 18:09
@openclaw-barnacle openclaw-barnacle Bot added channel: discord Channel integration: discord channel: googlechat Channel integration: googlechat channel: imessage Channel integration: imessage channel: line Channel integration: line channel: matrix Channel integration: matrix channel: mattermost Channel integration: mattermost channel: msteams Channel integration: msteams channel: nextcloud-talk Channel integration: nextcloud-talk channel: nostr Channel integration: nostr channel: signal Channel integration: signal channel: slack Channel integration: slack channel: telegram Channel integration: telegram channel: tlon Channel integration: tlon channel: voice-call Channel integration: voice-call channel: whatsapp-web Channel integration: whatsapp-web channel: zalo Channel integration: zalo channel: zalouser Channel integration: zalouser extensions: copilot-proxy Extension: copilot-proxy extensions: diagnostics-otel Extension: diagnostics-otel labels May 12, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@clawsweeper

clawsweeper Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix: add resilient fallback policy for user model overrides This is item 1/1 in the current shard. Shard 5/22.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: M stale Marked as stale due to inactivity status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant