Skip to content

fix(agents): make model pins survive fallback, hot reloads, and stale auth profiles#103417

Closed
obviyus wants to merge 3 commits into
mainfrom
fix/model-pin-fallback
Closed

fix(agents): make model pins survive fallback, hot reloads, and stale auth profiles#103417
obviyus wants to merge 3 commits into
mainfrom
fix/model-pin-fallback

Conversation

@obviyus

@obviyus obviyus commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

A production incident (#103324) showed three compounding failures that make /model unreliable: a user pinned openai/gpt-5.6-luna for a Telegram session, the command confirmed and /status showed the pin — then the next turn silently ran a different provider and persisted the fallback over the user's pin. Root causes, each fixed here at its own boundary:

  1. Fallback rotation persisted over user pins. The rotation machinery wrote modelOverride/providerOverride (source "auto"), so a transient availability failure permanently replaced the user's explicit selection.
  2. Reply resolution judged models against a config snapshot captured at channel startup. dispatchReplyFromConfig marked the channel-boot config as a complete runtime config, so hot-reloaded config (config hot reload applied in logs) never reached model allowlisting — models added after gateway start were "not in agents.defaults.models" to the run layer while the command layer accepted them.
  3. One expired OAuth profile could poison provider availability. With several profiles for a provider, an expired one participating in ordering could front availability probes and veto a model the account actually serves.

Closes #103324.

Why This Change Was Made

  • Fallback is now turn-local (fix(agents): keep model fallback turn-local…): the entire fallback-selection persistence machinery (patch builders, rollback state, "auto" override provenance writers) is deleted — fallback can only record runtime facts (model, modelProvider, fallbackNotice*). The user's pin survives; the next turn retries it. The auto-fallback primary probe in agent-command.ts likewise no longer converts a successful fallback candidate into a persisted auto pin. Net −700 lines of prod+test code.
  • Reply config is resolved from the live runtime snapshot once per turn (fix(agents): resolve reply config from the live runtime snapshot…): dispatch builds replyConfig from getRuntimeConfig() (plus the explicit configOverride patch when provided) and passes that same object to both reply resolution and dispatch callbacks — restoring single-config-authority per turn while making hot reload effective. A regression test replays the incident: a model added to config after channel startup resolves in the next turn, with config loaded exactly once.
  • Expired OAuth profiles are demoted, not removed (fix(agents): demote locally-expired OAuth profiles…): resolveAuthProfileOrder sorts locally-expired OAuth profiles behind healthy siblings so probes and rotation lead with refreshable-and-live credentials. Expired profiles stay selectable (OAuth refresh runs at use) and an explicit preferredProfile still wins over expiry, so targeted account selection never silently switches identity.

Deliberately not done: issue #103324's suggestion 3 ("never persist liveModelSwitchPending without an override") turned out to be a misdiagnosis — flag-without-override is main's intentional, test-covered encoding for "pending switch back to the agent default" (applyModelOverrideToSessionEntry clears runtime model fields so live-switch resolution lands on the default). The orphaned-state harm in the incident came from the destroyed override, which fix 1 eliminates at the source. A contract comment now documents the encoding at the writer.

User Impact

/model selections stick. If the pinned model fails, the turn falls back with the existing visible fallback notice and the pin is retried next turn instead of being silently replaced. Models added to config at runtime are usable immediately after the hot reload instead of requiring a gateway restart. Multi-profile OpenAI/OAuth setups no longer lose model availability to one stale credential. /model default mid-run keeps its live-restart-to-default behavior.

Evidence

  • Behavior tests per fix: pin preservation through failed turns (e2e + unit, replacing seven deleted tests that asserted the old persist-over-pin behavior), hot-added model resolution with single config load, suppressToolErrors callback policy following the runtime config (proving dispatch callbacks and reply resolution share one config), expired-profile demotion / all-expired attempt / expired-preferred-still-first.
  • pnpm tsgo, pnpm check:test-types: exit 0. All touched test files green locally (node scripts/run-vitest.mjs, 5 shards).
  • Structured review (repo autoreview helper, three cycles): first two cycles surfaced real findings — /model default live-switch regression and expired-preferred-profile bypass — both fixed as above; final cycle clean ("no accepted/actionable findings").
  • Blacksmith Testbox tbx_01kx56z3p5hvsjmph9te4e9szd (pnpm test:changed, 40 shards): all touched surfaces green remotely (dispatch-from-config 256 tests, agent-runner-execution 237, auth-profiles/order 21, model-overrides suites across 4 shards). Three failures are unrelated to this diff and pass locally on this branch: ui/cron-page, ui/agent-select (locale assertion; no UI files touched), and clawrouter-managed-gateway.e2e (passes locally in 50s); the agents shard additionally hit Worker exited unexpectedly crashes (box memory pressure).
  • Production repro validation of the incident chain is documented in [Bug]: Model fallback silently rewrites user session model pin; expired OAuth profile poisons availability probe #103324 (session-state forensics + live /model retest after a gateway restart confirmed the stale-snapshot trigger).

@obviyus
obviyus requested a review from a team as a code owner July 10, 2026 05:22
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: L maintainer Maintainer-authored PR labels Jul 10, 2026
@obviyus obviyus self-assigned this Jul 10, 2026
@steipete

Copy link
Copy Markdown
Contributor

Superseded by #103510, which landed as d133f28.

The landed replacement preserves your turn-local fallback commit with original authorship and incorporates the root-cause work from this PR. It moves config freshness to the Telegram authorization/assembled-turn boundary so one exact live snapshot flows through routing and dispatch, instead of adding a late global config read inside generic reply dispatch. It also leaves configured OAuth profile order unchanged because globally demoting expired-but-refreshable profiles can override explicit identity/account precedence.

Exact-head CI passed 67 checks, including the model-pin, hot-reload, callback authorization, grouped-media, native-command, frozen-config, fallback, type, lint, build, and security lanes. Thank you @obviyus for the incident analysis, design invariants, and the clean net-negative fallback work; those materially shaped the landed fix.

@steipete steipete closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Model fallback silently rewrites user session model pin; expired OAuth profile poisons availability probe

2 participants