fix(agents): make model pins survive fallback, hot reloads, and stale auth profiles#103417
fix(agents): make model pins survive fallback, hot reloads, and stale auth profiles#103417obviyus wants to merge 3 commits into
Conversation
|
Superseded by #103510, which landed as d133f28. The landed replacement preserves your turn-local fallback commit with original authorship and incorporates the root-cause work from this PR. It moves config freshness to the Telegram authorization/assembled-turn boundary so one exact live snapshot flows through routing and dispatch, instead of adding a late global config read inside generic reply dispatch. It also leaves configured OAuth profile order unchanged because globally demoting expired-but-refreshable profiles can override explicit identity/account precedence. Exact-head CI passed 67 checks, including the model-pin, hot-reload, callback authorization, grouped-media, native-command, frozen-config, fallback, type, lint, build, and security lanes. Thank you @obviyus for the incident analysis, design invariants, and the clean net-negative fallback work; those materially shaped the landed fix. |
What Problem This Solves
A production incident (#103324) showed three compounding failures that make
/modelunreliable: a user pinnedopenai/gpt-5.6-lunafor a Telegram session, the command confirmed and/statusshowed the pin — then the next turn silently ran a different provider and persisted the fallback over the user's pin. Root causes, each fixed here at its own boundary:modelOverride/providerOverride(source"auto"), so a transient availability failure permanently replaced the user's explicit selection.dispatchReplyFromConfigmarked the channel-boot config as a complete runtime config, so hot-reloaded config (config hot reload appliedin logs) never reached model allowlisting — models added after gateway start were "not in agents.defaults.models" to the run layer while the command layer accepted them.Closes #103324.
Why This Change Was Made
fix(agents): keep model fallback turn-local…): the entire fallback-selection persistence machinery (patch builders, rollback state,"auto"override provenance writers) is deleted — fallback can only record runtime facts (model,modelProvider,fallbackNotice*). The user's pin survives; the next turn retries it. The auto-fallback primary probe inagent-command.tslikewise no longer converts a successful fallback candidate into a persisted auto pin. Net −700 lines of prod+test code.fix(agents): resolve reply config from the live runtime snapshot…): dispatch buildsreplyConfigfromgetRuntimeConfig()(plus the explicitconfigOverridepatch when provided) and passes that same object to both reply resolution and dispatch callbacks — restoring single-config-authority per turn while making hot reload effective. A regression test replays the incident: a model added to config after channel startup resolves in the next turn, with config loaded exactly once.fix(agents): demote locally-expired OAuth profiles…):resolveAuthProfileOrdersorts locally-expired OAuth profiles behind healthy siblings so probes and rotation lead with refreshable-and-live credentials. Expired profiles stay selectable (OAuth refresh runs at use) and an explicitpreferredProfilestill wins over expiry, so targeted account selection never silently switches identity.Deliberately not done: issue #103324's suggestion 3 ("never persist
liveModelSwitchPendingwithout an override") turned out to be a misdiagnosis — flag-without-override is main's intentional, test-covered encoding for "pending switch back to the agent default" (applyModelOverrideToSessionEntryclears runtime model fields so live-switch resolution lands on the default). The orphaned-state harm in the incident came from the destroyed override, which fix 1 eliminates at the source. A contract comment now documents the encoding at the writer.User Impact
/modelselections stick. If the pinned model fails, the turn falls back with the existing visible fallback notice and the pin is retried next turn instead of being silently replaced. Models added to config at runtime are usable immediately after the hot reload instead of requiring a gateway restart. Multi-profile OpenAI/OAuth setups no longer lose model availability to one stale credential./model defaultmid-run keeps its live-restart-to-default behavior.Evidence
suppressToolErrorscallback policy following the runtime config (proving dispatch callbacks and reply resolution share one config), expired-profile demotion / all-expired attempt / expired-preferred-still-first.pnpm tsgo,pnpm check:test-types: exit 0. All touched test files green locally (node scripts/run-vitest.mjs, 5 shards)./model defaultlive-switch regression and expired-preferred-profile bypass — both fixed as above; final cycle clean ("no accepted/actionable findings").tbx_01kx56z3p5hvsjmph9te4e9szd(pnpm test:changed, 40 shards): all touched surfaces green remotely (dispatch-from-config256 tests,agent-runner-execution237,auth-profiles/order21,model-overridessuites across 4 shards). Three failures are unrelated to this diff and pass locally on this branch:ui/cron-page,ui/agent-select(locale assertion; no UI files touched), andclawrouter-managed-gateway.e2e(passes locally in 50s); the agents shard additionally hitWorker exited unexpectedlycrashes (box memory pressure)./modelretest after a gateway restart confirmed the stale-snapshot trigger).