-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Don't trigger model fallback when abort reason is the run's own timeout budget #60388
Copy link
Copy link
Closed as not planned
BingqingLyu/openclaw
#2114Closed as not planned
Copy link
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.Auth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Problem
When a cron job (or any embedded agent run) exceeds its configured
timeoutSeconds, the embedded runner callsabortRun(true), which setstimedOut=true. This triggers the model fallback cascade:scheduleAbortTimer(params.timeoutMs)firesabortRun(true)→timedOut = true"Profile X timed out. Trying next account..."fallback_modelrunWithModelFallbackcatches theFailoverError→ tries next candidate modelWhy this is counterproductive
When a run's own timeout budget fires, the task is out of time regardless of which model handles it. Falling back to another model with near-zero remaining budget is guaranteed to fail, wastes API calls, increases lane occupancy, and delays the final error delivery.
This is different from legitimate fallback triggers (provider down, rate limited, auth error) where trying another model makes sense.
Observed impact
On a fleet of ~100 cron jobs:
Suggested fix
In the embedded runner's abort handler, distinguish between:
scheduleAbortTimerfired) → skip model fallback, fail immediately with a clear timeout errorThis could be as simple as passing a flag from
abortRunthrough to the failover decision path indicatingreason=run_timeout_exceeded(vsreason=provider_timeout), and havingrunWithModelFallbackskip candidates when the reason is the run's own budget.Environment
pi-embedded-BYdcxQ5A.jsline ~38336 (scheduleAbortTimer) and line ~40158 (Profile timed out)resolveFallbackCandidatesline ~7126Workaround
Setting
agents.defaults.model.fallbacks = []reduces the cascade from 3-4 candidates to 2 (requested + global primary). Raising cron timeout budgets reduces the trigger frequency. Neither eliminates the issue.