-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
cron isolated agent pre-execution watchdog aborts runs despite setup progress #93912
Copy link
Copy link
Open
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Summary
Isolated cron
agentTurnjobs can still be aborted by the 60s pre-execution watchdog even while the isolated runner is alive and reporting setup/context progress. This presents as intermittent failures such as:The related reports are #93530 and #93867. This issue narrows the remaining bug to the watchdog transition that treats setup-like phases as non-progress.
Root cause
src/cron/service/agent-watchdog.tsmaps phases intopre_executionandexecutionbuckets. The watchdog currently clears the pre-execution timer only when the phase is classified asexecutionorfirstModelCallStartedis set.That means a live isolated run can emit phases such as:
model_resolutionauthcontext_engineruntime_pluginsbut still remain in
waiting_for_execution. After 60s, the pre-execution watchdog aborts the run even though progress is being observed and the configured job timeout may be much longer.The fallback re-arm path makes this worse: after
before_agent_reply, a fallback/setup phase can move the state back towaiting_for_execution, causing the same false pre-execution timeout.Expected behavior
onExecutionStarted()with no subsequent phase should still be guarded by the pre-execution watchdog.Proposed fix
Treat any known watchdog phase as progress:
Also stop re-arming the pre-execution watchdog when fallback/setup phases follow
before_agent_reply; those phases are progress and should remain under the normal job timeout.Regression coverage
Add/adjust cron timer regression coverage for:
stalled before execution startafter the pre-execution watchdog.model_resolution,auth,context_engine): stays alive past 60s and only fails at the configured job timeout.before_agent_reply: does not re-arm the 60s pre-execution watchdog.