-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
cron isolated-agent turn: pre-execution watchdog race with runEmbeddedAgent phase reporting #93530
Copy link
Copy link
Open
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Problem
Cron isolated-agent turns (e.g. a daily briefing at 09:00) can abort with
externalAbort: true,stopReason: "aborted", anderror: "LLM request failed."despite the model provider being healthy and no rate-limit being hit. The abort fires after ~6 minutes, well within the configuredtimeoutSeconds: 600(10 min) wall clock.Root Cause Analysis
The abort originates from the watchdog in
executeJobCoreWithTimeout(server-cron, line 1037). The watchdog'striggerTimeout()callsrunAbortController.abort(reason), which flows down torunEmbeddedAttempt→onExternalAbortSignal()→externalAbort=true.The specific watchdog that fires is the pre-execution timeout (
resolveCronAgentPreExecutionWatchdogMs→CRON_AGENT_PRE_EXECUTION_WATCHDOG_MS = 60s). It starts whennoteRunnerStarted(info)is called, but fails to clear because of a phase/stage gap in the communication betweenrunEmbeddedAgentand the cron watchdog.Detailed chain
executeCronRun→runEmbeddedAgentis called withtrigger: "cron"embedded-agent.js:hookRunner.hasHooks("before_agent_reply")returnsfalse(no hook registered for that event), the"before_agent_reply"phase is never emittednoteExecutionProgressdepends on this phase (mapped to stage"execution") to clear the pre-execution timer:triggerTimeout()→runAbortController.abort("cron: isolated agent run stalled before execution start")"model_resolution"arrives too late — the abort signal has already propagatedTiming
executeJobCoreWithTimeoutstart()→setupTimeoutId(60s)noteRunnerStarted→clearSetupTimeout(),startTimeout()(10 min),noteExecutionProgress(),startPreExecutionTimeout()(60s)setTimedOut→runAbortController.abort()externalAbort: true,"LLM request failed."Note: The
"before_agent_reply"phase is gated behindhookRunner?.hasHooks("before_agent_reply"). When no hooks register for that event, the watchdog never learns the agent entered execution phase, and pre-execution timeout dominates.Impact
payload.kind="agentTurn"on isolated sessions can abort prematurely when the model provider takes more than ~60 seconds from prompt injection to first model call (the pre-execution watchdog window)opencode-go/deepseek-v4-flashbut impacts any provider wherebefore_agent_replyhook is absentSuggested Fix
Two options:
Option A (robust): In the cron watchdog's
noteExecutionProgress, also clear pre-execution timeout when any phase with a defined stage arrives (not justexecutionstage). Any phase signal proves the runner is alive and making progress:Option B (targeted): In
runEmbeddedAgent, emit the"before_agent_reply"phase unconditionally for cron triggers (remove thehookRunner?.hasHooks("before_agent_reply")guard).I lean toward Option A since it is defensive against any future phase gap.
Environment
sessionTarget: "isolated",timeoutSeconds: 600