Skip to content

cron isolated agent pre-execution watchdog aborts runs despite setup progress #93912

Description

@parthjayaram

Summary

Isolated cron agentTurn jobs can still be aborted by the 60s pre-execution watchdog even while the isolated runner is alive and reporting setup/context progress. This presents as intermittent failures such as:

cron: isolated agent run stalled before execution start (last phase: context-engine)
cron: job execution timed out (last phase: model-call-started)

The related reports are #93530 and #93867. This issue narrows the remaining bug to the watchdog transition that treats setup-like phases as non-progress.

Root cause

src/cron/service/agent-watchdog.ts maps phases into pre_execution and execution buckets. The watchdog currently clears the pre-execution timer only when the phase is classified as execution or firstModelCallStarted is set.

That means a live isolated run can emit phases such as:

  • model_resolution
  • auth
  • context_engine
  • fallback runtime_plugins

but still remain in waiting_for_execution. After 60s, the pre-execution watchdog aborts the run even though progress is being observed and the configured job timeout may be much longer.

The fallback re-arm path makes this worse: after before_agent_reply, a fallback/setup phase can move the state back to waiting_for_execution, causing the same false pre-execution timeout.

Expected behavior

  • onExecutionStarted() with no subsequent phase should still be guarded by the pre-execution watchdog.
  • Any known phase reported after runner start should count as runner progress and clear the pre-execution watchdog.
  • Long model setup/context work should be governed by the job's configured timeout, not the hard-coded 60s pre-execution guard.

Proposed fix

Treat any known watchdog phase as progress:

if (stage !== undefined || info.firstModelCallStarted) {
  state = "executing";
  clearPreExecutionTimeout();
}

Also stop re-arming the pre-execution watchdog when fallback/setup phases follow before_agent_reply; those phases are progress and should remain under the normal job timeout.

Regression coverage

Add/adjust cron timer regression coverage for:

  • Runner starts but emits no phase: still fails with stalled before execution start after the pre-execution watchdog.
  • Runner emits setup phases (model_resolution, auth, context_engine): stays alive past 60s and only fails at the configured job timeout.
  • Fallback/setup progress after before_agent_reply: does not re-arm the 60s pre-execution watchdog.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions