Skip to content

cron isolated-agent turn: pre-execution watchdog race with runEmbeddedAgent phase reporting #93530

Description

@kumaxs

Problem

Cron isolated-agent turns (e.g. a daily briefing at 09:00) can abort with externalAbort: true, stopReason: "aborted", and error: "LLM request failed." despite the model provider being healthy and no rate-limit being hit. The abort fires after ~6 minutes, well within the configured timeoutSeconds: 600 (10 min) wall clock.

Root Cause Analysis

The abort originates from the watchdog in executeJobCoreWithTimeout (server-cron, line 1037). The watchdog's triggerTimeout() calls runAbortController.abort(reason), which flows down to runEmbeddedAttemptonExternalAbortSignal()externalAbort=true.

The specific watchdog that fires is the pre-execution timeout (resolveCronAgentPreExecutionWatchdogMsCRON_AGENT_PRE_EXECUTION_WATCHDOG_MS = 60s). It starts when noteRunnerStarted(info) is called, but fails to clear because of a phase/stage gap in the communication between runEmbeddedAgent and the cron watchdog.

Detailed chain

  1. executeCronRunrunEmbeddedAgent is called with trigger: "cron"
  2. At line 2310 of embedded-agent.js:
    if (params.trigger === "cron" && hookRunner?.hasHooks("before_agent_reply")) {
      notifyExecutionPhase("before_agent_reply", ...);
    }
    
  3. If hookRunner.hasHooks("before_agent_reply") returns false (no hook registered for that event), the "before_agent_reply" phase is never emitted
  4. The watchdog's noteExecutionProgress depends on this phase (mapped to stage "execution") to clear the pre-execution timer:
    if (stage === "execution" || info.firstModelCallStarted) {
      state = "executing";
      clearPreExecutionTimeout();
    }
    
  5. Without the clear, the pre-execution timeout fires, triggerTimeout()runAbortController.abort("cron: isolated agent run stalled before execution start")
  6. The next phase "model_resolution" arrives too late — the abort signal has already propagated

Timing

  • 0s: cron triggers executeJobCoreWithTimeout
  • 0s: watchdog start()setupTimeoutId (60s)
  • ~5s: noteRunnerStartedclearSetupTimeout(), startTimeout() (10 min), noteExecutionProgress(), startPreExecutionTimeout() (60s)
  • +60s: pre-execution timeout fires → setTimedOutrunAbortController.abort()
  • LLM call continues for ~5 more minutes before being interrupted by the abort signal → total ~6 min from start
  • OpenClaw reports externalAbort: true, "LLM request failed."

Note: The "before_agent_reply" phase is gated behind hookRunner?.hasHooks("before_agent_reply"). When no hooks register for that event, the watchdog never learns the agent entered execution phase, and pre-execution timeout dominates.

Impact

  • Cron jobs configured with payload.kind="agentTurn" on isolated sessions can abort prematurely when the model provider takes more than ~60 seconds from prompt injection to first model call (the pre-execution watchdog window)
  • Observed specifically with opencode-go/deepseek-v4-flash but impacts any provider where before_agent_reply hook is absent

Suggested Fix

Two options:

Option A (robust): In the cron watchdog's noteExecutionProgress, also clear pre-execution timeout when any phase with a defined stage arrives (not just execution stage). Any phase signal proves the runner is alive and making progress:

if (stage !== undefined || info.firstModelCallStarted) {
  state = "executing";
  clearPreExecutionTimeout();
}

Option B (targeted): In runEmbeddedAgent, emit the "before_agent_reply" phase unconditionally for cron triggers (remove the hookRunner?.hasHooks("before_agent_reply") guard).

I lean toward Option A since it is defensive against any future phase gap.

Environment

  • OpenClaw v2026.6.6
  • macOS 26.4.1 (arm64)
  • Model: opencode-go/deepseek-v4-flash (openai-completions API)
  • Cron config: sessionTarget: "isolated", timeoutSeconds: 600

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions