Skip to content

fix(cron): treat isolated setup phases as progress#93914

Open
parthjayaram wants to merge 2 commits into
openclaw:mainfrom
parthjayaram:pj/fix-cron-pre-exec-watchdog
Open

fix(cron): treat isolated setup phases as progress#93914
parthjayaram wants to merge 2 commits into
openclaw:mainfrom
parthjayaram:pj/fix-cron-pre-exec-watchdog

Conversation

@parthjayaram

Copy link
Copy Markdown

Summary

  • Fixes the isolated cron pre-execution watchdog treating setup/context phase progress as a stall.
  • Keeps the 60s pre-execution guard for the case where the runner starts but emits no phase progress at all.
  • Moves setup-like phase progress (model_resolution, auth, context_engine, fallback runtime_plugins) under the configured job timeout instead of the hard-coded pre-execution watchdog.
  • Out of scope: changing cron config defaults, timeout durations, delivery behavior, or provider/model routing.

Reviewers should focus on whether the watchdog should classify any known phase as progress after runner start, and whether the updated regression expectations match the intended timeout contract.

Linked context

Closes #93912

Related #93530
Related #93867

Requested by an operator trying to decide whether isolated cron jobs are safe to run again.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: isolated cron agentTurn jobs can be aborted by the 60s pre-execution watchdog while setup/context phases continue to report progress.
  • Real environment tested: local OpenClaw source checkout on Linux arm64 with repository dependencies installed.
  • Exact steps or command run after this patch: node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
[test] starting test/vitest/vitest.cron.config.ts

 RUN  v4.1.8 /tmp/openclaw-fix-clean

 Test Files  1 passed (1)
      Tests  64 passed (64)
   Start at  03:54:04
   Duration  2.14s (transform 873ms, setup 114ms, import 1.23s, tests 722ms, environment 0ms)

[test] passed 1 Vitest shard in 5.04s
  • Observed result after fix: the regression file passes, including coverage that setup/context phase progress stays alive past 60s and only times out on the configured job timeout.
  • What was not tested: live scheduled cron against a real provider/channel.
  • Proof limitations or environment constraints: this is focused source-level regression proof, not a live production cron run.
  • Before evidence (optional but encouraged): before the test expectation update, the old stale test timed out after 120s because it expected the pre-execution watchdog to re-arm on fallback setup progress.

Tests and validation

Commands run:

  • corepack pnpm install --frozen-lockfile
  • node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts
  • corepack pnpm exec oxfmt --check --threads=1 src/cron/service/agent-watchdog.ts src/cron/service/timer.regression.test.ts
  • git diff --check

Regression coverage added/updated:

  • Runner starts but emits no phase progress: still fails with stalled before execution start from the pre-execution watchdog.
  • Runner emits setup phases (model_resolution, auth, context_engine): stays alive past 60s and eventually hits the configured job timeout.
  • Fallback/setup progress after before_agent_reply: no longer re-arms the 60s pre-execution watchdog.

Risk checklist

Did user-visible behavior change? Yes

Did config, environment, or migration behavior change? No

Did security, auth, secrets, network, or tool execution behavior change? No

Highest-risk area: cron isolated-agent timeout behavior.

Risk mitigation: the pre-execution watchdog still protects the no-progress case; known phase progress now uses the existing configured job timeout, and the touched behavior is covered in src/cron/service/timer.regression.test.ts.

Current review state

Next action: maintainer review and CI.

Still waiting on: maintainer decision on whether focused regression proof is enough or whether a live cron repro should be attached before merge.

No bot or reviewer comments addressed yet.

@openclaw-barnacle openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 17, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 19, 2026
@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 9:04 AM ET / 13:04 UTC.

Summary
The PR changes the isolated cron agent watchdog so phase progress after runner entry clears the pre-execution guard, while runner entry alone remains guarded, and expands timer regression coverage.

PR surface: Source -12, Tests +158. Total +146 across 2 files.

Reproducibility: yes. Source inspection shows current main still routes isolated cron start and phase updates into the watchdog, maps setup phases to pre_execution, and only clears the guard for execution-stage progress or firstModelCallStarted; I did not run a live scheduled cron repro in this read-only review.

Review metrics: 1 noteworthy metric.

  • Pre-execution phases newly clearing guard: 5 newly clear, 1 excluded. workspace, runtime_plugins, model_resolution, auth, and context_engine would move under the full job timeout while runner_entered remains under the short guard, which is the key availability tradeoff.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #93912
Summary: This PR is an open candidate fix for the narrowed setup-progress watchdog issue; sibling PRs overlap but none is a merged canonical replacement.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦪 silver shellfish
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Use the existing stage lookup in the new guard so check-prod-types and check-lint can pass.
  • [P1] Add redacted live scheduled-cron terminal output, copied live output, or runtime logs showing setup progress survives past the old 60-second guard.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body and latest comment provide copied Vitest/fake-timer terminal output only; external real behavior proof still needs redacted live scheduled-cron output, logs, or an explicit maintainer override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P2] The PR intentionally moves workspace, runtime_plugins, model_resolution, auth, and context_engine stalls from the 60-second pre-execution guard to the configured job timeout; maintainers need to accept that availability tradeoff before merge.
  • [P1] The supplied proof is copied Vitest/fake-timer output, not redacted live scheduled-cron output or runtime logs showing the after-fix path in a real setup.
  • [P1] Several sibling PRs still propose narrower before_agent_reply-based fixes, so maintainers should keep one canonical cron watchdog contract before landing this cluster.

Maintainer options:

  1. Fix Stage Use And Require Proof (recommended)
    Use the existing stage lookup in the new guard, keep the runner_entered regression, and require redacted real scheduled-cron proof before merge.
  2. Accept Focused Proof Only
    Maintainers can choose to override the external proof gate if they consider the fake-timer regression sufficient for this watchdog timeout contract.
  3. Defer To A Canonical Sibling
    Pause or close this PR if a narrower sibling watchdog or phase-emission PR becomes the chosen landing path.

Next step before merge

  • [P1] Human follow-up is needed because the PR has a narrow type/lint blocker plus a contributor real-behavior-proof gate that ClawSweeper should not repair around automatically.

Maintainer decision needed

  • Question: After the unused-stage blocker is fixed, should this external PR require redacted live scheduled-cron proof, or should maintainers override with focused source-level regression proof?
  • Rationale: ClawSweeper policy treats tests and fake timers as supplemental only for external PRs, while this PR changes availability-sensitive cron timeout behavior that maintainers may decide to waive or require live proof for.
  • Likely owner: steipete — The current watchdog module boundary dates to steipete's cron timer split, and the remaining choice is proof sufficiency for that runtime behavior.
  • Options:
    • Require live cron proof (recommended): Ask for redacted scheduled-cron terminal output, copied live output, or runtime logs showing setup progress surviving past the old 60-second cutoff before merge.
    • Override with focused regression proof: A maintainer can explicitly accept the fake-timer regression proof as sufficient for this narrow watchdog contract after the type/lint blocker is fixed.
    • Pick a sibling contract instead: Pause or close this PR if maintainers prefer a narrower before_agent_reply phase-emission fix from a sibling PR.

Security
Cleared: Cleared: the diff changes cron watchdog logic and regression tests only, with no dependency, CI, secret, package, or external code-execution surface changed.

Review findings

  • [P2] Use the stage lookup in the new guard — src/cron/service/agent-watchdog.ts:118-120
Review details

Best possible solution:

Use the watchdog-layer fix, keep runner_entered-only starts under the short guard, fix the unused stage lookup, and require either redacted live scheduled-cron proof or an explicit maintainer proof override before merge.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main still routes isolated cron start and phase updates into the watchdog, maps setup phases to pre_execution, and only clears the guard for execution-stage progress or firstModelCallStarted; I did not run a live scheduled cron repro in this read-only review.

Is this the best way to solve the issue?

Mostly yes in direction, but not merge-ready as written. The watchdog layer is the right boundary and the latest commit preserves the runner_entered guard, but the patch must use the stage lookup so type/lint pass and still needs real scheduled-cron proof or an explicit proof override.

Full review comments:

  • [P2] Use the stage lookup in the new guard — src/cron/service/agent-watchdog.ts:118-120
    The new predicate stops reading stage, leaving the lookup at line 117 unused; core tsgo has noUnusedLocals and lint treats unused vars as errors, and CI is already reporting TS6133/no-unused-vars for this file. Use stage !== undefined && info.phase !== "runner_entered" or otherwise remove the lookup and map so the branch can pass type/lint checks.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.93

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 124119a8985e.

Label changes

Label justifications:

  • P1: The PR targets a broken isolated cron agent workflow that can abort scheduled runs before user output is produced.
  • merge-risk: 🚨 availability: Changing which cron setup phases clear the pre-execution guard can turn short startup-stall failures into full job-timeout waits.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body and latest comment provide copied Vitest/fake-timer terminal output only; external real behavior proof still needs redacted live scheduled-cron output, logs, or an explicit maintainer override. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source -12, Tests +158. Total +146 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 3 15 -12
Tests 1 178 20 +158
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 181 35 +146

What I checked:

  • PR head leaves stage unused: At PR head, noteExecutionProgress still computes stage from CRON_AGENT_PHASE_WATCHDOG_STAGE but the new observedProgressAfterRunnerEntry predicate no longer reads it. (src/cron/service/agent-watchdog.ts:117, 60413c7ed723)
  • Typecheck failure matches the diff: The current CI check-prod-types job reports TS6133 because stage is declared but never read in src/cron/service/agent-watchdog.ts. (src/cron/service/agent-watchdog.ts:117, 60413c7ed723)
  • Lint failure matches the diff: The current CI check-lint job reports eslint(no-unused-vars) for the same unused stage variable. (src/cron/service/agent-watchdog.ts:117, 60413c7ed723)
  • Repository unused-local contract: The core TypeScript project enables noUnusedLocals, so an unused local in src/cron/service/agent-watchdog.ts is a real merge blocker rather than cosmetic lint noise. (tsconfig.core.json:4, 124119a8985e)
  • Repository lint contract: The repo lint config treats eslint/no-unused-vars as an error, matching the CI failure for the PR head. (.oxlintrc.json:40, 124119a8985e)
  • Current main watchdog behavior: Current main still maps setup-like phases to pre_execution and clears the short guard only for execution-stage phases or firstModelCallStarted, so the central bug is not already solved on main. (src/cron/service/agent-watchdog.ts:31, 124119a8985e)

Likely related people:

  • steipete: Authored the commit that split the current agent-watchdog module out of the cron timer helpers, making this person a strong routing candidate for the module boundary. (role: cron service refactor contributor; confidence: medium; commits: 9d31cbbd6ad8, 1e7510ae103e; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
  • ghitafilali: Recent isolated cron setup-timeout work touched the watchdog, timer, and regression tests near the timeout behavior changed by this PR. (role: recent adjacent cron watchdog contributor; confidence: medium; commits: b2c5e790b441; files: src/cron/service/agent-watchdog.ts, src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
  • vincentkoc: Recent cron timeout and wrapper cleanup commits touched the isolated cron execution and timer regression area, though not as directly as the watchdog-specific commits. (role: recent adjacent cron contributor; confidence: low; commits: efbefceb0e2e, 44ae2fd93601; files: src/cron/isolated-agent/run.ts, src/cron/service/timer.ts, src/cron/service/timer.regression.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-06-22T03:16:42.895Z sha 57d9452 :: needs real behavior proof before merge. :: [P1] Preserve the runner-start watchdog
  • reviewed 2026-07-01T07:01:28.316Z sha 57d9452 :: needs real behavior proof before merge. :: [P1] Preserve the runner-start watchdog

@parthjayaram

Copy link
Copy Markdown
Author

Addressed the current review blocker around runner_entered.

What changed in 60413c7ed7:

  • runner_entered no longer clears the pre-execution watchdog by itself.
  • Only phases after runner entry, or firstModelCallStarted, move the watchdog to normal execution timeout.
  • Added regression coverage that runner_entered alone still fails as stalled before execution start (last phase: runner-entered).
  • Existing setup-progress and fallback-progress coverage still passes, so workspace / runtime_plugins / model_resolution / auth / context_engine continue to count as meaningful progress.

Validation after the narrower fix:

node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts
Test Files  1 passed (1)
Tests       65 passed (65)

Additional checks:

corepack pnpm exec oxfmt --check --threads=1 src/cron/service/agent-watchdog.ts src/cron/service/timer.regression.test.ts
All matched files use the correct format.

git diff --check
# no output

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 24, 2026
@clawsweeper

clawsweeper Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(cron): treat isolated setup phases as progress This is item 1/1 in the current shard. Shard 3/4.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron isolated agent pre-execution watchdog aborts runs despite setup progress

1 participant