Skip to content

fix(cron): retry isolated setup timeouts#94199

Closed
aaroneden wants to merge 1 commit into
openclaw:mainfrom
aaroneden:fix/cron-setup-watchdog-queue-saturation
Closed

fix(cron): retry isolated setup timeouts#94199
aaroneden wants to merge 1 commit into
openclaw:mainfrom
aaroneden:fix/cron-setup-watchdog-queue-saturation

Conversation

@aaroneden

@aaroneden aaroneden commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • classify isolated cron setup timeouts and pre-execution stalls as retryable timeout errors
  • cover the retry hint wording for setup watchdog failures
  • add a cron timer regression proving recurring jobs schedule retry backoff before the next regular fire

Tests

  • pnpm exec vitest run src/cron/retry-hint.test.ts src/cron/service/timer.regression.test.ts
  • pnpm test (started; unrelated failures in state DB permissions, git-root boundary, and plugin tooling/postbuild tests; tooling shard hung after failures and was stopped)

Real behavior proof

  • Behavior or issue addressed: Isolated cron watchdog setup failures like cron: isolated agent setup timed out before runner start are now classified as retryable timeout failures instead of terminal cron errors.
  • Real environment tested: Local OpenClaw checkout on macOS in /Users/aaroneden/.openclaw/workspace/openclaw-fix-cron-setup-watchdog-queue-saturation, branch fix/cron-setup-watchdog-queue-saturation, commit 9cc60413a1b7a55cf956d8b22bd5a918ae278ee0.
  • Exact steps or command run after this patch: Ran node --import tsx against ./src/cron/retry-hint.ts and called resolveCronExecutionRetryHint(message, ["timeout"]) for both watchdog messages. Also ran the targeted cron vitest command listed above.
  • Evidence after fix: Copied terminal output from the live source command:
cron: isolated agent setup timed out before runner start -> {"retryable":true,"category":"timeout"}
cron: isolated agent run stalled before execution start -> {"retryable":true,"category":"timeout"}
  • Observed result after fix: The exact setup-timeout and pre-execution-stall strings both return retryable: true with category: "timeout"; the recurring scheduler regression also passes and verifies retry backoff is scheduled earlier than the six-hour normal schedule.
  • What was not tested: A forced live production cron cluster saturation was not performed; the live proof exercises the patched classifier and the regression test exercises the scheduler consequence without modifying production cron cadence.

@openclaw-barnacle openclaw-barnacle Bot added size: S triage: blank-template Candidate: PR template appears mostly untouched. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 17, 2026
@aaroneden
aaroneden force-pushed the fix/cron-setup-watchdog-queue-saturation branch from b76a76b to 3da3b20 Compare June 17, 2026 15:42
@aaroneden
aaroneden force-pushed the fix/cron-setup-watchdog-queue-saturation branch from 3da3b20 to 9cc6041 Compare June 17, 2026 15:44
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 17, 2026
@clawsweeper

clawsweeper Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 17, 2026, 12:08 PM ET / 16:08 UTC.

Summary
The PR broadens cron timeout retry matching for setup/pre-execution watchdog messages and adds classifier plus recurring-scheduler regression tests.

PR surface: Source 0, Tests +55. Total +55 across 3 files.

Reproducibility: Do we have a high-confidence way to reproduce the issue? Yes at source level: current main emits the watchdog strings but the timeout retry regex does not match timed out or stalled before execution start; the PR body also includes after-fix classifier output for both messages.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] Broad CI was still in progress in the live status; the PR body supplies targeted cron test output and classifier live output, but its full pnpm test attempt stopped on unrelated failures.
  • [P2] This is a bounded retry mitigation for timeout symptoms, not the broader root-cause fix for open isolated-cron watchdog and runtime-plugin stall reports.

Maintainer options:

  1. Decide the mitigation before merge
    Land the narrow classifier/test fix if required CI finishes cleanly, while keeping the broader cron watchdog and root-cause trackers open.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No automated repair is needed; the PR has no blocking review findings and is ready for normal maintainer review plus CI completion.

Security
Cleared: The diff only changes cron retry classification and tests; it does not touch dependencies, CI, secrets, packaging, or code-execution surfaces.

Review details

Best possible solution:

Land the narrow classifier/test fix if required CI finishes cleanly, while keeping the broader cron watchdog and root-cause trackers open.

Do we have a high-confidence way to reproduce the issue?

Do we have a high-confidence way to reproduce the issue? Yes at source level: current main emits the watchdog strings but the timeout retry regex does not match timed out or stalled before execution start; the PR body also includes after-fix classifier output for both messages.

Is this the best way to solve the issue?

Is this the best way to solve the issue? Yes for the retry-policy layer: update the shared classifier and prove the recurring scheduler consequence, while leaving watchdog timing/root-cause work to the existing broader issues.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3c1b346115b5.

Label changes

Label changes:

  • add P1: The PR targets real scheduled isolated-agent cron failures that can otherwise skip work until the next scheduled fire or disable one-shot jobs instead of retrying.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied after-fix live output from running the patched classifier against both watchdog messages, plus targeted cron regression test commands; it does not claim production cluster saturation proof.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied after-fix live output from running the patched classifier against both watchdog messages, plus targeted cron regression test commands; it does not claim production cluster saturation proof.

Label justifications:

  • P1: The PR targets real scheduled isolated-agent cron failures that can otherwise skip work until the next scheduled fire or disable one-shot jobs instead of retrying.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied after-fix live output from running the patched classifier against both watchdog messages, plus targeted cron regression test commands; it does not claim production cluster saturation proof.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied after-fix live output from running the patched classifier against both watchdog messages, plus targeted cron regression test commands; it does not claim production cluster saturation proof.
Evidence reviewed

PR surface:

Source 0, Tests +55. Total +55 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 1 1 0
Tests 2 55 0 +55
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 56 1 +55

What I checked:

  • Repository policy read and applied: Root AGENTS.md was read fully; its PR-review guidance affected the review by requiring caller/callee, sibling tests, related issue search, proof, security, and history checks for this core cron runtime change. (AGENTS.md:1, 3c1b346115b5)
  • Current main classifier misses these watchdog strings: On current main, the timeout retry pattern only matches timeout or etimedout, so setup timed out before runner start and stalled before execution start are not covered by the timeout category. (src/cron/retry-hint.ts:28, 3c1b346115b5)
  • Current main produces the exact watchdog messages: The setup and pre-execution cron helpers format the exact strings this PR teaches the retry classifier to recognize. (src/cron/service/execution-errors.ts:19, 3c1b346115b5)
  • Retry scheduling consumes the shared classifier: Cron retry decisions call resolveCronExecutionRetryHint and use its category to schedule bounded transient retry backoff, so the classifier is the right narrow boundary for this behavior. (src/cron/service/timer.ts:451, 3c1b346115b5)
  • PR diff is focused: The live PR diff changes one retry regex and adds two focused tests: one classifier test and one recurring cron scheduler regression for setup-timeout retry backoff. (src/cron/retry-hint.ts:28, 9cc60413a1b7)
  • Related issue search found broader open work: Live GitHub search and related issue reads found open isolated-cron stall/watchdog issues, including a broader runtime-plugins stall tracker and newer watchdog-progress reports; this PR is a partial mitigation, not a cluster-closing fix.

Likely related people:

  • vincentkoc: Blame and log history show the current retry classifier, cron timer, watchdog, and execution-error helper files were introduced in commit 2195b44. (role: introduced current cron retry/watchdog implementation; confidence: high; commits: 2195b446d488; files: src/cron/retry-hint.ts, src/cron/service/timer.ts, src/cron/service/agent-watchdog.ts)
  • alexzhu0: Live PR history shows an open related diagnostic PR that threads runtime-plugin progress into cron timeout messages for the same isolated pre-execution stall family. (role: open adjacent diagnostic PR author; confidence: medium; commits: 1260a5febd69; files: src/cron/service/execution-errors.ts, src/cron/isolated-agent/run.ts, src/plugins/loader.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 High-priority user-facing bug, regression, or broken workflow. labels Jun 17, 2026
@Takhoffman

Takhoffman commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@clawsweeper automerge

@clawsweeper

clawsweeper Bot commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🔧
ClawSweeper automerge is enabled.

Draft PRs stay fix-only until GitHub marks them ready for review. Pause with /clawsweeper stop.

Automerge progress:

  • 2026-06-18 12:19:34 UTC review queued 9cc60413a1b7 (queued)

@clawsweeper clawsweeper Bot added the clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge label Jun 18, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 18, 2026
Merged
@clawsweeper

clawsweeper Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper 🐠 reef update

Thanks for the work on this. ClawSweeper did not have permission to update this branch directly, so it opened a narrow replacement PR instead. that's a branch access thing, not a knock on the contribution.

Why replacement: ClawSweeper could not update the source PR branch directly; GitHub did not grant sufficient push rights to the bot for that branch.
Replacement PR: #94588
Why close: this run explicitly closes the superseded source PR after the credited replacement PR is open, so review continues in one place.
This source PR is being closed only under the explicit source-close setting for this ClawSweeper run.
The replacement PR keeps the contributor trail visible for review and release notes.
Co-author credit kept:

fish notes: reasoning high; reviewed against 1786e6a.

@clawsweeper clawsweeper Bot closed this Jun 18, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 19, 2026
Closed
@clawsweeper clawsweeper Bot mentioned this pull request Jun 28, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. triage: blank-template Candidate: PR template appears mostly untouched.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aaroneden @Takhoffman