Skip to content

fix(cron): set active marker for startup catch-up runs (fixes #91695)#91707

Closed
zenglingbiao wants to merge 1 commit into
openclaw:mainfrom
zenglingbiao:fix/issue-91695-catchup-cron-active-marker
Closed

fix(cron): set active marker for startup catch-up runs (fixes #91695)#91707
zenglingbiao wants to merge 1 commit into
openclaw:mainfrom
zenglingbiao:fix/issue-91695-catchup-cron-active-marker

Conversation

@zenglingbiao

Copy link
Copy Markdown
Contributor

Summary

  • Problem: Startup catch-up cron runs never set the in-process active marker (markCronJobActive), so while a catch-up job is running isCronJobActive(jobId) returns false. The task-registry reconcile uses this marker to detect lost tasks — any catch-up run exceeding TASK_RECONCILE_GRACE_MS (5 min) is misclassified as lost, emitting a spurious Background task lost system message.
  • Root Cause: runStartupCatchupCandidate in src/cron/service/timer.ts calls tryCreateCronTaskRun and emits action:"started" but never calls markCronJobActive, unlike the sibling tick paths (runDueJob at line 888, executeJob at line 1672) which both mark active. The completion path still calls clearCronJobActive via applyOutcomeToStoredJob, creating an asymmetric clear-without-mark.
  • Fix: Add markCronJobActive(candidate.job.id) in runStartupCatchupCandidate before tryCreateCronTaskRun, matching the pattern used by runDueJob.
  • What changed:
    • src/cron/service/timer.ts — added markCronJobActive(candidate.job.id) in runStartupCatchupCandidate
  • What did NOT change (scope boundary):
    • The completion path (applyOutcomeToStoredJobclearCronJobActive) — it already works correctly for all paths
    • The HEARTBEAT_SKIP_CRON_IN_PROGRESS guard — engaging it for catch-up is a separate concern

Reproduction

  1. Configure a command-kind cron job (non-agentTurn)
  2. Restart the gateway so the missed job is replayed as startup catch-up
  3. The catch-up run takes longer than 5 minutes (TASK_RECONCILE_GRACE_MS)
  4. Before this PR: The task registry reconcile marks the job lost and emits Background task lost while the job is still actively running
  5. After this PR: isCronJobActive(jobId) returns true during the catch-up run, preventing false lost classification

Real behavior proof

Behavior or issue addressed (91695): Startup catch-up cron runs now call markCronJobActive so the task registry reconcile correctly recognizes them as in-progress rather than classifying long-running catch-up jobs as lost.

Real environment tested: Linux, Node 22 — Vitest with the cron vitest config against restart-catchup, timer, and jobs test suites

Exact steps or command run after this patch: node scripts/run-vitest.mjs src/cron/service.restart-catchup.test.ts src/cron/service/timer.test.ts src/cron/service/timer.regression.test.ts src/cron/service.jobs.test.ts src/cron/active-jobs-manual-run.test.ts src/cron/service.failure-alert.test.ts

Evidence after fix:

Test Files  6 passed (6)
     Tests  145 passed (145)
  Start at  21:17:29
  Duration  5.71s

Observed result after fix: The runStartupCatchupCandidate function now calls markCronJobActive(candidate.job.id) before dispatching the job, matching the symmetric mark-then-clear pattern already used by runDueJob and executeJob. The restart-catchup test suite, timer regression tests, and active-jobs tests all continue to pass.

What was not tested: A live gateway restart with a long-running catch-up cron job that exceeds the 5-minute reconcile grace period. The unit tests exercise the cron service state machine but not the full task-registry reconcile cycle with real time.

Repro confirmation: Before this patch, runStartupCatchupCandidate has no markCronJobActive call between tryCreateCronTaskRun and executeJobCoreWithTimeout, while the sibling runDueJob path (line 888) calls it. After this patch, both paths call markCronJobActive before dispatching.

Risk / Mitigation

  • Risk: Marking the job active for catch-up could prevent the heartbeat skip guard (HEARTBEAT_SKIP_CRON_IN_PROGRESS) from engaging during startup
    Mitigation: The heartbeat skip guard is a separate concern that was already not engaged for catch-up runs. This PR only fixes the reconcile false-positive; engaging the heartbeat guard for catch-up is a follow-up.
  • Risk: If runStartupCatchupCandidate throws after markCronJobActive but before clearCronJobActive, the marker could leak
    Mitigation: The executeJobCoreWithTimeout call is wrapped in try/catch, and applyOutcomeToStoredJob (which calls clearCronJobActive) runs in the finally block via the outcome processing path.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • cron / scheduling

Regression Test Plan

  • node scripts/run-vitest.mjs src/cron/service.restart-catchup.test.ts
  • node scripts/run-vitest.mjs src/cron/service/timer.test.ts
  • node scripts/run-vitest.mjs src/cron/service/timer.regression.test.ts
  • node scripts/run-vitest.mjs src/cron/service.jobs.test.ts

Review Findings Addressed

N/A (initial submission)

Linked Issue/PR

Fixes #91695

runStartupCatchupCandidate never called markCronJobActive, so the
task-registry reconcile misclassified long-running catch-up jobs as
lost after 5 minutes and emitted spurious 'Background task lost'
messages while the job was still actively running.

Add markCronJobActive(candidate.job.id) to match the symmetric
mark-then-clear pattern used by runDueJob and executeJob.

Fixes openclaw#91695
@openclaw-barnacle openclaw-barnacle Bot added size: XS proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 9, 2026
@clawsweeper

clawsweeper Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 9, 2026, 9:30 AM ET / 13:30 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +1. Total +1 across 1 file.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Merge readiness
Overall: 🌊 off-meta tidepool
Proof: 🌊 off-meta tidepool
Patch quality: 🌊 off-meta tidepool
Result: rating does not apply to this item.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P1] Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 98d5c465308a.

Label changes

Label changes:

  • add rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.

Label justifications:

  • rating: 🌊 off-meta tidepool: Overall readiness is 🌊 off-meta tidepool; proof is 🌊 off-meta tidepool and patch quality is 🌊 off-meta tidepool.
Evidence reviewed

PR surface:

Source +1. Total +1 across 1 file.

View PR surface stats
Area Files Added Removed Net
Source 1 1 0 +1
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 1 1 0 +1

What I checked:

  • failure reason: timeout.
  • codex failure detail: Codex review failed for this PR: spawnSync codex ETIMEDOUT.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 9, 2026
@zenglingbiao
zenglingbiao deleted the fix/issue-91695-catchup-cron-active-marker branch June 22, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

proof: supplied External PR includes structured after-fix real behavior proof. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: startup catch-up cron runs never set the active marker, so long command jobs are reconciled as lost while still running

1 participant