fix(cron): skip completed restart catchup slots by LiLan0125 · Pull Request #101998 · openclaw/openclaw

LiLan0125 · 2026-07-08T03:38:39Z

Closes #101988

What Problem This Solves

Gateway restart catch-up could treat a stale persisted cron nextRunAtMs as runnable before checking whether the job had already completed that schedule slot. For isolated agent-turn cron jobs, this could defer and rerun a successful daily reminder after restart, causing duplicate notifications.

Why This Change Was Made

The runnable check now recognizes the specific restart state where a cron job has a due nextRunAtMs, lastRunStatus: ok, and lastRunAtMs already covers that due slot. Instead of immediately marking the job runnable, it falls through to the existing previous-slot guard so only genuinely missed later slots are replayed. Failed or skipped jobs still keep their existing retry behavior.

User Impact

Users should no longer receive duplicate isolated-agent cron messages after restarting the Gateway when the current schedule tick already ran successfully. Legitimate missed cron work and failure retry paths remain eligible for catch-up.

Evidence

node scripts/run-vitest.mjs src/cron/service.restart-catchup.test.ts src/cron/service/timer.regression.test.ts

[test] starting test/vitest/vitest.cron.config.ts
[test] passed 1 Vitest shard in 20.51s

git diff --check

passed

Real behavior proof

Behavior addressed: Restart catch-up skips a stale due cron slot when the isolated agent-turn job already completed that slot successfully.
Environment tested: Linux, Node v22.22.0, local source checkout on branch fix/101988-cron-catchup-window, isolated temporary OPENCLAW_HOME and OPENCLAW_STATE_DIR.
Steps run after the patch: Created a real cron store with an isolated agent-turn cron job whose persisted nextRunAtMs was 2025-12-13 09:10 UTC, lastRunStatus was ok, and lastRunAtMs was 2025-12-13 09:10:30 UTC; started the production CronService at 2025-12-13 11:00 UTC and listed the stored job.
Evidence after fix:

node --import tsx --no-warnings -e '
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
const root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-101988-proof-")));
process.env.OPENCLAW_HOME = path.join(root, "home");
process.env.OPENCLAW_STATE_DIR = path.join(root, "state");
const { CronService } = await import("./src/cron/service.js");
const { saveCronStore } = await import("./src/cron/store.js");
const storePath = path.join(root, "cron.json");
const startNow = Date.parse("2025-12-13T11:00:00.000Z");
const dueAt = Date.parse("2025-12-13T09:10:00.000Z");
const completedAt = Date.parse("2025-12-13T09:10:30.000Z");
const runIsolatedAgentJobCalls = [];
const heartbeatCalls = [];
const events = [];
await saveCronStore(storePath, {
  version: 1,
  jobs: [{
    id: "startup-isolated-agent-already-ok",
    name: "startup isolated agent already ok",
    enabled: true,
    createdAtMs: Date.parse("2025-12-10T12:00:00.000Z"),
    updatedAtMs: completedAt,
    schedule: { kind: "cron", expr: "10 9 * * *", tz: "UTC" },
    sessionTarget: "isolated",
    wakeMode: "next-heartbeat",
    payload: { kind: "agentTurn", message: "daily reminder" },
    state: { nextRunAtMs: dueAt, lastRunAtMs: completedAt, lastRunStatus: "ok" },
  }],
});
const cron = new CronService({
  storePath,
  cronEnabled: true,
  log: { debug() {}, info() {}, warn() {}, error() {} },
  nowMs: () => startNow,
  enqueueSystemEvent: async () => ({ status: "queued" }),
  requestHeartbeat: async (...args) => { heartbeatCalls.push(args); },
  runIsolatedAgentJob: async (...args) => {
    runIsolatedAgentJobCalls.push(args);
    return { status: "ok" };
  },
  onEvent: (evt) => events.push(evt),
  startupDeferredMissedAgentJobDelayMs: 120_000,
});
try {
  await cron.start();
  const [job] = await cron.list({ includeDisabled: true });
  console.log(JSON.stringify({
    runIsolatedAgentJobCalls: runIsolatedAgentJobCalls.length,
    heartbeatCalls: heartbeatCalls.length,
    startedEvents: events.filter((evt) => evt.action === "started").length,
    lastRunStatus: job?.state.lastRunStatus,
    lastRunAtIso: new Date(job?.state.lastRunAtMs ?? 0).toISOString(),
    nextRunAtIso: new Date(job?.state.nextRunAtMs ?? 0).toISOString(),
  }, null, 2));
} finally {
  cron.stop();
  await fs.rm(root, { recursive: true, force: true });
}
'

{
  "runIsolatedAgentJobCalls": 0,
  "heartbeatCalls": 0,
  "startedEvents": 0,
  "lastRunStatus": "ok",
  "lastRunAtIso": "2025-12-13T09:10:30.000Z",
  "nextRunAtIso": "2025-12-14T09:10:00.000Z"
}

Observed result after the fix: Gateway startup did not run or defer the already-completed isolated agent-turn slot; no agent job, heartbeat, or started event was emitted, and the job advanced to the next natural daily cron slot.
Not tested: A live macOS Gateway restart with real channel delivery credentials was not exercised locally.

clawsweeper · 2026-07-08T03:43:17Z

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 9:15 PM ET / July 18, 2026, 01:15 UTC.

Summary
The branch prevents completed cron restart-catch-up slots from replaying, adds inclusive previous-slot scheduling for exact restart boundaries, and covers completed ok and skipped states with regression tests.

PR surface: Source +44, Tests +156. Total +200 across 4 files.

Reproducibility: yes. from source and supplied real-service evidence: a persisted due cron slot with lastRunAtMs >= nextRunAtMs and a completed status exercises the affected startup predicate; the submitted A/B report also states the new regression test fails against the base and passes on the fix.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/cron/service.restart-catchup.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #101988
Summary: This PR is the viable open candidate implementation for the canonical restart catch-up duplicate-notification report; the two other related PRs are closed unmerged alternatives.

Members:

canonical: [Feature] Skip catchup rerun for cron jobs with lastRunStatus=ok within current schedule window #101988 - The PR body formally targets this report and its source-level restart catch-up failure mode.
same_root_cause: fix: skip already-run cron catchup after restart #102004 - It addressed the same stale completed-slot restart path and raised the exact-boundary case now covered by this branch, but it is closed unmerged.
same_root_cause: fix(cron): skip startup catchup for missed jobs already completed with status=ok #102000 - It proposed the same completed-slot suppression but is closed unmerged and lacks the current branch's boundary-preserving implementation.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P1] Restart catch-up determines whether user-facing cron delivery is replayed or skipped; the code is focused and proof-backed, but maintainers should preserve the intended skipped-slot semantics when reviewing the exact head.

Maintainer options:

Accept the bounded restart-delivery behavior (recommended)
After normal exact-head review, accept the focused rule that avoids replaying completed slots while still replaying a genuinely newer missed slot.

Next step before merge

[P2] The focused repair and proof are already present; the remaining action is ordinary maintainer review of the current head, not an automated branch repair.

Security
Cleared: The diff is confined to cron scheduling logic and tests; it adds no dependencies, permissions, workflow changes, credential handling, or external execution path.

Review details

Best possible solution:

Land the bounded scheduler fix after normal exact-head maintainer review, preserving the inclusive slot comparison and its coverage for stale completed, later-missed, and exact-restart-boundary cron slots.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and supplied real-service evidence: a persisted due cron slot with lastRunAtMs >= nextRunAtMs and a completed status exercises the affected startup predicate; the submitted A/B report also states the new regression test fails against the base and passes on the fix.

Is this the best way to solve the issue?

Yes. The inclusive previous-slot helper is the narrow maintainable solution because it fixes the stale-completed-slot replay without losing the later missed slot or the slot due exactly at restart.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 1c331b9b0ff3.

Label changes

Label justifications:

P1: The branch fixes duplicate or suppressed scheduled notifications caused by Gateway restart catch-up.
merge-risk: 🚨 message-delivery: The changed runnable predicate directly controls whether an overdue cron job is replayed into a user-facing delivery path.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR includes an after-fix production CronService run against a persisted cron store and shows the observable corrected state; focused tests supplement that direct runtime proof.
proof: sufficient: Contributor real behavior proof is sufficient. The PR includes an after-fix production CronService run against a persisted cron store and shows the observable corrected state; focused tests supplement that direct runtime proof.

Evidence reviewed

PR surface:

Source +44, Tests +156. Total +200 across 4 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	52	8	+44
Tests	2	156	0	+156
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	4	208	8	+200

What I checked:

Current-head runnable predicate: The head changes the due-slot branch to suppress only a completed cron slot and then compares against the latest effective cron slot at or before restart, preserving catch-up when a newer slot—including one exactly at restart—has become due. (src/cron/service/timer.ts:1834, 17a48ce509cb)
Inclusive scheduling primitive and boundary coverage: The branch adds computeJobPreviousRunAtOrBeforeMs plus tests for an exact cron boundary, an in-between cursor, and a staggered effective boundary; this directly addresses the earlier review comment about strict-before scheduler queries. (src/cron/service/jobs.ts:525, 17a48ce509cb)
Restart regression coverage: The restart-catch-up tests cover completed persisted slots for both ok and skipped, assert no isolated run, heartbeat, or system event is emitted, and include later-slot and exact-restart-boundary behavior in the updated branch context. (src/cron/service.restart-catchup.test.ts:270, 17a48ce509cb)
Current-main comparison: Current main at 1c331b9b0ff3170a5471c2a7ff4b60eafa36963a still exposes only the strict computeJobPreviousRunAtMs path; it does not contain the PR's inclusive helper or its runnable-predicate guard, so the central fix is not already implemented. (src/cron/service/jobs.ts:510, 1c331b9b0ff3)
Feature-history provenance: The current branch's boundary-preserving follow-up was authored by steipete in commit 992b50a9b9b7414c2c7eab6d9fd54a92c80ce82e; the subsequent head commit extends the same restart-slot suppression to skipped outcomes. (src/cron/service/jobs.ts:181, 992b50a9b9b7)
Real behavior evidence: The PR body records an after-fix run of production CronService against a persisted isolated-agent cron store: startup emitted zero isolated runs, heartbeats, and started events, then advanced the stored next run to the next natural daily slot. (src/cron/service.restart-catchup.test.ts:270, 17a48ce509cb)

Likely related people:

steipete: Authored both current branch commits that preserve exact restart boundaries and extend completed-slot suppression, and is assigned/review-requesting on this PR. (role: recent area contributor; confidence: high; commits: 992b50a9b9b7, 17a48ce509cb; files: src/cron/service/jobs.ts, src/cron/service/timer.ts, src/cron/service.restart-catchup.test.ts)
NianJiuZst: The earlier related restart-boundary proposal supplied the exact-boundary concern later incorporated into the current branch; the current follow-up commit credits this related work. (role: adjacent contributor; confidence: medium; commits: 992b50a9b9b7; files: src/cron/service/jobs.ts, src/cron/service.restart-catchup.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (2 earlier review cycles)

reviewed 2026-07-08T03:48:06.872Z sha 5870f97 :: needs maintainer review before merge. :: none
reviewed 2026-07-18T00:41:10.268Z sha d279df5 :: needs maintainer review before merge. :: none

harjothkhara · 2026-07-08T04:08:29Z

I think this branch still has one restart-boundary gap before it becomes the canonical fix for #101988.

Best-fix comparison:

fix(cron): skip completed restart catchup slots #101998 has the stronger real CronService proof for the stale successful-slot skip.
fix: skip already-run cron catchup after restart #102004 has the stronger scheduler boundary coverage: stale-success skip, newer missed-slot replay, and a cron slot due exactly at restart.

The concrete issue I see in this branch is the exact-boundary case. This patch falls through from the due nextRunAtMs branch into the existing missed-slot guard when lastRunStatus === "ok" and lastRunAtMs >= nextRunAtMs, but that guard calls computeJobPreviousRunAtMs(job, nowMs). For non-staggered cron schedules, that ultimately uses computePreviousRunAtMs, which intentionally returns only timestamps < nowMs.

So with a persisted minutely cron state like nextRunAtMs = 04:01:00, lastRunAtMs = 04:01:00, lastRunStatus = ok, and Gateway restart at exactly 04:02:00, this branch falls through, the previous-slot guard still sees 04:01:00 rather than the boundary 04:02:00, and previousRunAtMs > lastRunAtMs is false. That skips the 04:02 slot even though it is due at restart.

I would not treat this as the best final fix unless it ports the boundary replay coverage/fix from #102004 or explicitly proves that due-exactly-at-restart cron slots are intentionally outside startup catch-up. The strongest canonical path looks like combining this PR's real CronService proof with #102004's boundary-preserving test shape.

knoal · 2026-07-13T12:21:08Z

A/B verified locally against base SHA. Ran vitest with the PR's test file against the PR's fix (B) and against the base's fix (A).

Result: CLEAN A/B

A (PR test + base fix): FAIL - 1 test failed
B (PR test + PR fix):   PASS

The new test does not defer an isolated cron agent-turn whose persistence fails on base, passes with the fix. Failing-first signature is correct. Looks ready to merge.

Co-authored-by: 李兰 0668001394 <[email protected]> Co-authored-by: NianJiuZst <[email protected]>

steipete · 2026-07-18T01:19:05Z

Merged via squash.

Prepared head SHA: 17a48ce509cbe9a8c0979ba2e16b7976ae8c858a
Landed commit: 3ec5ee6bb62d64ec05006bbd53f5a66f45d0dbcd

* fix(cron): preserve exact restart boundaries Co-authored-by: 李兰 0668001394 <[email protected]> Co-authored-by: NianJiuZst <[email protected]> * fix(cron): suppress skipped restart slots --------- Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: NianJiuZst <[email protected]>

openclaw-barnacle Bot added the size: S label Jul 8, 2026

This was referenced Jul 8, 2026

fix(cron): skip startup catchup for missed jobs already completed with status=ok #102000

Closed

fix: skip already-run cron catchup after restart #102004

Closed

clawsweeper Bot mentioned this pull request Jul 8, 2026

Expose maxMissedJobsPerRestart and missedJobStaggerMs in cron config #42039

Open

steipete self-assigned this Jul 18, 2026

steipete requested a review from a team as a code owner July 18, 2026 00:37

openclaw-barnacle Bot added size: M and removed size: S labels Jul 18, 2026

clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label Jul 18, 2026

steipete and others added 2 commits July 18, 2026 01:52

fix(cron): preserve exact restart boundaries

992b50a

Co-authored-by: 李兰 0668001394 <[email protected]> Co-authored-by: NianJiuZst <[email protected]>

fix(cron): suppress skipped restart slots

17a48ce

steipete force-pushed the fix/101988-cron-catchup-window branch from d279df5 to 17a48ce Compare July 18, 2026 01:11

steipete merged commit 3ec5ee6 into openclaw:main Jul 18, 2026
113 checks passed

github-actions Bot mentioned this pull request Jul 18, 2026

📡 Upstream Digest — 2026-07-18 01:45 UTC curtismercier/openclaw-mods#1329

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(cron): skip completed restart catchup slots#101998

fix(cron): skip completed restart catchup slots#101998
steipete merged 2 commits into
openclaw:mainfrom
LiLan0125:fix/101988-cron-catchup-window

LiLan0125 commented Jul 8, 2026

Uh oh!

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

harjothkhara commented Jul 8, 2026

Uh oh!

knoal commented Jul 13, 2026

Uh oh!

Uh oh!

steipete commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

LiLan0125 commented Jul 8, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Real behavior proof

Uh oh!

clawsweeper Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harjothkhara commented Jul 8, 2026

Uh oh!

knoal commented Jul 13, 2026

Uh oh!

Uh oh!

steipete commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading