fix(cron): bound restart catch-up to schedule activation#92398
fix(cron): bound restart catch-up to schedule activation#92398amersheeny wants to merge 5 commits into
Conversation
|
Codex review: found issues before merge. Reviewed June 30, 2026, 11:06 PM ET / 03:06 UTC. Summary PR surface: Source +118, Tests +403. Total +521 across 14 files. Reproducibility: yes. Current main is source-reproducible: cron.update can move nextRunAtMs without recording schedule activation, and restart catch-up still replays the current expression's previous slot solely because it is newer than lastRunAtMs. Review metrics: 2 noteworthy metrics.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Rebase onto current main, stamp scheduleActivatedAtMs only when scheduling inputs actually change, keep the doctor/protocol ownership model, and have a maintainer explicitly accept the conservative upgrade boundary before landing. Do we have a high-confidence way to reproduce the issue? Yes. Current main is source-reproducible: cron.update can move nextRunAtMs without recording schedule activation, and restart catch-up still replays the current expression's previous slot solely because it is newer than lastRunAtMs. Is this the best way to solve the issue? No, not as currently mergeable. The scheduler-owned activation boundary plus doctor backfill is the right fix shape, but the branch must be rebased to preserve current main's idempotent-update guard before maintainers accept the upgrade semantics. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against fa3c9de45965. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +118, Tests +403. Total +521 across 14 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (1 earlier review cycle)
|
|
@clawsweeper re-review The previous review run failed to execute ("Review failed before ClawSweeper could summarize the requested change") and overwrote the prior completed verdict with an off-meta/needs-proof fallback. CI is green, the branch is mergeable/clean against latest main, and the Real behavior proof check passes — please re-run so the durable verdict reflects the actual patch. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review The prior re-review run (clawsweeper run 27493343782) has been stalled in |
|
Noting the competing implementation #92305 (@fsdwen) for this same issue, since the review flagged that maintainers should pick one cron state model before landing either. A factual comparison to help that decision: Both PRs guard restart catch-up so a slot computed from a newly-updated cron expression isn't replayed as "missed." The difference is the state each relies on:
The decisive behavioral difference is coverage of the reported scenario, where the user updated the schedule and then upgraded:
Where #92305 is genuinely better: a metadata-only edit (rename/description) doesn't move So the trade-off: #92398 is leaner, adds no cron state, and fixes the as-reported upgrade case, at the cost of a documented fail-safe on metadata-only edits. #92305 is more precise on metadata-only edits, at the cost of a new persisted field (without a doctor migration) that doesn't cover already-updated jobs. Happy to align this PR with whichever state model maintainers prefer. |
|
Reconsidering my earlier comparison: I was over-weighting "no new field = leaner" and under-crediting precision — candidly, a slant toward my own approach. A single optional state field that makes the catch-up boundary exact is cheap and pays for itself; "additional field = expense" was the wrong frame. So I've reworked this PR onto the dedicated-field design ( The one thing I kept from the Verified via the real gateway in both states (field present and field stripped to simulate a legacy job) — both suppress the pre-activation slot with zero missed-job log lines; details and output in the updated PR description. Net diff is test-heavy; prod change is +28/−7. No disagreement with #92305's core idea — it's the right one, and credit to @fsdwen for it. The remaining difference is just the legacy fallback; if maintainers prefer to land #92305, adding the same |
|
Land-ready verification for
Best-fix verdict: scheduler-owned activation state plus a doctor migration is the canonical fix. Inferring activation from create/update/run timestamps would preserve ambiguous legacy behavior in the runtime path. |
0e30df1 to
989cc7b
Compare
|
@vincentkoc heads-up before you land this: I rebased it onto current New head is One thing to note: your |
|
@vincentkoc following up here since you'd already done the land-ready pass — all three of my open PRs are now green on current The two siblings are stuck purely on ClawSweeper's Codex capacity, not on the changes: the re-review keeps failing at the "Wait for exact-review Codex capacity" step (latest run 27692800832 failed 2026-06-17 13:54 UTC after the 20-min wait). The diffs are byte-identical to the SHAs ClawSweeper already reviewed positively — #92399 ("ready for maintainer review / proof sufficient") and #93547 ("correct fix, supersedes #93315"). Two asks when you have a moment:
Happy to provide whatever helps. Thanks for the time on this batch. |
…on write Startup catch-up classified a cron job as "missed" whenever the current expression's previous occurrence was newer than lastRunAtMs, ignoring a nextRunAtMs that a cron.update deliberately moved into the future. After an API schedule change, the new expression's previous slot never existed while that schedule was live, so gateway restarts (e.g. during openclaw update) fired the job immediately after the startup deferral. Bound the replay to slots that fell after updatedAtMs: writers that recompute the schedule (update, run outcomes, interrupted-run marking) bump it, so older slots were scheduled away on purpose. Applies to both startup replay and error-backoff slot deferral, which shared the same unbounded comparison. Fixes openclaw#91944. Reported by @suifatt7799-oss.
…recisely Replaces the updatedAtMs-only replay guard with a dedicated state.scheduleActivatedAtMs, recorded whenever an update recomputes nextRunAtMs from a schedule/enabled change (and at job creation). The catch-up guard now skips slots that predate the schedule's activation exactly, so a later metadata-only edit (rename/description) no longer masks a genuinely missed slot — removing the fail-safe trade-off the updatedAtMs proxy carried. Jobs predating the field fall back to updatedAtMs at runtime: the reported misfire happens at the upgrade restart, before any doctor migration could backfill the field, and updatedAtMs is the only activation proxy available then (it over-approximates activation, so the fallback only ever over-suppresses in the safe direction). This keeps the reported pre-upgrade-update scenario fixed for existing users while giving post-upgrade updates exact precision. Regression tests cover the field path, the legacy fallback, the metadata-edit precision win, and that update() records the timestamp only on schedule/enabled changes.
…e schema The new scheduleActivatedAtMs cron state field was added to the runtime type and persisted via state_json, but the closed CronJobStateSchema (and its patch schema) reject unknown properties, so schema-validating cron.get/list consumers could reject jobs carrying it. Declare the field in both protocol schemas, matching the existing timestamp fields, and assert its presence in the schema test.
989cc7b to
6a00c36
Compare
|
This assigned pull request has been automatically marked as stale after being open for 27 days. |
|
Closing due to inactivity. |
Summary
Fixes #91944.
Cron restart catch-up now replays only schedule slots that occurred after the schedule was activated. The scheduler records
state.scheduleActivatedAtMswhen a job is created or when its schedule/enabled state changes, and both startup catch-up and error-backoff replay use that boundary.Existing cron rows are handled by
openclaw doctor --fix: the cron preflight first imports legacy cron state, then transactionally stamps rows missing the activation field with the migration time. Runtime reads only the canonical field and fails closed when it is absent, so an upgrade cannot infer and send stale pre-upgrade outbound slots.Metadata-only edits do not move the boundary. Public create/update inputs cannot inject scheduler-owned activation state.
Verification
cbx_4ce772051720, runrun_6f4988261933, 120 tests.0e30df1f5a45d4a83d095e6cdf559db1ee05eda6.Real behavior proof
Behavior addressed: An API-updated cron schedule must not replay a slot that predates the schedule activation after a gateway restart.
Real environment tested: The contributor ran a real macOS Gateway with a scratch SQLite cron store, created and edited the job through the real CLI/Gateway RPC path, injected the historical prior-schedule run, and restarted the Gateway. The final repaired head was additionally validated on Azure Linux Crabbox against the focused cron/doctor/SQLite suites.
Exact steps or command run after this patch: Create the cron job through
openclaw cron add, change its expression throughopenclaw cron edit, set the prior-schedulelastRunAtMsin the scratch SQLite row, restart the Gateway, and inspect the job state and missed-job logs. For the final doctor-owned legacy path, removescheduleActivatedAtMs, run the cron doctor preflight migration, and verify the canonical activation boundary is written before restart evaluation.Evidence after fix:
Observed result after fix: The updated job remains on its future API-selected schedule after restart. Existing rows receive one conservative doctor-owned activation boundary, preventing stale pre-upgrade outbound replay.
What was not tested: A full version-to-version
openclaw updateinstall flow. The real Gateway restart covered the field-present runtime path; the final legacy migration replacement is covered by current-head SQLite integration tests and the Azure Linux run, so a maintainer proof override is applied for that portion.Risk
This changes restart delivery behavior and persisted cron state. The conservative migration may skip a pre-upgrade missed slot, but it will not send an inferred stale outbound job. No config or environment surface is added.
Release-note context: cron jobs no longer fire slots that predate their current schedule activation after a gateway restart or upgrade. Reported by @suifatt7799-oss in #91944.