fix(cron): guard task liveness during startup by nanookclaw · Pull Request #80707 · openclaw/openclaw

nanookclaw · 2026-05-11T16:14:47Z

Summary

Task registry maintenance now treats cron active-job liveness as non-authoritative until cron startup reconciliation exits. This prevents a pre-start maintenance sweep from marking persisted cron task records lost while the process-local active-job set is still empty after restart.

The change keeps the boundary narrow:

activeJobIds starts non-authoritative after process boot
cron.start() marks liveness reconciling at entry and authoritative in finally
task maintenance preserves only mapped cron tasks while cron liveness is non-authoritative
once cron start exits, missing active-job state is trusted again

This is AI-assisted.

Proof

Real setup: local checkout of openclaw/openclaw, branch rebased onto origin/main at 435e1a74fa.

Commands run after the rebase/conflict resolution:

git diff --check
node scripts/run-tsgo.mjs -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo
pnpm exec vitest run src/cron/service/ops.test.ts --config test/vitest/vitest.config.ts
pnpm exec vitest run src/tasks/task-registry.maintenance.issue-60299.test.ts src/tasks/task-registry.test.ts --config test/vitest/vitest.config.ts

Observed result:

git diff --check: passed
core tsgo: passed
src/cron/service/ops.test.ts: 13 passed
src/tasks/task-registry.maintenance.issue-60299.test.ts: 21 passed
src/tasks/task-registry.test.ts: 63 passed

Real behavior proof

Behavior addressed: Task registry maintenance should preserve mapped stale cron task records while cron active-job liveness is non-authoritative during startup, then resume marking missing mapped cron tasks lost after cron startup exits and liveness is authoritative.

Real setup tested: Local openclaw/openclaw checkout on the PR branch, using an isolated temporary state root under /tmp/openclaw-pr-80707-proof-* and the repository's TypeScript runtime via node --import tsx.

Exact steps or command run after this patch:

node --import tsx .artifacts/pr-80707-real-proof.mjs

The script seeds 10-minute-stale mapped cron task records with runtime: cron and sourceId, runs task maintenance once while cron active-job liveness is non-authoritative, then runs task maintenance again after liveness is marked authoritative.

Evidence after fix: Redacted terminal output from the isolated local state-dir run:

{
  "stateRoot": "/tmp/openclaw-pr-80707-proof-s9f07s",
  "scenario": "mapped stale cron task maintenance across cron startup liveness authority",
  "staleAgeMs": 600000,
  "beforeStartupExit": {
    "livenessAuthoritative": false,
    "maintenance": {
      "reconciled": 0,
      "recovered": 0,
      "cleanupStamped": 0,
      "pruned": 0
    },
    "task": {
      "runtime": "cron",
      "sourceId": "proof-cron-before-startup-exit",
      "status": "running",
      "endedAtPresent": false
    }
  },
  "afterStartupExit": {
    "livenessAuthoritative": true,
    "maintenance": {
      "reconciled": 2,
      "recovered": 0,
      "cleanupStamped": 0,
      "pruned": 0
    },
    "task": {
      "runtime": "cron",
      "sourceId": "proof-cron-after-startup-exit",
      "status": "lost",
      "endedAtPresent": true,
      "error": "backing session missing"
    }
  }
}

Observed result after fix: The pre-start/non-authoritative maintenance pass left the stale mapped cron task running with reconciled: 0. The post-start/authoritative maintenance pass marked the stale mapped cron task lost with error: "backing session missing".

Not tested: I did not run the full monorepo pnpm build && pnpm check && pnpm test locally.

This shows the fixed behavior: task maintenance does not mark a mapped stale cron task lost during the pre-start/non-authoritative window, and does mark mapped stale cron tasks lost after startup liveness becomes authoritative again.

Also ran a focused Codex review against the rebased diff; it reported no blocker. I did not run the full monorepo pnpm build && pnpm check && pnpm test locally.

clawsweeper · 2026-05-11T16:18:17Z

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Keep open: the startup liveness invariant is still useful on current main, but the live PR branch is conflict-dirty and applies the guard to an older activeJobIds-only active-job model that current main has since replaced with generation-aware active job markers.

Canonical path: Close this PR as superseded by #96174.

So I’m closing this here and keeping the remaining discussion on #96174.

Review details

Best possible solution:

Close this PR as superseded by #96174.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection gives a high-signal path: gateway task maintenance is runtime-authoritative early, lazy cron startup may not have reconciled active jobs, and hasBackingSession then trusts an empty process-local active-job map. I did not run a local repro because this is a read-only review.

Is this the best way to solve the issue?

No, this exact branch is not currently the best mergeable fix because it applies the right invariant to an older activeJobIds-only implementation. The best fix is a focused current-main port onto the activeJobs/generation owner model.

Security review:

Security review cleared: The diff is limited to cron/task runtime code and tests, with no dependency, workflow, secret, permission, or supply-chain surface changes.

AGENTS.md: found and applied where relevant.

What I checked:

linked superseding PR: fix(cron): mark interrupted restart runs failed instead of lost #96174 (fix(cron): mark interrupted restart runs failed instead of lost) is still open as the canonical replacement.
cluster evidence: the durable review links that PR in the work cluster or recommended risk path.
no human follow-up: live comments and timeline hydrated by apply contain no non-automation activity after the ClawSweeper review.

Likely related people:

vincentkoc: Recent live path history shows active-job and task-maintenance refactors around reset/helper ownership in June 2026. (role: recent adjacent owner; confidence: medium; commits: d29c3a5d6f59, 29e44f5ebad3, 2e27a37791c9; files: src/cron/active-jobs.ts, src/tasks/task-registry.maintenance.ts)
ghitafilali: Recent active-jobs history includes restart/setup-timeout hardening touching stale cron-task finalization around restart boundaries. (role: recent active-job lifecycle contributor; confidence: medium; commits: b2c5e790b441; files: src/cron/active-jobs.ts, src/cron/service/ops.ts)
yetval: Recent ops.ts history includes multiple cron startup/scheduler fixes, including preserved startup overflow catch-up deferrals and due-run preservation. (role: recent cron service contributor; confidence: medium; commits: e896ca963430, e38d64c5f88b; files: src/cron/service/ops.ts)
liaoandi: Merged terminal lost cron recovery work recently changed the same task-registry maintenance surface and related regression tests. (role: recent task-maintenance contributor; confidence: medium; commits: be94853de0d0; files: src/tasks/task-registry.maintenance.ts, src/tasks/task-registry.maintenance.issue-60299.test.ts)

Codex review notes: model internal, reasoning high; reviewed against 738b2be4b49b.

nanookclaw · 2026-05-11T16:30:20Z

@clawsweeper re-review

Added real behavior proof to the PR body from an isolated local state dir. The proof seeds stale mapped cron task records, runs task maintenance while cron active-job liveness is non-authoritative, then again after liveness is authoritative. Output shows the pre-start task remains running with reconciled: 0, while post-start authoritative maintenance marks mapped stale cron tasks lost.

clawsweeper · 2026-05-11T16:31:45Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/25687972344
Updated: 2026-05-11T18:08:30.540Z

nanookclaw · 2026-05-11T17:33:16Z

@clawsweeper re-review

Moved the isolated local state-dir proof under an explicit ## Real behavior proof heading so the required check can detect it. The proof content itself is unchanged.

nanookclaw · 2026-05-11T17:34:21Z

@clawsweeper re-review

Structured the real behavior proof section with the checker-required fields: behavior, environment, steps, evidence, observed result, and not tested. The terminal proof remains the same isolated local state-dir run.

nanookclaw · 2026-05-11T17:35:40Z

@clawsweeper re-review

Converted the proof fields to the exact labels accepted by the gate (Behavior addressed, Real setup tested, Exact steps or command run after this patch, Evidence after fix, Observed result after fix, Not tested).

clawsweeper · 2026-05-11T22:42:08Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/25701633371
Updated: 2026-05-11T22:48:33.650Z

clawsweeper · 2026-05-12T17:18:35Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26556268241
Updated: 2026-05-28T05:23:50.348Z

openclaw-barnacle · 2026-05-27T04:52:22Z

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

clawsweeper · 2026-05-27T04:58:58Z

ClawSweeper PR egg: 🔥 warming; proof passed, review follow-up or readiness checks remain. Hatch with @clawsweeper hatch when eligible.

Rules and details

Hatchability:

Merged PRs are hatchable.
Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

About:

Eggs appear after real-behavior proof passes. They are collectible flavor only.
Review momentum changes the shell state: follow-up work warms it, re-review makes it wobble, and a clean final review lets it hatch.
The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

nanookclaw · 2026-05-27T18:35:53Z

Still ready for maintainer look. No code changes since the last update; the branch was already rebased onto main and ClawSweeper re-review completed again on the current head.

Current state from the live PR:

labels include proof: sufficient, rating: 🐚 platinum hermit, and status: 👀 ready for maintainer look
only visible non-success checks are cancelled automation/proof reruns, not code failures
the remaining state is maintainer review

nanookclaw · 2026-06-03T05:59:40Z

@clawsweeper re-review

Rebased the branch onto current main and resolved the dirty branch conflict. The cron liveness guard is preserved, and the interrupted-startup emit path now keeps current main's failureNotificationDelivery: job ? failureNotificationDeliveryFromJobState(job) : undefined metadata.

Local verification after the rebase:

git diff --check
node_modules/.bin/tsgo -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo --declaration false --singleThreaded --checkers 1
NODE_OPTIONS='--max-old-space-size=1536' npx vitest run src/cron/service/ops.test.ts --config test/vitest/vitest.cron.config.ts --pool=forks --maxWorkers=1 — 16 passed
NODE_OPTIONS='--max-old-space-size=1536' npx vitest run src/tasks/task-registry.maintenance.issue-60299.test.ts --config test/vitest/vitest.tasks.config.ts --pool=forks --maxWorkers=1 — 24 passed
NODE_OPTIONS='--max-old-space-size=1536' npx vitest run src/tasks/task-registry.test.ts --config test/vitest/vitest.tasks.config.ts --pool=forks --maxWorkers=1 -t "rechecks current task state before marking a task lost" — 1 passed / 73 skipped
NODE_OPTIONS='--max-old-space-size=1536' npx vitest run src/tasks/task-registry.test.ts --config test/vitest/vitest.tasks.config.ts --pool=forks --maxWorkers=1 -t "does not leak unhandled rejections when the scheduled maintenance sweep fails" — 1 passed / 73 skipped

clawsweeper · 2026-06-03T05:59:44Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26924332740
Updated: 2026-06-04T01:37:07.520Z

…-guard Signed-off-by: Nanook <[email protected]> # Conflicts: # src/cron/service/ops.test.ts

clawsweeper · 2026-06-07T15:25:44Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27096715413
Updated: 2026-06-07T15:34:52.267Z

clawsweeper · 2026-06-30T11:38:08Z

🦞✅
ClawSweeper autoclose is complete.

Reason: structured ClawSweeper close marker: close-required (sha=a94fcc4e5e73787222a5b1d342da064af230a5f9)

Closed:

Closed fix(cron): guard task liveness during startup #80707: fix(cron): guard task liveness during startup

openclaw-barnacle Bot added size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 11, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 11, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

nanookclaw force-pushed the fix/cron-startup-race-guard branch from dc6739b to 3ba4d18 Compare May 11, 2026 22:36

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 11, 2026

clawsweeper Bot mentioned this pull request May 12, 2026

False-positive lost cron task records after gateway restart due to transient activeJobIds backing-session check #68191

Closed

nanookclaw force-pushed the fix/cron-startup-race-guard branch from 3ba4d18 to d8dd1b7 Compare May 12, 2026 17:10

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 12, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 12, 2026

openclaw-barnacle Bot added the stale Marked as stale due to inactivity label May 27, 2026

nanookclaw force-pushed the fix/cron-startup-race-guard branch from d8dd1b7 to 2cb121d Compare June 3, 2026 05:59

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 3, 2026

Merge remote-tracking branch 'origin/main' into fix/cron-startup-race…

a94fcc4

…-guard Signed-off-by: Nanook <[email protected]> # Conflicts: # src/cron/service/ops.test.ts

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 7, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 7, 2026

clawsweeper Bot mentioned this pull request Jun 19, 2026

fix(tasks): recover terminal lost cron rows #86088

Merged

yetval mentioned this pull request Jun 23, 2026

cron: restart-interrupted run mislabeled lost instead of failed (recovery matches reservation timestamp vs run id started-at) #96161

Open

clawsweeper Bot mentioned this pull request Jun 24, 2026

fix(cron): mark interrupted restart runs failed instead of lost #96174

Open

clawsweeper Bot added status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 24, 2026

clawsweeper Bot closed this Jun 30, 2026

Uh oh!

Uh oh!

Conversation

nanookclaw commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof

Real behavior proof

Uh oh!

clawsweeper Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nanookclaw commented May 11, 2026

Uh oh!

clawsweeper Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nanookclaw commented May 11, 2026

Uh oh!

nanookclaw commented May 11, 2026

Uh oh!

nanookclaw commented May 11, 2026

Uh oh!

clawsweeper Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openclaw-barnacle Bot commented May 27, 2026

Uh oh!

clawsweeper Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nanookclaw commented May 27, 2026

Uh oh!

nanookclaw commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nanookclaw commented May 11, 2026 •

edited

Loading

clawsweeper Bot commented May 11, 2026 •

edited

Loading

clawsweeper Bot commented May 11, 2026 •

edited

Loading

clawsweeper Bot commented May 11, 2026 •

edited

Loading

clawsweeper Bot commented May 12, 2026 •

edited

Loading

clawsweeper Bot commented May 27, 2026 •

edited

Loading

nanookclaw commented Jun 3, 2026 •

edited

Loading

clawsweeper Bot commented Jun 3, 2026 •

edited

Loading

clawsweeper Bot commented Jun 7, 2026 •

edited

Loading