Skip to content

Slim & harden cron subsystem: Phase 1+2+3 (subprocess execution, atomic-remove, doctor migrations)#1

Open
Desicool wants to merge 31 commits into
desicool-releasefrom
openclaw_cron
Open

Slim & harden cron subsystem: Phase 1+2+3 (subprocess execution, atomic-remove, doctor migrations)#1
Desicool wants to merge 31 commits into
desicool-releasefrom
openclaw_cron

Conversation

@Desicool

Copy link
Copy Markdown
Owner

Summary

Re-architects the cron subsystem to fix gateway OOMs and pending-job buildup. Three phases, fully reviewed, deployed to and verified on instance-1n7ut600-25 (Silver Chariot, openclaw 2026.5.29 fork build).

  • Phase 1: openclaw cron purge CLI + doctor --fix wiring for preventive cleanup
  • Phase 2: Subprocess execution model (each fire is a short-lived OS process), result-file authoritative terminal record, scheduler.lock, boot reconcile, atomic-remove for kind:at, idempotencyKey dedup, agent-tool key computation
  • Phase 3: Drop legacy kind:"every", per-job tz/staggerMs, non-isolated session targets; shrink session-reaper; doctor migrations for all dropped fields

Why

Original cron subsystem fired each job as an in-process isolated agent session inside the gateway. Sessions leaked (reaper gaps, deleteAfterRun bugs, Codex harness init hangs), pile-up to 800+ pending entries per state file, OOMs the gateway. Phase 2 replaces in-process execution with subprocess-per-fire so the OS reclaims memory on child exit.

Phases at a glance

Phase Commit range LOC
1 881bf70a2234dc6ea461 +700 / -50
2 5655c4ea2c0827f98473 +5400 / -200
3 a7c846cbbd6116753e9e +2200 / -2700 (net −500)
Deploy hardening 4e01fe1e97a359bd5d4f +200 / -100

Net: 18 commits added on top of desicool-release, 962 cron tests passing, tsgo:core + tsgo:test:src + check:import-cycles + format:check all green.

Verification

  • Behavior addressed: gateway OOM driven by leaked isolated-agent sessions; pending-job pile-up driven by buggy auto-remove for one-shot reminders
  • Real environment tested: instance-1n7ut600-25 (openclaw 2026.5.29 fork build, Ubuntu Linux, Node 24.14.0)
  • Exact steps run after this patch: systemctl --user stop openclaw-gateway.servicenpm install -g openclaw-2026.5.29-fork.tgzopenclaw doctor --yes --fixsystemctl --user start openclaw-gateway.service
  • Evidence after fix: 6 cron jobs loaded; subprocess pid table active; archives at ~/.openclaw/cron/jobs.json.{preinstall,purged,target-migrated,tz-migrated,before-restore}-*.{bak,json} for rollback
  • Observed result after fix: gateway running stably; subprocess pathway activates per spec (argv[1] basename contains "openclaw")
  • What was not tested: full happy-path tick under production load (only deployed; not yet observed firing); cross-process atomic-remove crash-recovery in production (verified only by integration tests on dev)

Known follow-ups (not in this PR)

  1. Test/production divergence for subprocess pathway: 6 subprocess tests use real spawn() against fake scripts; broader scheduler tests still fall back to inline via schedulerLockPath: null. Future: OPENCLAW_BIN=<fixture> integration mode.
  2. Several pre-existing tests use \"main\" as unknown as \"isolated\" casts to keep legacy runtime branches in timer.ts exercised. Follow-up: grep + delete confirmed-dead branches.
  3. Upstream sync: at PR-creation time, today's scheduled sync hasn't fired yet; desicool-release is ~1 day behind upstream openclaw/openclaw:main (latest f2dfb67f2c). May need a rebase after sync.

🤖 Generated with Claude Code

actions-user and others added 30 commits May 24, 2026 16:52
Direct 'git fetch origin main:refs/heads/main' fails when main is the
currently checked-out branch ('refusing to fetch into branch X checked
out at Y'). Update remote-tracking ref via plain 'git fetch origin main'
and push origin/main directly.

Signed-off-by: Chiliang Yao <[email protected]>
Co-Authored-By: Claude Opus 4.7 <[email protected]>
…ort flow

Cron-triggered TaskFlow extension that drafts a weekly report from the agent's
own session transcript, sends a Feishu confirmation card, loops on user
supplements, and writes the result into a configured Feishu doc using invisible
HTML-comment sentinels for idempotent splicing.

v2 adds an optional git-activity fact source: a clone-on-demand, author-filtered
`git log` runner exposed as `fetch_git_activity`, so completed bullets are
anchored in real commits. SSH-only by default; URL allowlist, name regex,
argv-only invocation, hooks/submodules/file-protocol disabled, per-op and
overall timeout caps, concurrency-bounded parallel ops, and offline tests with
local bare-repo fixtures.

Five tools: submit_weekly_report_draft, respond_to_weekly_report_card,
splice_weekly_report_doc, finalize_weekly_report, fetch_git_activity. CAS-
idempotent timeout sweeper for reminders + fail-on-expiry. Card-action trust
guards (controllerId / sessionKey / status / weekKey match before any flow
mutation). 84 colocated vitest cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…2.2)

Introduces the child-side entrypoint for the Phase 2 cron subprocess architecture:
- `src/cli/run-cron-job/runtime.ts`: assertCronRunnerContext, writeRunnerResultFile, emitStdoutMarker
- `src/cli/run-cron-job/main.ts`: runCronJobHandler (injectable deps) + registerRunCronJobCommand
- Result file written atomically before stdout marker per plan I1 ordering invariant
- Verb registered in subcli-descriptors and register.subclis-core

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…er lock (P2.3)

Piece 1 — Scheduler lock acquisition: start() acquires scheduler.lock via
acquireSchedulerLock(). When lock is held by another process, the scheduler
goes silent (schedulerLockHeld=true, no timer armed). Lock is released on
stop(). schedulerLockPath=null skips locking in tests.

Piece 2 — Subprocess spawn: executeDetachedCronJob routes isolated-agent jobs
through executeJobInSubprocess() when openClawNode+openClawBin are set.
Spawns: node <openclaw-bin> run-cron-job <jobId> --run-id <uuid>. Reads
result file on exit; falls back to stdout OPENCLAW_CRON_RESULT marker;
SIGTERM→SIGKILL grace on AbortSignal. Inline runIsolatedAgentJob dep is
preserved as fallback when bin paths are not set (tests, isolated modes).

resolveSubprocessBinPaths() only sets openClawBin when process.argv[1]
basename contains "openclaw" — prevents accidental subprocess activation in
vitest environments where argv[1] is vitest.mjs.

Piece 3 — Boot reconcile: loadCronStore() calls reconcileRunningJobStates()
after state merge. Four cases: no running key (no-op), running without runId
(cleared), result file present (terminal state applied), pid dead (cleared),
pid alive with matching start time (left alone). Uses readProcessStartTimeMs()
for pid-recycling detection.

Piece 4 — stopping flag: onTimer() returns early when state.stopping=true.
stop() sets stopping before releasing the lock handle.

New state fields: stopping, openClawNode, openClawBin, pidTable,
schedulerLockHandle, schedulerLockHeld.

New tests: subprocess-spawn.test.ts (6 cases), scheduler-lock-startup.test.ts
(2 cases), boot-reconcile.test.ts (6 cases).

Fixed scheduler-lock.test.ts "returns held" to use child 'spawn' event instead
of setTimeout to avoid fake-timer bleedover in --isolate=false worker.

All 990 cron tests pass.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…dd doctor migrations (P3.2)

Run `openclaw doctor --fix` immediately after upgrade to migrate existing jobs to the new schema. Legacy `kind: "every"` jobs and jobs with non-"isolated" sessionTargets will be skipped at runtime until migrated. The `doctor --fix` command rewrites them to `kind: "cron"` with auto-computed expressions and sessionTarget "isolated".
@github-actions
github-actions Bot force-pushed the desicool-release branch 7 times, most recently from 41ea1f6 to 2b2ee39 Compare July 3, 2026 07:24
@github-actions
github-actions Bot force-pushed the desicool-release branch 8 times, most recently from 26717a7 to b8913d4 Compare July 11, 2026 06:16
@github-actions
github-actions Bot force-pushed the desicool-release branch 8 times, most recently from f70d338 to 92230fb Compare July 19, 2026 06:34
@github-actions
github-actions Bot force-pushed the desicool-release branch 7 times, most recently from f9fbd00 to 6a44021 Compare July 26, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants