Summary

A cron job replayed as startup catch-up never has its in-process active marker set, so while the catch-up run is in flight isCronJobActive(jobId) stays false. The task-registry reconcile keys off exactly that marker for runtime:"cron" tasks, so a catch-up run that outlives TASK_RECONCILE_GRACE_MS (5 min) is misclassified as lost and emits a Background task lost system message while it is still actively running.

markCronJobActive / clearCronJobActive were introduced in 7d1575b5df (#60310) but wired into only the tick paths (runDueJob, executeJob). runStartupCatchupCandidate calls tryCreateCronTaskRun and emits action:"started" but never calls markCronJobActive, even though the completion path still calls clearCronJobActive — an asymmetric clear-without-mark.

• Impact: every gateway restart that replays a missed command job through startup catch-up; a run >5 min surfaces a spurious lost task (false terminal projection in the task UI) while the job is genuinely working. The heartbeat re-entrancy guard (HEARTBEAT_SKIP_CRON_IN_PROGRESS, which checks hasActiveCronJobs()) is also not engaged for the catch-up window.
• Affected: non-agentTurn (command-kind) missed jobs, the only startup-eligible kind under the production deferAgentTurnJobs:true path. Up to DEFAULT_MAX_MISSED_JOBS_PER_RESTART per restart.
• Version audited: main @ 5e1fbca3cbc

Root cause

src/cron/service/timer.ts — runStartupCatchupCandidate (≈1271-1320) creates the task ledger row and emits started, but the markCronJobActive call present on the sibling paths is missing:

// src/cron/service/timer.ts ~1276 (runStartupCatchupCandidate)
const taskRunId = tryCreateCronTaskRun({
  state,
  job: candidate.job,
  startedAt,
});
// <-- markCronJobActive(candidate.job.id) is NOT called here
emit(state, {
  jobId: candidate.job.id,
  action: "started",
  job: candidate.job,
  runAtMs: startedAt,
});

Compare the tick paths, which both mark active:

// runDueJob, timer.ts:888
markCronJobActive(job.id);
// executeJob, timer.ts:1672
markCronJobActive(job.id);

The clear side is still present for catch-up via applyOutcomeToStoredJob → clearCronJobActive(result.jobId) (timer.ts:705), so the mark/clear pair is asymmetric: clear-without-mark for the catch-up path.

Why it bites the reconcile. start() replays missed jobs with runMissedJobs(state, { deferAgentTurnJobs: true }) (src/cron/service/ops.ts:204). planStartupCatchup makes only non-agentTurn jobs startup-eligible (timer.ts startupEligible = sorted.filter((job) => job.payload.kind !== "agentTurn")), so command-kind jobs run immediately through runStartupCatchupCandidate while agentTurn jobs are deferred to the (marked) tick path. The reconcile depends solely on the marker:

// src/tasks/task-registry.maintenance.ts:483-490  hasBackingSession
if (task.runtime === "cron") {
  if (!taskRegistryMaintenanceRuntime.isRuntimeAuthoritative()) {
    return true;
  }
  const jobId = task.sourceId?.trim();
  return jobId ? taskRegistryMaintenanceRuntime.isCronJobActive(jobId) : false;
}

The gateway sets runtimeAuthoritative: true (src/gateway/server-startup-early.ts:131). The cron task row carries runtime:"cron", sourceId: job.id, lastEventAt: startedAt, status running (src/cron/service/task-runs.ts tryCreateCronTaskRun). A command run emits no progress events, so lastEventAt is pinned to startedAt and hasLostGraceExpired trips at exactly 5 min. With isCronJobActive(jobId) === false, shouldMarkLost returns true and the running task is reconciled to lost.

resolveDurableCronTaskRecovery does not save it mid-flight: resolveCronRunLogRecovery requires a "finished" run-log row at runAtMs === startedAt, and resolveCronJobStateRecovery requires job.state.lastRunAtMs === startedAt with a terminal status — both only true after the run completes. No production runtime registers tryRecoverTaskBeforeMarkLost, so that hook is a no-op for cron.

Reproduction

Drives the real CronService.start() startup replay of a missed command job with the production deferAgentTurnJobs:true, blocking the injected runCommandJob mid-flight, and asserts the producer-side invariant the reconcile depends on. Place at src/cron/service.startup-catchup-active-marker.repro.test.ts and run with the cron vitest config.

import { beforeEach, describe, expect, it } from "vitest";
import { isCronJobActive, resetCronActiveJobsForTests } from "./active-jobs.js";
import { CronService } from "./service.js";
import {
  createDeferred,
  setupCronServiceSuite,
  writeCronStoreSnapshot,
} from "./service.test-harness.js";
import type { CronJob } from "./types.js";

const BASE_TIME_ISO = "2025-12-13T17:00:00.000Z";
const { logger, makeStorePath } = setupCronServiceSuite({
  prefix: "openclaw-cron-startup-catchup-active-marker-",
  baseTimeIso: BASE_TIME_ISO,
});

type CommandRunResult = Awaited<
  ReturnType<NonNullable<ConstructorParameters<typeof CronService>[0]["runCommandJob"]>>
>;

function createMissedCommandJob(id: string): CronJob {
  const now = Date.parse(BASE_TIME_ISO);
  return {
    id,
    name: id.replaceAll("-", " "),
    enabled: true,
    createdAtMs: now - 3_600_000,
    updatedAtMs: now - 3_600_000,
    schedule: { kind: "cron", expr: "0 * * * *", tz: "UTC" },
    sessionTarget: "isolated",
    wakeMode: "next-heartbeat",
    payload: { kind: "command", argv: ["echo", "nightly-backup"] },
    delivery: { mode: "none" },
    state: {
      // A slot due an hour ago that the gateway "missed" while it was down.
      nextRunAtMs: now - 3_600_000,
      lastRunAtMs: now - 7_200_000,
    },
  };
}

async function createStartupCatchupHarness(jobId: string) {
  const store = await makeStorePath();
  await writeCronStoreSnapshot({ storePath: store.storePath, jobs: [createMissedCommandJob(jobId)] });

  const entered = createDeferred<void>();
  const release = createDeferred<CommandRunResult>();
  const cron = new CronService({
    storePath: store.storePath,
    cronEnabled: true,
    log: logger,
    enqueueSystemEvent: () => {},
    requestHeartbeat: () => {},
    runCommandJob: async () => {
      entered.resolve();
      return await release.promise;
    },
  });
  return { cron, entered, release, store };
}

describe("cron activeJobIds — startup catch-up mark/clear", () => {
  beforeEach(() => {
    resetCronActiveJobsForTests();
  });

  it("marks the job active while a missed command job is replayed on startup", async () => {
    const { cron, entered, release, store } = await createStartupCatchupHarness("catchup-command");
    try {
      // start() awaits the in-flight catch-up run, so kick it off without await.
      const startPromise = cron.start();
      await entered.promise;

      expect(isCronJobActive("catchup-command")).toBe(true); // FAILS on main (false)

      release.resolve({ status: "ok", summary: "ok" });
      await startPromise;
      expect(isCronJobActive("catchup-command")).toBe(false);
    } finally {
      cron.stop();
      await store.cleanup();
    }
  });
});

Observed (main @ 5e1fbca3cbc)

× marks the job active while a missed command job is replayed on startup
AssertionError: expected false to be true
- Expected  true
+ Received  false
  Tests  1 failed (1)

i.e. isCronJobActive("catchup-command") is false while the command job is actively running through startup catch-up. With runtimeAuthoritative:true (gateway), hasBackingSession therefore returns false and a >5-min run is reconciled lost.

After the one-line fix

Adding markCronJobActive(candidate.job.id); immediately after tryCreateCronTaskRun in runStartupCatchupCandidate:

✓ marks the job active while a missed command job is replayed on startup
  Tests  1 passed (1)

Existing src/cron/service.restart-catchup.test.ts + src/cron/active-jobs-manual-run.test.ts stay green (16/16) with the fix applied.

Why this is not already covered

• PR fix(cron): mirror active-jobs mark/clear on startup catchup and manual run #71040 (closed) proposed exactly this mark/clear pair on the startup catch-up + manual-run paths. It was closed citing deferAgentTurnJobs:true as superseding the agentTurn startup-catchup scenario — but that deferral only moves agentTurn jobs to the tick path; command jobs (the actual startup-eligible kind) still run through the unmarked runStartupCatchupCandidate.
• Issue cron: transient 'lost' marker on long-running manual runs before sweeper recovery #78233 (closed) covers only the manual-run path (prepareManualRun / finishPreparedManualRun).
• Issue [Bug] Cron tasks marked as lost after gateway restart — activeJobIds not persisted #79196 (closed) is a different gap — activeJobIds not persisted across a restart — not the never-set marker on the catch-up run itself.
• The merged manual-run guard (src/cron/active-jobs-manual-run.test.ts) covers its sibling path; no catch-up test asserts isCronJobActive, so CI stayed green.

Suggested direction

One insertion in runStartupCatchupCandidate restores mark/clear symmetry (clearCronJobActive is already invoked by applyOutcomeToStoredJob):

  const taskRunId = tryCreateCronTaskRun({ state, job: candidate.job, startedAt });
+ markCronJobActive(candidate.job.id);
  emit(state, { jobId: candidate.job.id, action: "started", job: candidate.job, runAtMs: startedAt });

Add the catch-up regression test above so the invariant is locked across all execution paths. Happy to send the fix + test as a PR.