Skip to content

fix(tasks): recover terminal lost cron rows#75036

Closed
liaoandi wants to merge 2 commits into
openclaw:mainfrom
liaoandi:fix/tasks-recover-terminal-lost-cron
Closed

fix(tasks): recover terminal lost cron rows#75036
liaoandi wants to merge 2 commits into
openclaw:mainfrom
liaoandi:fix/tasks-recover-terminal-lost-cron

Conversation

@liaoandi

@liaoandi liaoandi commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • allow durable cron recovery to repair terminal lost cron task rows when the loss was the backing-session-missing false-positive path
  • keep recovery constrained to cron rows whose lost error is empty or mentions backing session missing, so unrelated lost terminal rows are not rewritten
  • add regression coverage for terminal lost cron recovery and non-backing-session lost rows staying unchanged
  • add an Unreleased changelog entry for the user-facing task maintenance recovery behavior

Related #63196
Related #72884
Related #68191

Real behavior proof

  • Behavior or issue addressed: Cron-backed task rows that were already terminal/lost because the registry falsely concluded the backing session was missing can now be recovered from durable cron run state/logs, while unrelated lost terminal rows stay unchanged.
  • Real environment tested: Local OpenClaw checkout on macOS, PR branch head 38c58fc92115b830433ae5f6962d5a400ccbc48f, Node v24.5.0, pnpm 11.0.8. The current source CLI entrypoint from this checkout was run against an isolated temporary OPENCLAW_HOME and OPENCLAW_CONFIG_PATH; it did not touch the live user registry.
  • Exact steps or command run after the patch: An isolated task registry sqlite was seeded through the real task registry APIs with a cron task row:
    • runtime: "cron"
    • status: "lost"
    • error: "backing session missing"
    • runId: "cron:cron-proof-terminal-lost:<startedAt>"
    • matching durable cron run log entry with action: "finished", status: "ok", runAtMs: <startedAt>, and summary: "proof recovered from durable cron run log"
  • Commands run after the patch:
    OPENCLAW_HOME="$PROOF_HOME" \
    OPENCLAW_CONFIG_PATH="$PROOF_HOME/.openclaw/openclaw.json" \
    node --import tsx /private/tmp/openclaw_pr75036_current_proof/seed-terminal-lost-cron-proof.mjs
    
    OPENCLAW_HOME="$PROOF_HOME" \
    OPENCLAW_CONFIG_PATH="$PROOF_HOME/.openclaw/openclaw.json" \
    node --import tsx src/index.ts tasks maintenance --json
    
    OPENCLAW_HOME="$PROOF_HOME" \
    OPENCLAW_CONFIG_PATH="$PROOF_HOME/.openclaw/openclaw.json" \
    node --import tsx src/index.ts tasks maintenance --apply --json
    
    OPENCLAW_HOME="$PROOF_HOME" \
    OPENCLAW_CONFIG_PATH="$PROOF_HOME/.openclaw/openclaw.json" \
    node --import tsx src/index.ts tasks list --json
    
    node scripts/test-projects.mjs src/tasks/task-registry.test.ts src/tasks/task-registry.maintenance.issue-60299.test.ts
    git diff --check
  • Before evidence: The seeded isolated registry contained a terminal lost cron row:
    {
      "runtime": "cron",
      "sourceId": "cron-proof-terminal-lost",
      "status": "lost",
      "error": "backing session missing",
      "runId": "cron:cron-proof-terminal-lost:1778567791974"
    }
  • Evidence after fix: Copied live terminal output from the current source CLI preview and apply commands; both reported durable cron recovery:
    {
      "mode": "apply",
      "maintenance": {
        "tasks": {
          "reconciled": 0,
          "recovered": 1,
          "cleanupStamped": 0,
          "pruned": 0
        }
      },
      "tasks": {
        "total": 1,
        "terminal": 1,
        "failures": 0,
        "byStatus": {
          "succeeded": 1,
          "lost": 0
        },
        "byRuntime": {
          "cron": 1
        }
      }
    }
  • Observed result after fix: Copied live terminal output from tasks list --json showed the same task recovered to succeeded and the stale backing-session error was gone:
    {
      "runtime": "cron",
      "sourceId": "cron-proof-terminal-lost",
      "status": "succeeded",
      "terminalSummary": "proof recovered from durable cron run log"
    }
  • Supplemental targeted verification after the patch:
    Test Files  2 passed (2)
    Tests  83 passed (83)
    
    git diff --check completed without whitespace errors.
    
  • What was not tested: Did not run tasks maintenance --apply against the live user task registry; the real command was run against a seeded isolated registry only.

Test plan

  • node scripts/test-projects.mjs src/tasks/task-registry.test.ts src/tasks/task-registry.maintenance.issue-60299.test.ts
  • git diff --check

@clawsweeper

clawsweeper Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution. I reviewed the branch, and this PR is not a good landing base for OpenClaw.

Close as unmergeable cleanup: the intended task-registry fix is useful, but the latest branch head now carries a 65-file, 20k-addition cross-repo diff that is unrelated to terminal lost cron recovery.

So I’m closing this PR rather than keeping an unmergeable branch open. A new narrow PR that carries only the useful part is welcome.

Review details

Best possible solution:

Replace this with a narrow task-registry PR that keeps only the terminal lost cron recovery, focused regression tests, and required changelog entry.

Do we have a high-confidence way to reproduce the issue?

Yes, source-level. A terminal lost cron task with matching durable run evidence remains outside current main's recovery path because recovery requires an active cron task and lost-marking also ignores terminal rows.

Is this the best way to solve the issue?

No as submitted. The core task-registry approach is narrow, but the current branch is not the best merge vehicle because it bundles unrelated generated and formatting changes.

Security review:

Security review needs attention: Needs attention because the latest diff includes a large unexplained generated browser runtime asset unrelated to task recovery.

  • [medium] Unrelated generated browser runtime changed — extensions/diffs/assets/viewer-runtime.js:1
    The PR's stated task-maintenance fix does not justify changing extensions/diffs/assets/viewer-runtime.js; generated/minified browser runtime changes need provenance and focused review before merge.
    Confidence: 0.78

What I checked:

  • Current main recovery gate: Current main only attempts durable cron task recovery when the cron task is active, so an already-terminal lost cron row still does not enter this recovery path. (src/tasks/task-registry.maintenance.ts:451, 56eb23dda4dd)
  • Current main terminal skip: The maintenance lost-marking path returns false for non-active tasks, leaving terminal lost rows to be stamped or pruned rather than recovered from durable cron logs. (src/tasks/task-registry.maintenance.ts:542, 56eb23dda4dd)
  • Shipped adjacent behavior is narrower: The latest release contains active cron ledger recovery before marking rows lost, but the release source still requires isActiveTask(task) and does not include this PR's terminal-lost recovery guard. (src/tasks/task-registry.maintenance.ts:451, a374c3a5bfd5)
  • Branch scope is no longer reviewable for the stated fix: The provided live PR file list for head 7c7f042 shows 65 changed files and 20,691 additions, including 20,238 additions in a generated diff-viewer runtime asset unrelated to task recovery. (extensions/diffs/assets/viewer-runtime.js:1, 7c7f04256a38)
  • Current proof is stale for the current head: The PR body's real-behavior proof says it was run against 38c58fc, while the current head is the later 7c7f042 merge commit with many unrelated changes. (7c7f04256a38)
  • History provenance: Blame ties the current active-only durable cron recovery implementation to 6a48258, which helps route the remaining terminal-row cleanup to the task-maintenance area owner history. (src/tasks/task-registry.maintenance.ts:447, 6a482584ee62)

Likely related people:

  • steipete: Git history and blame tie this person to stale cron/task-maintenance reconciliation, durable cron recovery, and recent task-maintenance changes in the central files. (role: recent area contributor and active-recovery introducer; confidence: high; commits: 7d1575b5df79, 1fae716a04e5, 6009b86f0de4; files: src/tasks/task-registry.maintenance.ts, src/tasks/task-registry.maintenance.issue-60299.test.ts, CHANGELOG.md)
  • likewen-tech: Recent terminal task-ledger work changed src/tasks/task-registry.ts, adjacent to the terminal update path this PR changes to clear stale errors. (role: adjacent terminal task-ledger contributor; confidence: medium; commits: 86328585fac2; files: src/tasks/task-registry.ts, src/tasks/task-registry.test.ts)
  • NikolaFC: Related discussion mentions this person, and nearby restart/liveness work touched task-maintenance restart blockers adjacent to the false lost cron-row path. (role: adjacent restart/liveness contributor; confidence: medium; commits: 103cdd9d96f8; files: src/tasks/task-registry.maintenance.ts, src/gateway/server-reload-handlers.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 56eb23dda4dd.

@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label May 8, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from 2e8f157 to 712bad0 Compare May 8, 2026 07:57
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 8, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from 712bad0 to c598295 Compare May 8, 2026 09:39
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels May 8, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 8, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from c598295 to 726b54b Compare May 8, 2026 10:31
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from 726b54b to c7c4970 Compare May 8, 2026 10:37
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 8, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from c7c4970 to 4574ac8 Compare May 9, 2026 03:02
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 9, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 9, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from 4574ac8 to 2af5f84 Compare May 9, 2026 07:22
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 9, 2026
@liaoandi
liaoandi force-pushed the fix/tasks-recover-terminal-lost-cron branch from 2af5f84 to 137187c Compare May 12, 2026 04:31
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels May 12, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. labels May 24, 2026
@clawsweeper

clawsweeper Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui channel: discord Channel integration: discord channel: nostr Channel integration: nostr cli CLI command changes commands Command implementations docker Docker and sandbox tooling extensions: acpx extensions: codex extensions: deepinfra extensions: diagnostics-otel Extension: diagnostics-otel extensions: diffs extensions: google extensions: kimi-coding extensions: memory-wiki extensions: oc-path extensions: openrouter extensions: qa-lab gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. scripts Repository scripts size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant