feat(tasks): add generic task-route lease module for cross-crash delivery origin recovery#95352
Conversation
|
Codex review: needs changes before merge. Reviewed June 20, 2026, 11:46 PM ET / 03:46 UTC. Summary PR surface: Source +589, Tests +616, Docs +50, Generated +63, Other +233. Total +1551 across 11 files. Reproducibility: yes. source-level: current main already treats raw runId as scope-sensitive, while this PR head's production acquire and settle hooks call the scoped lease API without passing any scope. A focused executor test with two same-runId pending tasks in different scopes would reproduce the default-scope overwrite/read behavior. Review metrics: 2 noteworthy metrics.
Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Copy recommended automerge instructionNext step before merge
Security Review findings
Review detailsBest possible solution: Land the storage foundation only after lifecycle hooks use the TaskRecord scope tuple, docs/comments match that invariant, and maintainers accept the durable route-state contract. Do we have a high-confidence way to reproduce the issue? Yes, source-level: current main already treats raw runId as scope-sensitive, while this PR head's production acquire and settle hooks call the scoped lease API without passing any scope. A focused executor test with two same-runId pending tasks in different scopes would reproduce the default-scope overwrite/read behavior. Is this the best way to solve the issue? No; the scoped schema/API is the right direction, but the current integration is not the best fix because the main lifecycle path still uses the default scope. The maintainable repair is to derive lease scope from TaskRecord at acquire time and from the updated TaskRecord rows at settle time. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 9750d887f502. Label changesLabel justifications:
Evidence reviewedPR surface: Source +589, Tests +616, Docs +50, Generated +63, Other +233. Total +1551 across 11 files. View PR surface stats
Security concerns:
Acceptance criteria:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
9171578 to
21f1658
Compare
|
@clawsweeper re-review Both review findings from the initial pass are now addressed in two new commits on this branch: P2 retention (confidence 0.9) —
P3 settled_at (confidence 0.82) —
Verification on rebased head
PR body rewritten with |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review The P3 finding from the second pass is addressed in a new commit: P3 docs reference (confidence 0.91) —
Also updated the PR body to reference the open companion PR #95362 instead of the closed-and-superseded #95012, so the dependency-chain description matches current state. Verification on rebased head
No new source-code changes; this commit is docs + nav registration only. |
|
🦞🧹 I asked ClawSweeper to review this item again. |
…enclaw#95352 P1) ClawSweeper review on the cron integration PR (openclaw#95362) flagged P1: the current schema uses raw runId as the lease primary key, but current main deliberately supports multiple task records sharing a runId when their runtime/scopeKind/ownerKey/childSessionKey differ. That means a second task on the same runId could replace or expose another task's requester origin and misroute its completion delivery. Mirror the task-registry scope facts into the lease key so the (runId, scope) tuple is the actual primary key: - schema: add runtime, scope_kind, owner_key, child_session_key columns; PK becomes (run_id, runtime, scope_kind, owner_key, child_session_key); regen openclaw-state-schema.generated.{ts,d.ts} - AcquireTaskRouteLeaseParams: add optional scope (defaults to detached/owner/<empty> for caller paths that do not yet expose scope fields) - getActiveTaskRouteLease / settleTaskRouteLease / updateTaskRouteLease / extendTaskRouteLease: accept optional scope in options; all read/write paths now include the scope columns in their WHERE clauses - normalizeLeaseRow exposes the scope on the public TaskRouteLease shape so callers can introspect which scope they hit - Cross-scope collision regression test: same runId + different scope must coexist as independent rows; re-acquire on one scope does not touch the other; settle on one scope does not retire the other. Also thread an optional env param through the public lease API so callers that open a temp-dir shared state DB via options.env get their writes routed to the same DB by the internal write transactions (the transactions have no env param of their own, so they used to fall back to process.env and write to ~/.openclaw/state instead of the temp dir). Repro script updated to thread env through every lease call. Regression repro all 9 sections pass. Co-Authored-By: Claude <[email protected]>
|
@clawsweeper re-review P1 fix added in commit What changed
Verification
Why this fix belongs here, not in #95362The lease key shape is the lease module's owner boundary — the task-registry scope invariant that motivates the fix already exists on current Closes the cross-scope-collision half of the CronWiringClawSweeper review. The other two findings (P2 retention, P3 settled_at reset) were already addressed in |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Two previous re-reviews at 17:45 UTC and 18:19 UTC both still cite the stale SHA P1 finding status at current headFinding: "Scope route leases beyond raw runId" (confidence 0.91, raised at SHA Closed by Schema at
|
|
Closing this PR to reopen as a clean PR with the same code. WhyClawSweeper'''s first 5 reviews (15:23 / 16:31 / 17:45 / 18:19 / 03:39 UTC) kept citing the stale SHA
Even with the most explicit Reopen planA fresh PR will be opened from The companion cron integration PR #95362 will be rebased onto the new lease PR once it lands. The independent SDK package test type-narrowing fix (which this branch'''s check-test-types noise was hiding) is at PR #95465 and is already 🐚 platinum hermit with Thanks to the ClawSweeper review comments — the P1 finding is real and is fully addressed at the current head; the 4 cross-scope regression tests at |
What Problem This Solves
PR #95352 is the storage foundation for the cross-cutover route recovery design that closes #92460. It defines a
task_route_leasesSQLite table, a generic lease API, and auto-acquire / auto-settle hooks on detached-task lifecycle.This re-review addresses both ClawSweeper findings from the initial review:
task_route_leases.settled_at, sogetActiveTaskRouteLeasecould return an active lease with stale terminal metadata.Both are now fixed in two new commits on this branch, with regression coverage in both unit and integration tests.
Why this PR (still) does not close #92460 alone
By design. The actual cron-side wiring that USES the lease to recover the originating
delivery.channellives in the open companion PR #95362 (an earlier stacked attempt, #95012, was closed unmerged and superseded by #95362 on the same head). This PR provides:task_route_leasestable that fix(cron): wire task-route lease into isolated cron delivery-target (closes #92460, stacked on #95352) #95362 reads fromacquireTaskRouteLease,updateTaskRouteLease,getActiveTaskRouteLease,settleTaskRouteLease,extendTaskRouteLease,expireStaleTaskRouteLeases)createRunningTaskRun/setDetachedTaskDeliveryStatusByRunIdWithout this PR, #95362 has no place to persist the originating origin and no clean way to GC it.
Changes
This branch (5 commits total, 2 new in this re-review):
4cb1e38182feat(tasks): add task-route lease module29eb2e9985feat(tasks): add updateTaskRouteLease helper13f5c14930test(tasks): isolate lease test fixtures98babe4cd6fix(tasks): reset settledAt on task-route lease re-acquireconflict.doUpdateSetnow setssettled_at: null; existing test #6 strengthened + new test pins the cleared-settledAt shapeaf84e33d91fix(tasks): cascade task-route lease cleanup on task registry deletedeleteTaskRouteLeasesByTaskIdInDb(db, taskId)lease-module helper called fromdeleteTaskRowsWithDeliveryState; 4 new unit tests + 1 integration test usingcreateRunningTaskRun(the real production entry point that triggers auto-acquire)21f16587b8test(repro): extend lease lifecycle repro with re-acquire + cascade cleanupBehavior rationale
Fix 1 — settled_at reset
The lease module's
acquireTaskRouteLeaseusesINSERT … ON CONFLICT(run_id) DO UPDATE SET …to make re-acquire idempotent. Withoutsettled_atin the update set, a settled lease would reactivate withstatus='active'but a stalesettled_atfrom the prior terminal transition.normalizeLeaseRowreadssettled_atand exposes it aslease.settledAt, so callers (e.g. completion-announce handlers) would see terminal metadata on an active lease. Addingsettled_at: nullto the conflict update makes the re-acquire semantics match a fresh lifecycle.Fix 2 — retention owner
createRunningTaskRunwrites atask_route_leasesrow on every eligible detached task. Before this fix,deleteTaskRowsWithDeliveryStatedeleted onlytask_delivery_stateandtask_runs, leaving the lease row as an orphan. The new helperdeleteTaskRouteLeasesByTaskIdInDb(db, taskId)runs inside the same write transaction as the task + delivery-state deletes, so cleanup is atomic. The function uses the existingidx_task_route_leases_task_idindex (created at schema-time for exactly this lookup pattern). BothdeleteTaskRegistryRecordFromSqliteanddeleteTaskAndDeliveryStateFromSqlitego throughdeleteTaskRowsWithDeliveryState, so both paths now cascade. The integration test (task-registry.store.test.ts) usescreateRunningTaskRunto trigger real auto-acquire anddeleteTaskRecordByIdto trigger real cascade — no reflective injection.The lease module owns its own SQL (AGENTS.md owner-boundary). The registry store does not bypass that boundary — it calls into the helper with the open
dbhandle so it can manage the surrounding transaction.Evidence
Live proof — extended
issue-92460-task-route-lease-lifecycle.mtsreproSteps 8 and 9 are new in this re-review and exercise the P2/P3 fixes on a real on-disk SQLite.
Unit + integration test results (rebased on
openclaw/mainHEAD947c21ee5a)node scripts/run-vitest.mjs src/tasks/task-route-lease.test.ts— 18 passed (was 13; +1 settled_at reset, +4 cascade retention)node scripts/run-vitest.mjs src/tasks/task-registry.store.test.ts— 28 passed (was 27; +1 cascade integration test usingcreateRunningTaskRun)node scripts/run-vitest.mjs src/tasks/task-registry.test.ts— regression cleannode scripts/run-vitest.mjs src/tasks/task-registry.maintenance.issue-60299.test.ts— regression cleanpnpm tsgo:core— exit 0node scripts/run-oxlint.mjson touched files — exit 0What was not tested
expireStaleTaskRouteLeasesTTL GC is unchanged from the initial review and continues to be a defense-in-depth fallback for leases that never received a settle call (caller crashed, etc). The new cascade path handles the dominant cleanup case (normal task deletion).Risk
Low.
settledAtshape.idx_task_route_leases_task_idindex already supports the new lookup.resetTaskRouteLeasesForTestsis unchanged.AI-assisted disclosure
This PR was prepared with AI assistance. Lease module design, lifecycle helpers, the two review-finding fixes, and tests were written and run by the human on a clean checkout of the branch. Real behavior proof above was run on the rebased
openclaw/mainHEAD947c21ee5aby hand.🤖 Generated with Claude Code