Skip to content

Commit b267ac7

Browse files
committed
fix(delegation): close the four confirmed #1495 liveness and repair races
What: - Task.ts stops the delegated-child liveness heartbeat on the non-abort rejection paths of startTask and resumeTaskFromHistory. ClineProvider's cleanupFailedHistoryTask no longer returns early for non-settlement errors, so every failed scheduled run tears the task down. - isLivenessSignalFresh bounds the future side of the freshness window (Math.abs(now - timestamp) < thresholdMs). getChildFileMtimeMs moves its transient-stat sentinel one millisecond inside that bound so it still reads as live, and both comments document the threat model. - assertChildRepairPreconditions aborts when the locked disk record is absent, so mergeHistoryDelta can no longer resurrect a peer-deleted child from the stale cache copy. writeAdministrativeRepair releases a local ownership claim that lands after the guard but before the commit, because the guard's own check is synchronous and cannot see it. - recordTaskActivity's merge callback skips a beat whose ownership was released, so a heartbeat queued before markLocallyInactive cannot refresh mtime and lastActivityAt for a session that is gone. Why: Each of these let a dead delegated child keep looking alive, or let a repair recreate a record a peer had deleted. The parent then stayed delegated to a child no session owned, across startup and every periodic tick, which is the #1469 bug class this PR exists to close. Impact: Production change holds the mutation-diff gate at 498 of 500 changed executable lines, the same total the branch carried before this round. New coverage lives in spec files, which the gate excludes. Lifecycle mutations stay in the shared reducers; the model doc and its liveness comment now describe the bounded window and the sentinel.
1 parent 8c38908 commit b267ac7

10 files changed

Lines changed: 620 additions & 62 deletions

File tree

‎docs/architecture/task-lifecycle-model.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -42,19 +42,19 @@ TLA+/PlusCal or Quint with TLC becomes a better fit when the lifecycle needs tem
4242

4343
## Production mapping
4444

45-
| Model concept | Production concept |
46-
| --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
47-
| Task record and status | `HistoryItem` persisted by `TaskHistoryStore` |
48-
| `delegate(parent, child)` | `ClineProvider.delegateParentAndOpenChild` |
49-
| `interrupt(child)` | cancellation or eviction through `markDelegatedChildInterrupted` |
50-
| `complete(child)` | `ClineProvider.reopenParentFromDelegation` |
51-
| `abandon(child)` | `ClineProvider.abandonSubtask` |
52-
| Pending-action settlement | `TaskHistoryStore.clearPendingActionIfMatching` compare-and-clear in the rejected-delegation settlement path (#1714) |
53-
| `reconcileStartup(parent)` | startup/periodic `TaskHistoryStore.reconcileDelegationStateCore` orphan repair |
54-
| `markLiveElsewhere(child)` / `expireLiveElsewhere(child)` | child history-file mtime recent vs stale past `LIVE_CHILD_MTIME_THRESHOLD_MS` (abstracted; no wall clock in model) |
55-
| `heartbeat(child)` / `expireHeartbeat(child)` | persisted `lastActivityAt` liveness heartbeat fresh vs stale past `LIVE_CHILD_MTIME_THRESHOLD_MS` (production: the owning session's throttled heartbeat across the child's whole active lifetime — long streaming turns, long tool calls, and ask-idle periods alike — shared predicate `isDelegatedChildLive`) |
56-
| Atomic event step | `atomicReadAndUpdate`, `atomicUpdatePair`, and per-parent delegation transition lock |
57-
| Event interleaving | Competing completion, cancellation, abandonment, and new delegation calls |
45+
| Model concept | Production concept |
46+
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
47+
| Task record and status | `HistoryItem` persisted by `TaskHistoryStore` |
48+
| `delegate(parent, child)` | `ClineProvider.delegateParentAndOpenChild` |
49+
| `interrupt(child)` | cancellation or eviction through `markDelegatedChildInterrupted` |
50+
| `complete(child)` | `ClineProvider.reopenParentFromDelegation` |
51+
| `abandon(child)` | `ClineProvider.abandonSubtask` |
52+
| Pending-action settlement | `TaskHistoryStore.clearPendingActionIfMatching` compare-and-clear in the rejected-delegation settlement path (#1714) |
53+
| `reconcileStartup(parent)` | startup/periodic `TaskHistoryStore.reconcileDelegationStateCore` orphan repair |
54+
| `markLiveElsewhere(child)` / `expireLiveElsewhere(child)` | child history-file mtime inside the bounded freshness window (`\|now - mtime\| < LIVE_CHILD_MTIME_THRESHOLD_MS`) vs outside it; the store's transient-stat sentinel sits one millisecond inside the future bound (abstracted; no wall clock in model) |
55+
| `heartbeat(child)` / `expireHeartbeat(child)` | persisted `lastActivityAt` liveness heartbeat fresh vs stale against the same bounded window (production: the owning session's throttled heartbeat across the child's whole active lifetime — long streaming turns, long tool calls, and ask-idle periods alike — shared predicate `isDelegatedChildLive`) |
56+
| Atomic event step | `atomicReadAndUpdate`, `atomicUpdatePair`, and per-parent delegation transition lock |
57+
| Event interleaving | Competing completion, cancellation, abandonment, and new delegation calls |
5858

5959
The model has three fixed task slots, enough to cover competing siblings and a nested parent-child-grandchild chain, plus two abstract booleans per slot recording whether an active child's session shows life: one for ownership by another window (recent history-file mtime) and one for a fresh persisted `lastActivityAt` heartbeat from the owning session. It explores every reachable interleaving through depth 12, deduplicating canonical states. Representative checks also exercise rejected operations that do not create a new state: a second concurrent delegation while the first child is active, stale completion after re-delegation, late completion after abandonment, completion after interruption, and nested completion. Named semantic landmarks require the graph to retain interrupted-child re-delegation and nested delegation even when the raw state total changes, a delegated parent whose active child is live in another window surviving startup reconciliation unchanged, a delegated parent whose active child heartbeats through a long streaming turn (stale mtime, fresh `lastActivityAt`) likewise surviving unchanged, and a stale-mtime, stale-heartbeat (crash-orphan) active child being repaired to `interrupted` with the parent returned to `active` only through `reconcileStartup`.
6060

@@ -178,7 +178,7 @@ The task delegation checker currently enforces:
178178
5. Parent-child lineage is acyclic.
179179
6. Completed task records cannot be changed by later lifecycle events.
180180
7. Active-child re-delegation, stale completion after ownership moves to another child, duplicate/late completion, and abandonment of a live child are rejected by the shared production guards.
181-
8. No transition may clear a delegated parent's link to a child that is active and live, where live means either marked live-elsewhere (`ModelState.liveElsewhere` true, recent history-file mtime) or heartbeat-alive (`ModelState.heartbeatAlive` true, fresh persisted `lastActivityAt`); reconciliation repairs the link only when both flags are false. This encodes the PR #1495 cross-window misrepair bug class — which broke delegation links so subtask completion could not return to the parent — extended by the liveness-heartbeat fix so a child streaming a long turn (minutes without any other history-file write) is likewise preserved.
181+
8. No transition may clear a delegated parent's link to a child that is active and live, where live means either marked live-elsewhere (`ModelState.liveElsewhere` true, history-file mtime inside the bounded freshness window) or heartbeat-alive (`ModelState.heartbeatAlive` true, fresh persisted `lastActivityAt` inside the same window); reconciliation repairs the link only when both flags are false. This encodes the PR #1495 cross-window misrepair bug class — which broke delegation links so subtask completion could not return to the parent — extended by the liveness-heartbeat fix so a child streaming a long turn (minutes without any other history-file write) is likewise preserved. The window is bounded in BOTH directions: a timestamp far ahead of `now` (clock rollback, restored archive) is not evidence of life, and only the store's transient-stat sentinel, one millisecond inside the future bound, is treated as fresh.
182182
9. A rejected delegation settles only the exact matching pending `create_subtask` action. Settlement preserves status, lineage, and accounting, never clears a replacement or different-kind action, and never mutates a completed record. The settlement compare-and-clear reads the persisted record under the per-file lock, so it never clears from a stale host cache.
183183
10. A completion clears the completing child's pending action only when the completion event carries the exact matching action ID. A completion with no action ID or a different ID preserves the pending action.
184184

‎scripts/check-task-lifecycle.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,14 @@ type TaskMap = Record<TaskId, HistoryItem | undefined>
2323
*
2424
* - `liveElsewhere[id]` models the child's history-file mtime: true when the
2525
* mtime is "recent" (the child is owned by another window that is actively
26-
* persisting it), false when it is "stale" or the file is unreadable.
26+
* persisting it), false when it is "stale", far in the future, or the file
27+
* is unreadable.
2728
* - `heartbeatAlive[id]` models the persisted `lastActivityAt` heartbeat: true
2829
* when the owning session heartbeated within the threshold (Task's throttled
2930
* heartbeat runs across the child's whole active lifetime — long streaming
3031
* turns, long tool calls, and ask-idle periods that write nothing else to
31-
* the history file), false when the heartbeat is absent or stale.
32+
* the history file), false when the heartbeat is absent, stale, or far in
33+
* the future.
3234
*
3335
* The model never reads wall-clock time. Reconciliation treats the child as
3436
* live when EITHER signal is true and repairs it to interrupted only when

‎src/__tests__/removeClineFromStack-delegation.spec.ts‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -202,7 +202,11 @@ describe("ClineProvider failed history restoration cleanup", () => {
202202
expect(task.dispose).toHaveBeenCalledOnce()
203203
})
204204

205-
it("keeps the task active after an unrelated history resume failure", async () => {
205+
it("tears down the task after an unrelated history resume failure too", async () => {
206+
// Regression for the zombie-heartbeat cluster: a non-settlement rejection
207+
// used to return early, leaving the task registered with its liveness
208+
// heartbeat running. That kept the persisted record `active` and beating,
209+
// so local ticks and peer startup repair never saw the dead child.
206210
const cleanupListener = vi.fn()
207211
const task = {
208212
taskId: "failed-history-task",
@@ -213,19 +217,20 @@ describe("ClineProvider failed history restoration cleanup", () => {
213217
const taskRegistry = new TaskRegistry()
214218
taskRegistry.push(task)
215219
const taskEventListeners = new Map([[task, [cleanupListener]]])
220+
const log = vi.fn()
216221
const provider = {
217222
taskRegistry,
218223
taskEventListeners,
219-
log: vi.fn(),
224+
log,
220225
} as unknown as ClineProvider
221226

222227
await privateClineProvider.cleanupFailedHistoryTask.call(provider, task, new Error("history read failed"))
223228

224-
expect(taskRegistry.getById(task.taskId)).toBe(task)
225-
expect(taskRegistry.current).toBe(task)
226-
expect(cleanupListener).not.toHaveBeenCalled()
227-
expect(taskEventListeners.has(task)).toBe(true)
228-
expect(task.dispose).not.toHaveBeenCalled()
229+
expect(taskRegistry.getById(task.taskId)).toBeUndefined()
230+
expect(cleanupListener).toHaveBeenCalledOnce()
231+
expect(taskEventListeners.has(task)).toBe(false)
232+
expect(task.dispose).toHaveBeenCalledOnce()
233+
expect(log).toHaveBeenCalledWith(expect.stringContaining("history read failed"))
229234
})
230235
})
231236

0 commit comments

Comments
 (0)