Skip to content

Commit 86715bc

Browse files
committed
fix(delegation): make the repair commit mutually exclusive with a local claim
What: `safeWriteJson` gains a synchronous `preCommit` hook that runs immediately before the commit rename with no await in between, and `writeAdministrativeRepair` uses it to re-run the repair guard and the in-window ownership predicate at the commit point. The post-commit claim release is removed. Why: the round-1 guard was merge-time only. `markLocallyActive` is synchronous and takes no lock, so a claim landing during the temp-write awaits was invisible to it: the repair committed the child as `interrupted` and the parent as `active`, and releasing the claim afterwards undid nothing. A live resumed child was left interrupted and orphaned. Re-running the guard at the commit point aborts with NOTHING committed, so the parent half never runs and the live session keeps its record, its claim, and its delegation link. The same hook closes the missing-child cross-lock gap: the child-file absence check is now re-evaluated immediately before the parent rename. Impact: both the in-pass (`applyDelegationRepairIntent`) and replay (`replayDelegationRepairIntent`) paths get the protocol, since both write through `writeAdministrativeRepair`. The regression test now asserts child `active`, parent `delegated`, and the claim retained. Stale sentinel comments (threshold-1 / strict bound) are corrected in taskLifecycle.ts, its spec, TaskHistoryStore, and the lifecycle checker.
1 parent b267ac7 commit 86715bc

7 files changed

Lines changed: 266 additions & 65 deletions

File tree

‎scripts/check-task-lifecycle.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,10 @@ type TaskMap = Record<TaskId, HistoryItem | undefined>
2323
*
2424
* - `liveElsewhere[id]` models the child's history-file mtime: true when the
2525
* mtime is "recent" (the child is owned by another window that is actively
26-
* persisting it), false when it is "stale", far in the future, or the file
27-
* is unreadable.
26+
* persisting it) or unreadable (production reports the transient-stat
27+
* sentinel, one millisecond inside the future bound, so a stat failure counts
28+
* as life and a later tick retries), false when it is "stale" or far in the
29+
* future.
2830
* - `heartbeatAlive[id]` models the persisted `lastActivityAt` heartbeat: true
2931
* when the owning session heartbeated within the threshold (Task's throttled
3032
* heartbeat runs across the child's whole active lifetime — long streaming

‎src/core/task-persistence/TaskHistoryStore.ts‎

Lines changed: 42 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -503,10 +503,13 @@ export class TaskHistoryStore {
503503
*
504504
* A parent awaiting an `interrupted` or `delegated` child is left as-is — the child is
505505
* resumable. An `active` child is repaired only when it shows NO sign of life: its
506-
* history file mtime is stale (or unreadable) AND its persisted `lastActivityAt`
507-
* heartbeat is stale, per the shared `isDelegatedChildLive` predicate. A child that
508-
* is fresh on either signal is owned by a live session (another window, or this one
509-
* before a restart dropped its claim) and keeps its delegation link.
506+
* history file mtime is stale (or absent) AND its persisted `lastActivityAt`
507+
* heartbeat is stale, per the shared `isDelegatedChildLive` predicate. An
508+
* unreadable mtime is NOT evidence of absence: `getChildFileMtimeMs` reports the
509+
* transient-stat sentinel (`now + threshold - 1`), which counts as life and defers
510+
* repair to a later tick. A child that is fresh on either signal is owned by a live
511+
* session (another window, or this one before a restart dropped its claim) and keeps
512+
* its delegation link.
510513
*/
511514
private async reconcileDelegationState(persistedActiveIds: ReadonlySet<string>): Promise<void> {
512515
return this.withLock(() => this.reconcileDelegationStateCore(persistedActiveIds))
@@ -636,12 +639,9 @@ export class TaskHistoryStore {
636639
continue
637640
}
638641
// Re-check local ownership after the async stat await: the
639-
// persistedActiveIds snapshot was captured before this point, and
640-
// ClineProvider can claim the child for a live session in THIS
641-
// window (markLocallyActive, the eager claim in
642-
// createTaskWithHistoryItemUnlocked) while getChildFileMtimeMs was
643-
// in flight. The snapshot no longer reflects that claim, so the
644-
// child is no longer a crash orphan — skip the repair.
642+
// persistedActiveIds snapshot predates it, and ClineProvider can
643+
// claim the child for a live session in THIS window while the probe
644+
// was in flight, so the child is no longer a crash orphan.
645645
if (this.locallyActiveTaskIds.has(child.id)) {
646646
console.warn(
647647
`[TaskHistoryStore] Skipping repair for live child ${child.id} ` +
@@ -864,13 +864,7 @@ export class TaskHistoryStore {
864864
// Cross-instance liveness guard (same convention as reconcileDelegationStateCore): if this window crashed mid-repair and another window restarted the same child, the child's history file is being actively persisted there. Replaying the stale intent would overwrite the live child as "interrupted", so quarantine it instead. Only enforced when the replay would actually write the child record: a child already at its target needs no write, and parent-only completion must not be blocked by child liveness. A fresh mtime OR a fresh `lastActivityAt` heartbeat quarantines the intent; a transient stat failure reports the near-future sentinel (see `getChildFileMtimeMs`), which also counts as life and lets a later tick retry.
865865
if (!childAtTarget) {
866866
const mtimeMs = await this.getChildFileMtimeMs(child.id)
867-
const isLiveElsewhere = isDelegatedChildLive(
868-
child,
869-
Date.now(),
870-
TaskHistoryStore.LIVE_CHILD_MTIME_THRESHOLD_MS,
871-
mtimeMs,
872-
)
873-
if (isLiveElsewhere) {
867+
if (isDelegatedChildLive(child, Date.now(), TaskHistoryStore.LIVE_CHILD_MTIME_THRESHOLD_MS, mtimeMs)) {
874868
await this.quarantineDelegationRepairIntent(
875869
intent,
876870
`child live in another window (${this.describeLivenessSignal(child, mtimeMs)})`,
@@ -879,16 +873,11 @@ export class TaskHistoryStore {
879873
}
880874
}
881875

882-
// Same cross-process guard as the in-pass repair: each write
883-
// re-validates its preconditions under the target file's advisory
884-
// lock, so a peer write that landed after the pre-write liveness
885-
// check is observed and aborts the stale replay (the journal stays
886-
// for a later pass, and replay skips sides already at target).
887-
// Cache publication uses the committed records, so a side already at
888-
// target keeps whatever a peer wrote. The repaired record is also the
889-
// write payload (so a write path that bypasses the merge callback still
890-
// lands the target state) while the delta is applied on top of the
891-
// locked disk record (so peer-written fields survive).
876+
// Same guarded write as the in-pass repair: preconditions re-validate
877+
// under the target's advisory lock and again in `preCommit` right before
878+
// the commit, so a peer write or a local claim aborts the stale replay
879+
// (the journal stays for a later pass). Cache publication uses the
880+
// committed records, so a side already at target keeps a peer's write.
892881
let committedChild: HistoryItem = childAtTarget ? child : { ...child, ...CHILD_REPAIR_DELTA }
893882
let committedParent: HistoryItem = parentMatchesTargetState ? parent : { ...parent, ...PARENT_REPAIR_DELTA }
894883
try {
@@ -1038,10 +1027,15 @@ export class TaskHistoryStore {
10381027
* cache-derived copy; publishing the copy would re-introduce the stale
10391028
* fields in this window even though the disk write merged them correctly.
10401029
*
1041-
* Every repair write is an administrative one: it must never create a live
1042-
* session's ownership claim. The store lock does not cover `markLocallyActive`,
1043-
* which a provider can call between a repair guard and this commit, so a
1044-
* claim that appeared under the lock is released again after the commit.
1030+
* Every repair write is an administrative one: it must never interrupt a
1031+
* session that claimed the record while the repair was deciding. The store
1032+
* lock does not cover `markLocallyActive`, so the ownership predicate runs
1033+
* again in safeWriteJson's `preCommit`, synchronously and immediately before
1034+
* the commit rename. That window is the only one that can still change: the
1035+
* merge above already ran under the target's advisory lock, so no peer can
1036+
* write, and only an in-process claim can land. A claim observed there aborts
1037+
* with NOTHING committed, so the parent half never runs and the live session
1038+
* keeps its record, its claim, and its delegation link.
10451039
*/
10461040
private async writeAdministrativeRepair(
10471041
target: HistoryItem,
@@ -1051,14 +1045,18 @@ export class TaskHistoryStore {
10511045
const filePath = await this.getTaskFilePath(target.id)
10521046
const merge = mergeWithDisk(delta)
10531047
let committed: HistoryItem = target
1048+
let locked: unknown
10541049
await safeWriteJson(filePath, target, {
10551050
merge: (existing, incoming) => {
10561051
guard(existing, filePath)
1052+
locked = existing
10571053
return (committed = merge(existing, incoming) as HistoryItem)
10581054
},
1055+
preCommit: () => {
1056+
guard(locked, filePath)
1057+
this.assertNotClaimedLocally(target.id)
1058+
},
10591059
})
1060-
// A claim that landed after the guard but before this commit describes a session that started once the repair had already decided the record was an orphan. The write above interrupted it, so release the claim instead of letting it shield the interrupted child forever.
1061-
if (this.locallyActiveTaskIds.has(target.id)) this.markLocallyInactive(target.id)
10621060
// No cache update here: multi-file repairs must publish cache state
10631061
// atomically after ALL writes commit (see applyDelegationRepairIntent).
10641062
return committed
@@ -1093,8 +1091,17 @@ export class TaskHistoryStore {
10931091
) {
10941092
throw new RepairAbortedError("child mtime became fresh under lock (peer resumed)")
10951093
}
1096-
if (this.locallyActiveTaskIds.has(childId)) {
1097-
throw new RepairAbortedError("child claimed by this window during repair")
1094+
this.assertNotClaimedLocally(childId)
1095+
}
1096+
1097+
/**
1098+
* The in-window ownership predicate, shared by the under-lock guard and
1099+
* safeWriteJson's `preCommit`. A claim landing after the guard but before the
1100+
* commit must abort the write, not be compensated for after it.
1101+
*/
1102+
private assertNotClaimedLocally(taskId: string): void {
1103+
if (this.locallyActiveTaskIds.has(taskId)) {
1104+
throw new RepairAbortedError(`record ${taskId} claimed by this window during repair`)
10981105
}
10991106
}
11001107

‎src/core/task-persistence/__tests__/TaskHistoryStore.reconciliation.spec.ts‎

Lines changed: 148 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,10 @@ vi.mock("../../../utils/storage", () => ({
1818
const writeJson = async (
1919
filePath: string,
2020
data: unknown,
21-
options?: { merge?: (existing: unknown, incoming: unknown) => unknown },
21+
options?: {
22+
merge?: (existing: unknown, incoming: unknown) => unknown
23+
preCommit?: () => void
24+
},
2225
): Promise<void> => {
2326
await fs.mkdir(path.dirname(filePath), { recursive: true })
2427
let outgoing = data
@@ -34,6 +37,9 @@ const writeJson = async (
3437
}
3538
outgoing = options.merge(existing, data)
3639
}
40+
// Same contract as the real writer: the last-moment check runs immediately
41+
// before the write and a throw aborts it with nothing committed.
42+
options?.preCommit?.()
3743
await fs.writeFile(filePath, JSON.stringify(outgoing, null, "\t"), "utf8")
3844
}
3945

@@ -3269,16 +3275,18 @@ describe("TaskHistoryStore mutation-gate kill tests", () => {
32693275
// Later calls = the startup reconcile guard (line 506): recent -> child stays live.
32703276
return Promise.resolve(probeCalls === 1 ? FIXED_NOW - LIVE_CHILD_MTIME_THRESHOLD_MS : FIXED_NOW - 1_000)
32713277
})
3272-
// The under-lock guard's sync probe must mirror the same sequencing:
3273-
// the replay's child write re-checks exactly-threshold (NOT live ->
3278+
// The under-lock guard's sync probe must mirror the same sequencing.
3279+
// The replay's child write evaluates the guard twice: once in the
3280+
// merge callback and once in `preCommit` immediately before the
3281+
// commit rename. Both must see exactly-threshold (NOT live ->
32743282
// proceed); any later sync read (startup reconcile) sees recent.
32753283
let syncCalls = 0
32763284
spySyncMtimeProbe((fp, real) => {
32773285
if (!fp.endsWith(path.join(child.id, GlobalFileNames.historyItem))) {
32783286
return real(fp)
32793287
}
32803288
syncCalls++
3281-
return syncCalls === 1 ? FIXED_NOW - LIVE_CHILD_MTIME_THRESHOLD_MS : FIXED_NOW - 1_000
3289+
return syncCalls <= 2 ? FIXED_NOW - LIVE_CHILD_MTIME_THRESHOLD_MS : FIXED_NOW - 1_000
32823290
})
32833291

32843292
const s = (store = new TaskHistoryStore(tmpDir))
@@ -3801,13 +3809,16 @@ describe("TaskHistoryStore under-lock repair guards and heartbeat release", () =
38013809
}
38023810
})
38033811

3804-
it("releases a local ownership claim that lands under the repair lock so the interrupted child stays repairable", async () => {
3805-
// Regression for the K31PRO ownership-claim race in thread 4419747309.
3806-
// `markLocallyActive` does not take the store lock, so a provider can claim
3807-
// the child after the repair guard ran but before the commit lands. The
3808-
// repair then interrupts a child this window now believes it owns, and the
3809-
// claim would shield that interrupted record from every later pass. The
3810-
// commit must release the claim it observes after the write.
3812+
it("aborts the repair when a local ownership claim lands after the guard but before the commit", async () => {
3813+
// Regression for the K31PRO ownership-claim race (threads 4419747309 and
3814+
// 4420013366). `markLocallyActive` is synchronous and takes no lock, so a
3815+
// provider can claim the child after the repair guard ran and before the
3816+
// commit. A merge-time-only check cannot see that claim, and releasing the
3817+
// claim after the commit undoes nothing: the live child is already
3818+
// interrupted and its parent already severed. The claim must instead be
3819+
// mutually exclusive with the complete repair commit, so the repair aborts
3820+
// with NOTHING written and the live session keeps its record, its claim,
3821+
// and its delegation link.
38113822
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {})
38123823
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {})
38133824
try {
@@ -3822,7 +3833,7 @@ describe("TaskHistoryStore under-lock repair guards and heartbeat release", () =
38223833

38233834
// Simulate the claim landing inside the locked window: the guard has
38243835
// already passed (no claim visible to it), then the provider claims the
3825-
// id before the merge commits.
3836+
// id before the commit rename. `preCommit` must observe it and abort.
38263837
const probe = TaskHistoryStore.prototype as unknown as {
38273838
assertChildRepairPreconditions: (existing: unknown, filePath: string, childId: string) => void
38283839
}
@@ -3846,12 +3857,133 @@ describe("TaskHistoryStore under-lock repair guards and heartbeat release", () =
38463857
guardSpy.mockRestore()
38473858
}
38483859

3849-
expect(s.get(childId)?.status).toBe("interrupted")
3850-
// The claim that arrived after the guard must not survive the commit.
3860+
// Nothing was committed: the child is still active and the parent still
3861+
// awaits it, with both delegation pointers intact.
3862+
expect(s.get(childId)?.status).toBe("active")
3863+
expect(s.get(parentId)?.status).toBe("delegated")
3864+
expect(s.get(parentId)?.awaitingChildId).toBe(childId)
3865+
expect(s.get(parentId)?.delegatedToId).toBe(childId)
3866+
const childOnDisk = JSON.parse(
3867+
await fs.readFile(path.join(tmpDir, "tasks", childId, GlobalFileNames.historyItem), "utf8"),
3868+
) as HistoryItem
3869+
expect(childOnDisk.status).toBe("active")
3870+
// The claim the provider made mid-repair survives, so the session that
3871+
// owns the child keeps its heartbeat and its shield from later passes.
38513872
const owned = (s as unknown as { locallyActiveTaskIds: Set<string> }).locallyActiveTaskIds
3852-
expect(owned.has(childId)).toBe(false)
3873+
expect(owned.has(childId)).toBe(true)
38533874
const released = (s as unknown as { releasedLocalOwnershipIds: Set<string> }).releasedLocalOwnershipIds
3854-
expect(released.has(childId)).toBe(true)
3875+
expect(released.has(childId)).toBe(false)
3876+
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining("claimed by this window during repair"))
3877+
expect(errorSpy).not.toHaveBeenCalled()
3878+
} finally {
3879+
warnSpy.mockRestore()
3880+
errorSpy.mockRestore()
3881+
}
3882+
})
3883+
3884+
it("aborts a replayed repair when a local ownership claim lands before its commit", async () => {
3885+
// The replay path writes through the same `writeAdministrativeRepair`, so
3886+
// it needs the same claim/commit protocol: a claim landing between the
3887+
// replay's liveness check and the child commit must abort the replay
3888+
// instead of interrupting the session that just claimed the child.
3889+
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {})
3890+
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {})
3891+
try {
3892+
const childId = "child-claim-replay"
3893+
const parentId = "parent-claim-replay"
3894+
const [parent, child] = delegatedPair(parentId, childId)
3895+
await seedItems(tmpDir, [parent, child])
3896+
await fs.writeFile(
3897+
path.join(tmpDir, "tasks", GlobalFileNames.delegationRepairIntent),
3898+
JSON.stringify(makeRepairIntent(parent, child)),
3899+
)
3900+
3901+
const s = (store = new TaskHistoryStore(tmpDir))
3902+
await s.reconcile({ forceRefresh: true })
3903+
installStaleChildInjector(childId)
3904+
3905+
const probe = TaskHistoryStore.prototype as unknown as {
3906+
assertChildRepairPreconditions: (existing: unknown, filePath: string, childId: string) => void
3907+
}
3908+
const originalGuard = probe.assertChildRepairPreconditions
3909+
const guardSpy = vi.spyOn(probe, "assertChildRepairPreconditions").mockImplementation(function (
3910+
this: TaskHistoryStore,
3911+
existing: unknown,
3912+
filePath: string,
3913+
id: string,
3914+
) {
3915+
originalGuard.call(this, existing, filePath, id)
3916+
s.markLocallyActive(id)
3917+
})
3918+
3919+
try {
3920+
const internals = TaskHistoryStore.prototype as unknown as {
3921+
replayDelegationRepairIntent: () => Promise<void>
3922+
}
3923+
await internals.replayDelegationRepairIntent.call(s)
3924+
} finally {
3925+
guardSpy.mockRestore()
3926+
}
3927+
3928+
expect(s.get(childId)?.status).toBe("active")
3929+
expect(s.get(parentId)?.status).toBe("delegated")
3930+
expect(s.get(parentId)?.awaitingChildId).toBe(childId)
3931+
const owned = (s as unknown as { locallyActiveTaskIds: Set<string> }).locallyActiveTaskIds
3932+
expect(owned.has(childId)).toBe(true)
3933+
expect(errorSpy).not.toHaveBeenCalled()
3934+
} finally {
3935+
warnSpy.mockRestore()
3936+
errorSpy.mockRestore()
3937+
}
3938+
})
3939+
3940+
it("aborts the parent half when a claim for the parent lands before its commit", async () => {
3941+
// The parent half's guard checks only the parent record's delegation, never
3942+
// ownership, so the explicit claim check in `preCommit` is what keeps a
3943+
// freshly resumed parent from being severed while it runs. Without it the
3944+
// parent commits as active with the link cleared, and the resumed session
3945+
// is left claim-less and beat-skipped for the rest of its life.
3946+
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {})
3947+
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {})
3948+
try {
3949+
const childId = "child-parent-claim"
3950+
const parentId = "parent-parent-claim"
3951+
const [parent, child] = delegatedPair(parentId, childId)
3952+
await seedItems(tmpDir, [parent, child])
3953+
3954+
const s = (store = new TaskHistoryStore(tmpDir))
3955+
await s.initialize()
3956+
installStaleChildInjector(childId)
3957+
3958+
const probe = TaskHistoryStore.prototype as unknown as {
3959+
assertParentRepairPreconditions: (existing: unknown, childId: string) => void
3960+
}
3961+
const originalGuard = probe.assertParentRepairPreconditions
3962+
const guardSpy = vi.spyOn(probe, "assertParentRepairPreconditions").mockImplementation(function (
3963+
this: TaskHistoryStore,
3964+
existing: unknown,
3965+
id: string,
3966+
) {
3967+
originalGuard.call(this, existing, id)
3968+
s.markLocallyActive(parentId)
3969+
})
3970+
3971+
try {
3972+
const internals = TaskHistoryStore.prototype as unknown as {
3973+
runPeriodicDelegationReconciliation: () => Promise<void>
3974+
}
3975+
await internals.runPeriodicDelegationReconciliation.call(s)
3976+
} finally {
3977+
guardSpy.mockRestore()
3978+
}
3979+
3980+
// The child half committed (the repair was already past its guard), but
3981+
// the parent half aborted: the link is intact and the parent's claim
3982+
// survives, so the resumed parent keeps its heartbeat and its shield.
3983+
expect(s.get(parentId)?.status).toBe("delegated")
3984+
expect(s.get(parentId)?.awaitingChildId).toBe(childId)
3985+
const owned = (s as unknown as { locallyActiveTaskIds: Set<string> }).locallyActiveTaskIds
3986+
expect(owned.has(parentId)).toBe(true)
38553987
expect(errorSpy).not.toHaveBeenCalled()
38563988
} finally {
38573989
warnSpy.mockRestore()

‎src/core/task-persistence/__tests__/taskLifecycle.spec.ts‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,10 @@ describe("isLivenessSignalFresh", () => {
124124
})
125125

126126
it("treats a near-future timestamp as fresh (clock skew and the transient-stat sentinel)", () => {
127-
// The transient-stat sentinel is exactly `now + THRESHOLD`; keeping it
128-
// inside the window is what makes a stat failure skip repair instead of
129-
// cutting a possibly-live child out of its delegation link.
127+
// The transient-stat sentinel is `now + THRESHOLD - 1`, one millisecond
128+
// inside the strict bound; keeping it inside the window is what makes a
129+
// stat failure skip repair instead of cutting a possibly-live child out
130+
// of its delegation link.
130131
expect(isLivenessSignalFresh(NOW + THRESHOLD - 1, NOW, THRESHOLD)).toBe(true)
131132
expect(isLivenessSignalFresh(NOW + 1_000, NOW, THRESHOLD)).toBe(true)
132133
})
@@ -174,7 +175,7 @@ describe("isDelegatedChildLive", () => {
174175
})
175176

176177
it("stays live when only the mtime is near-future (the transient-stat sentinel)", () => {
177-
// The sentinel must keep working: a stat failure reports `now + THRESHOLD`,
178+
// The sentinel must keep working: a stat failure reports `now + THRESHOLD - 1`,
178179
// which is inside the bounded window, so repair is skipped and retried.
179180
expect(isDelegatedChildLive({}, NOW, THRESHOLD, NOW + THRESHOLD - 1)).toBe(true)
180181
})

0 commit comments

Comments
 (0)