Skip to content

Commit a17edba

Browse files
committed
fix: prepare interrupted resume history with compare-and-set
1 parent 1e0e968 commit a17edba

4 files changed

Lines changed: 149 additions & 19 deletions

File tree

‎docs/architecture/task-lifecycle-model.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ Production completion also accepts a recovery-compatible `active` parent that st
6363

6464
Explicit resume validates both records under file locks. A linked child resume holds the parent file's advisory lock across the parent ownership check and the child activation, so a competing host cannot re-delegate the parent in between. The re-delegation write (`atomicUpdateParentDelegation`) re-reads the awaited child's on-disk record under the child record's own advisory lock inside the parent-file lock, so a peer write's backup-rename window cannot make a valid child record look missing, and rejects the delegation unless that child is still interrupted. The two guards share one lock order, parent file before child file, so they close the competing resume and re-delegation interleavings in both directions. The child-file merge still compares the caller's expected parent with the authoritative child backlink. The resume rejects an abandonment that committed before it, even in its partial-write window.
6565

66-
Representative reducer scenarios cover stale linked and standalone approvals. Two-store tests cover abandonment after the parent precheck and both resume/re-delegation orderings. A declined resume returns before any store transition for both `resume_task` and `resume_completed_task`. This acceptance gate is deliberate and replaces the previous fall-through into the task loop. An accepted resume completes its fallible history reads and writes before the activation commit, so a preparation failure leaves the record interrupted and retryable. A failed preparation write also stops the resume, and the Task restores the authoritative API conversation history in memory. Before its destructive preparation write the resume first stores a durable pre-resume recovery copy of the original history in the task directory under a UUID-distinct name, so a failed preparation or restore cannot lose the original across a restart. When the activation commit rejects, the Task restores the persisted history through a conditional write whose status gate, content compare, and commit all run under the API-history file's advisory lock; the gate reads the task record under the record file's lock inside that scope, and no existing writer takes the reverse lock order. The restore commits only while the persisted record still shows the status the resume started from and the disk still holds the prepared slice, so an active record and a newer foreign history always win. An unreadable record status or an unreadable history fails the restore closed, and the restore never runs on unknown state. A record that is already active on disk keeps the prepared history, and the skipped restore is logged as a partial failure that names the recovery copy, which is retained because the active status alone does not prove this attempt's prepared history was the one accepted. A restore write that fails logs the same partial failure and names the recovery copy that still holds the original bytes. These guarantees bind writers that honor the shared advisory locks, which covers every in-repo writer of both files. The denied pending-action replay uses the same checked order: it persists the denial tool result before the activation commit, a failed append stops before activation and leaves the record interrupted with its durable pending action, and an activation rejection keeps the committed denial without an API-history restore, because the denial is a settled user decision rather than an unaccepted prepared slice. An accepted non-interrupted resume still runs without a store transition.
66+
Representative reducer scenarios cover stale linked and standalone approvals. Two-store tests cover abandonment after the parent precheck and both resume/re-delegation orderings. A declined resume returns before any store transition for both `resume_task` and `resume_completed_task`. This acceptance gate is deliberate and replaces the previous fall-through into the task loop. An accepted resume completes its fallible history reads and writes before the activation commit, so a preparation failure leaves the record interrupted and retryable. A failed preparation write also stops the resume, and the Task restores the authoritative API conversation history in memory. The preparation write of an interrupted resume is a compare-and-set under the API-history file's advisory lock. It replaces the original history only while the disk still holds that original and the record is still interrupted, so only one of two accepted resumes can prepare and the other stops before activation without writing. Before that write the resume first stores a durable pre-resume recovery copy of the original history in the task directory under a UUID-distinct name, so a failed preparation or restore cannot lose the original across a restart. When the activation commit rejects, the Task restores the persisted history through a conditional write whose status gate, content compare, and commit all run under the API-history file's advisory lock; the gate reads the task record under the record file's lock inside that scope, and no existing writer takes the reverse lock order. The restore commits only while the persisted record still shows the status the resume started from and the disk still holds the prepared slice, so an active record and a newer foreign history always win. An unreadable record status or an unreadable history fails the restore closed, and the restore never runs on unknown state. A record that is already active on disk keeps the prepared history, and the skipped restore is logged as a partial failure that names the recovery copy, which is retained because the active status alone does not prove this attempt's prepared history was the one accepted. A restore write that fails logs the same partial failure and names the recovery copy that still holds the original bytes. These guarantees bind writers that honor the shared advisory locks, which covers every in-repo writer of both files. The denied pending-action replay uses the same checked order: it persists the denial tool result before the activation commit, a failed append stops before activation and leaves the record interrupted with its durable pending action, and an activation rejection keeps the committed denial without an API-history restore, because the denial is a settled user decision rather than an unaccepted prepared slice. An accepted non-interrupted resume still runs without a store transition.
6767

6868
Rehydrated interrupted tasks omit lineage from message metadata saves, so their stale construction snapshots cannot restore severed links after rejection. Reverse-order stale abandonment and other live-task writers remain within the ownership/generation gaps below.
6969

‎src/core/task/Task.ts‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3409,7 +3409,18 @@ export class Task extends EventEmitter<TaskEvents> implements TaskLike {
34093409
messages: existingApiConversationHistory,
34103410
})
34113411

3412-
const persisted = await this.overwriteApiConversationHistory(modifiedApiConversationHistory)
3412+
let persisted: boolean
3413+
if (this.initialStatus === "interrupted") {
3414+
// Compare-and-set under the API-history file lock. Only one of two
3415+
// accepted resumes can replace the original history, so a rejected
3416+
// attempt never leaves its own prepared history on disk.
3417+
persisted = await this.prepareInterruptedResumeHistory(
3418+
existingApiConversationHistory,
3419+
modifiedApiConversationHistory,
3420+
)
3421+
} else {
3422+
persisted = await this.overwriteApiConversationHistory(modifiedApiConversationHistory)
3423+
}
34133424
if (!persisted) {
34143425
// The failed write leaves the disk file with the authoritative
34153426
// history or its safeWriteJson backup. Keep the recovery copy so
@@ -3498,6 +3509,34 @@ export class Task extends EventEmitter<TaskEvents> implements TaskLike {
34983509
}
34993510
}
35003511

3512+
private async prepareInterruptedResumeHistory(original: ApiMessage[], prepared: ApiMessage[]): Promise<boolean> {
3513+
const provider = this.providerRef.deref()
3514+
if (!provider) {
3515+
return false
3516+
}
3517+
let result: ResumeHistoryRestoreResult
3518+
try {
3519+
result = await restoreApiMessagesIfUnchanged({
3520+
taskId: this.taskId,
3521+
globalStoragePath: this.globalStoragePath,
3522+
expected: original,
3523+
replacement: prepared,
3524+
verifyCanRestore: async () =>
3525+
(await provider.taskHistoryStore.readValidatedDiskStatus(this.taskId)) === "interrupted"
3526+
? "proceed"
3527+
: "abort",
3528+
})
3529+
} catch (error) {
3530+
console.error(`[Task#${this.taskId}] prepareInterruptedResumeHistory: write failed`, error)
3531+
return false
3532+
}
3533+
if (result.outcome !== "restored") {
3534+
return false
3535+
}
3536+
this.hydrateApiConversationHistory(prepared)
3537+
return true
3538+
}
3539+
35013540
/**
35023541
* Restores the authoritative API history after the resume persisted
35033542
* prepared history but never activated the record. The interrupted record

‎src/core/task/__tests__/Task.persistence.spec.ts‎

Lines changed: 29 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2509,10 +2509,16 @@ describe("Task persistence", () => {
25092509
)
25102510
}
25112511

2512+
/** The interrupted resume prepares through the conditional write. The first call lands. */
2513+
function prepareLands(): void {
2514+
mockRestoreApiMessagesIfUnchanged.mockResolvedValueOnce({ outcome: "restored" })
2515+
}
2516+
25122517
it("stops before activation when the prepared api history fails to persist", async () => {
25132518
mockReadTaskMessages.mockResolvedValue([])
25142519
mockReadApiMessages.mockResolvedValue([{ role: "user", content: "Continue" }])
2515-
mockSaveApiMessages.mockRejectedValueOnce(new Error("disk full"))
2520+
mockRestoreApiMessagesIfUnchanged.mockRejectedValueOnce(new Error("disk full"))
2521+
vi.spyOn(console, "error").mockImplementation(() => {})
25162522

25172523
const task = makeResumeTask()
25182524
const loop = vi.spyOn(getTaskPersistenceAccess(task), "initiateTaskLoop")
@@ -2523,8 +2529,9 @@ describe("Task persistence", () => {
25232529

25242530
expect(mockProvider.resumeInterruptedTask).not.toHaveBeenCalled()
25252531
expect(loop).not.toHaveBeenCalled()
2526-
// Only the failed preparation write ran. The disk file never changed.
2527-
expect(mockSaveApiMessages).toHaveBeenCalledTimes(1)
2532+
// Only the failed conditional preparation write ran.
2533+
expect(mockSaveApiMessages).not.toHaveBeenCalled()
2534+
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(1)
25282535
// The durable pre-resume copy exists and stays, because the
25292536
// safeWriteJson rollback that restored the original on disk may
25302537
// itself have failed.
@@ -2547,6 +2554,7 @@ describe("Task persistence", () => {
25472554
mockProvider.resumeInterruptedTask = vi
25482555
.fn()
25492556
.mockRejectedValue(new LifecycleTransitionError("parent linkage changed"))
2557+
prepareLands()
25502558
runGateInsideRestore()
25512559

25522560
const task = makeResumeTask()
@@ -2557,12 +2565,16 @@ describe("Task persistence", () => {
25572565
expect(mockProvider.resumeInterruptedTask).toHaveBeenCalledTimes(1)
25582566
expect(loop).not.toHaveBeenCalled()
25592567
// The preparation slice is the only content that reached the disk layer.
2560-
const savedMessages = vi.mocked(mockSaveApiMessages).mock.calls.map((call) => call[0].messages)
2561-
expect(savedMessages).toEqual([[]])
2568+
expect(mockSaveApiMessages).not.toHaveBeenCalled()
2569+
const prepareOptions = vi.mocked(mockRestoreApiMessagesIfUnchanged).mock.calls[0][0]
2570+
expect(prepareOptions.expected).toEqual([
2571+
{ role: "user", content: "Continue", messageId: expect.any(String) },
2572+
])
2573+
expect(prepareOptions.replacement).toEqual([])
25622574
// The restore compared against the prepared slice and carried the
25632575
// authoritative history as its replacement.
2564-
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(1)
2565-
const restoreOptions = vi.mocked(mockRestoreApiMessagesIfUnchanged).mock.calls[0][0]
2576+
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(2)
2577+
const restoreOptions = vi.mocked(mockRestoreApiMessagesIfUnchanged).mock.calls[1][0]
25662578
expect(restoreOptions.expected).toEqual([])
25672579
expect(restoreOptions.replacement).toEqual([
25682580
{ role: "user", content: "Continue", messageId: expect.any(String) },
@@ -2577,7 +2589,7 @@ describe("Task persistence", () => {
25772589
.mockResolvedValueOnce([{ role: "user", content: "Continue" }])
25782590
.mockResolvedValueOnce([{ role: "user", content: "Continue" }])
25792591
// The preparation write lands. The locked restore write then fails.
2580-
mockSaveApiMessages.mockResolvedValueOnce(undefined)
2592+
prepareLands()
25812593
mockRestoreApiMessagesIfUnchanged.mockRejectedValueOnce(new Error("restore disk full"))
25822594
mockProvider.resumeInterruptedTask = vi
25832595
.fn()
@@ -2597,8 +2609,7 @@ describe("Task persistence", () => {
25972609
// through the Task save helper.
25982610
expect(retry).not.toHaveBeenCalled()
25992611
// The prepared slice is the last content that reached the disk layer.
2600-
const savedMessages = vi.mocked(mockSaveApiMessages).mock.calls.map((call) => call[0].messages)
2601-
expect(savedMessages).toEqual([[]])
2612+
expect(mockSaveApiMessages).not.toHaveBeenCalled()
26022613
const recoveryCopyPath = await mockCreateResumeHistoryRecoveryCopy.mock.results[0].value
26032614
// The partial-failure log names the durable recovery copy, so the
26042615
// original history stays identifiable on disk after a restart.
@@ -2619,17 +2630,18 @@ describe("Task persistence", () => {
26192630
mockProvider.taskHistoryStore.readValidatedDiskStatus = vi
26202631
.fn()
26212632
.mockRejectedValue(new LifecycleTransitionError("no valid disk record"))
2633+
prepareLands()
26222634
runGateInsideRestore()
26232635
const consoleError = vi.spyOn(console, "error").mockImplementation(() => {})
26242636

26252637
const task = makeResumeTask()
26262638

26272639
await expect(task["resumeTaskFromHistory"]()).rejects.toThrow("parent linkage changed")
26282640

2629-
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(1)
2641+
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(2)
26302642
// The gate rejected inside the lock scope, so the restore never
26312643
// compared or wrote, and the recovery copy stays on disk.
2632-
expect(mockSaveApiMessages).toHaveBeenCalledTimes(1)
2644+
expect(mockSaveApiMessages).not.toHaveBeenCalled()
26332645
expect(mockRemoveResumeHistoryRecoveryCopy).not.toHaveBeenCalled()
26342646
expect(consoleError).toHaveBeenCalledWith(expect.stringContaining("failed closed"), expect.anything())
26352647
consoleError.mockRestore()
@@ -2644,6 +2656,7 @@ describe("Task persistence", () => {
26442656
.fn()
26452657
.mockRejectedValue(new LifecycleTransitionError("parent linkage changed"))
26462658
mockProvider.taskHistoryStore.readValidatedDiskStatus = vi.fn().mockResolvedValue("active")
2659+
prepareLands()
26472660
runGateInsideRestore()
26482661
const consoleError = vi.spyOn(console, "error").mockImplementation(() => {})
26492662

@@ -2670,14 +2683,15 @@ describe("Task persistence", () => {
26702683
mockProvider.resumeInterruptedTask = vi
26712684
.fn()
26722685
.mockRejectedValue(new LifecycleTransitionError("parent linkage changed"))
2686+
prepareLands()
26732687
mockRestoreApiMessagesIfUnchanged.mockResolvedValueOnce({ outcome: "foreign_write" })
26742688
const consoleWarn = vi.spyOn(console, "warn").mockImplementation(() => {})
26752689

26762690
const task = makeResumeTask()
26772691

26782692
await expect(task["resumeTaskFromHistory"]()).rejects.toThrow("parent linkage changed")
26792693

2680-
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(1)
2694+
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(2)
26812695
// The newer disk content is never clobbered, and the original stays
26822696
// recoverable in the copy.
26832697
expect(mockRemoveResumeHistoryRecoveryCopy).not.toHaveBeenCalled()
@@ -2699,7 +2713,8 @@ describe("Task persistence", () => {
26992713

27002714
expect(mockProvider.resumeInterruptedTask).toHaveBeenCalledTimes(1)
27012715
expect(loop).toHaveBeenCalledTimes(1)
2702-
expect(mockRestoreApiMessagesIfUnchanged).not.toHaveBeenCalled()
2716+
// Only the conditional preparation write ran. No restore ran.
2717+
expect(mockRestoreApiMessagesIfUnchanged).toHaveBeenCalledTimes(1)
27032718
expect(mockRemoveResumeHistoryRecoveryCopy).toHaveBeenCalledTimes(1)
27042719
// The copy is removed only after the activation commit landed.
27052720
expect(mockRemoveResumeHistoryRecoveryCopy.mock.invocationCallOrder[0]).toBeLessThan(

‎src/core/task/__tests__/Task.resumeApiHistoryPersistence.spec.ts‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ vi.mock("../../condense", () => ({
127127
type TaskTestAccess = {
128128
initiateTaskLoop: (userContent: unknown[]) => Promise<void>
129129
retrySaveApiConversationHistory: () => Promise<boolean>
130+
prepareInterruptedResumeHistory: (original: ApiMessage[], prepared: ApiMessage[]) => Promise<boolean>
130131
}
131132

132133
function taskAccess(task: Task): TaskTestAccess {
@@ -453,9 +454,9 @@ describe("Task resume api-history persistence on real files", () => {
453454
const { task, loop, resumeActivation } = prepareResumedTask(
454455
makeChildRecord({ parentTaskId: undefined, rootTaskId: undefined }),
455456
)
456-
const realOverwrite = task.overwriteApiConversationHistory.bind(task)
457-
vi.spyOn(task, "overwriteApiConversationHistory").mockImplementation(async (history, persist) => {
458-
const result = await realOverwrite(history, persist)
457+
const realPrepare = taskAccess(task).prepareInterruptedResumeHistory.bind(task)
458+
vi.spyOn(taskAccess(task), "prepareInterruptedResumeHistory").mockImplementation(async (original, prepared) => {
459+
const result = await realPrepare(original, prepared)
459460
await task.abortTask(true)
460461
return result
461462
})
@@ -471,6 +472,81 @@ describe("Task resume api-history persistence on real files", () => {
471472
expect(await findRecoveryCopyName(taskDir)).toBeUndefined()
472473
})
473474

475+
describe.each([
476+
["standalone", true],
477+
["linked child", false],
478+
])("two accepted resumes of one interrupted %s record", (_name, isStandalone) => {
479+
const standalone = () =>
480+
isStandalone ? makeChildRecord({ parentTaskId: undefined, rootTaskId: undefined }) : makeChildRecord()
481+
482+
async function raceTwoResumes() {
483+
if (!isStandalone) {
484+
await provider.taskHistoryStore.upsert({
485+
id: parentId,
486+
number: 2,
487+
ts: 1,
488+
task: "Parent task",
489+
tokensIn: 0,
490+
tokensOut: 0,
491+
totalCost: 0,
492+
status: "delegated",
493+
awaitingChildId: childId,
494+
delegatedToId: childId,
495+
childIds: [childId],
496+
} as HistoryItem)
497+
}
498+
await provider.taskHistoryStore.upsert(standalone())
499+
await seedTaskFiles(storagePath, childId, authoritativeApiHistory)
500+
const first = prepareResumedTask(standalone())
501+
const second = prepareResumedTask(standalone())
502+
// Hold both attempts at the resume prompt, so both read the same
503+
// original history before either one prepares.
504+
let release!: () => void
505+
const gate = new Promise<void>((resolve) => (release = resolve))
506+
let waiting = 0
507+
for (const { task } of [first, second]) {
508+
vi.spyOn(task, "ask").mockImplementation(async () => {
509+
if (++waiting === 2) {
510+
release()
511+
}
512+
await gate
513+
return { response: "yesButtonClicked" }
514+
})
515+
}
516+
const results = await Promise.allSettled([
517+
first.task["resumeTaskFromHistory"](),
518+
second.task["resumeTaskFromHistory"](),
519+
])
520+
return { first, second, results }
521+
}
522+
523+
it("lets only one attempt prepare and activate", async () => {
524+
const { first, second, results } = await raceTwoResumes()
525+
526+
expect(results.filter((r) => r.status === "fulfilled")).toHaveLength(1)
527+
const rejected = results.find((r) => r.status === "rejected") as PromiseRejectedResult
528+
expect(String(rejected.reason)).toContain("Failed to persist the prepared API conversation history")
529+
expect(first.loop.mock.calls.length + second.loop.mock.calls.length).toBe(1)
530+
// Both tasks share the one provider spy.
531+
expect(first.resumeActivation).toHaveBeenCalledTimes(1)
532+
})
533+
534+
it("leaves the winner's prepared history on disk and never the loser's", async () => {
535+
const { first, second, results } = await raceTwoResumes()
536+
537+
const winner = results[0].status === "fulfilled" ? first : second
538+
const loser = winner === first ? second : first
539+
expect(await readPersistedApiHistory(storagePath, childId)).toEqual(preparedApiHistory)
540+
expect(winner.task.apiConversationHistory.map(({ role }) => role)).toEqual(
541+
preparedApiHistory.map(({ role }) => role),
542+
)
543+
// The loser wrote nothing and holds the original history in memory.
544+
expect(loser.task.apiConversationHistory.map(({ role }) => role)).toEqual(
545+
authoritativeApiHistory.map(({ role }) => role),
546+
)
547+
})
548+
})
549+
474550
it("reports a partial failure and proves the original bytes in the recovery copy when the restore write fails", async () => {
475551
await provider.taskHistoryStore.upsert({
476552
id: parentId,

0 commit comments

Comments
 (0)