Skip to content

Commit c3639c5

Browse files
committed
fix: guard interrupted-child resume and preserve history backups
1 parent b8f22ba commit c3639c5

14 files changed

Lines changed: 870 additions & 169 deletions

‎docs/architecture/task-lifecycle-model.md‎

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -41,25 +41,29 @@ TLA+/PlusCal or Quint with TLC becomes a better fit when the lifecycle needs tem
4141

4242
## Production mapping
4343

44-
| Model concept | Production concept |
45-
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
46-
| Task record and status | `HistoryItem` persisted by `TaskHistoryStore` |
47-
| `delegate(parent, child)` | `ClineProvider.delegateParentAndOpenChild` |
48-
| `interrupt(child)` | cancellation or eviction through `markDelegatedChildInterrupted` |
49-
| `resume(child)` | accepted `resume_task` prompt through `TaskHistoryStore.resumeInterruptedTask` |
50-
| `complete(child)` | `ClineProvider.reopenParentFromDelegation` |
51-
| `abandon(child)` | `ClineProvider.abandonSubtask` |
52-
| Pending-action settlement | `TaskHistoryStore.clearPendingActionIfMatching` compare-and-clear in the rejected-delegation settlement path (#1714) |
53-
| Atomic event step | `atomicReadAndUpdate`, `atomicUpdatePair`, and per-parent delegation transition lock |
54-
| Event interleaving | Competing completion, cancellation, abandonment, and new delegation calls |
55-
56-
The model has three fixed task slots, enough to cover competing siblings and a nested parent-child-grandchild chain. It explores every reachable interleaving through depth 13, deduplicating canonical states. Representative checks also exercise rejected operations that do not create a new state: a second concurrent delegation while the first child is active, stale completion after re-delegation, late completion after abandonment, completion after interruption, and nested completion. Named semantic landmarks require the graph to retain interrupted-child re-delegation, nested delegation, and an explicitly resumed interrupted child delegating to a grandchild even when the raw state total changes. The resume action calls the production reducer and carries a model-only provenance bit so the explorer can distinguish that path from a child that was never interrupted; generic persisted transitions still reject `interrupted → active`.
44+
| Model concept | Production concept |
45+
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
46+
| Task record and status | `HistoryItem` persisted by `TaskHistoryStore` |
47+
| `delegate(parent, child)` | `ClineProvider.delegateParentAndOpenChild` |
48+
| `interrupt(child)` | cancellation or eviction through `markDelegatedChildInterrupted` |
49+
| `resume(child)` | accepted `resume_task` prompt through `TaskHistoryStore.resumeInterruptedTask` |
50+
| `complete(child)` | `ClineProvider.reopenParentFromDelegation` |
51+
| `abandon(child)` | `ClineProvider.abandonSubtask` |
52+
| Pending-action settlement | `TaskHistoryStore.clearPendingActionIfMatching` compare-and-clear in the rejected-delegation settlement path (#1714) |
53+
| Atomic event step | `atomicReadAndUpdate`, `atomicUpdatePair`, `atomicUpdateParentDelegation`, the parent-file-locked resume, and the per-parent delegation transition lock |
54+
| Event interleaving | Competing completion, cancellation, abandonment, resume, and new delegation calls |
55+
56+
The model has three fixed task slots, enough to cover competing siblings and a nested parent-child-grandchild chain. It explores every reachable interleaving through depth 13, deduplicating canonical states. Representative checks also exercise rejected operations that do not create a new state: a second concurrent delegation while the first child is active, stale completion after re-delegation, late completion after abandonment, completion after interruption, and nested completion. Named semantic landmarks require the graph to retain interrupted-child re-delegation, nested delegation, and an explicitly resumed interrupted child delegating to a grandchild even when the raw state total changes. The resume action calls the production `resumeInterruptedChild` composition, which validates the parent's delegation pointer before applying the child reducer, and carries a model-only provenance bit so the explorer can distinguish that path from a child that was never interrupted. Generic persisted transitions still reject `interrupted → active`. A `resume-stale` rejected action covers the competing-host class: after the parent re-delegates elsewhere, the parent-locked ownership check rejects the resume and every record stays unchanged.
5757

5858
Each task slot can also hold one of two pending `create_subtask` actions. A `stage` action mirrors `setPendingTaskAction` overwrite semantics, delegation clears the action its request carried, completion clears the child's action only when its event carries the matching action ID, and a `settle-rejected` action models the settlement that follows an authoritative delegation rejection (#1714). Production settles through the typed `LifecycleTransitionError` from the shared guards: the provider calls the disk-authoritative `TaskHistoryStore.clearPendingActionIfMatching` compare-and-clear under the per-file lock, then propagates the original rejection. Six named witnesses must remain reachable: settlement from an interrupted record after rejection, settlement through a successful active delegation, unrelated-action preservation during completion, stale-action protection where a settlement targeting one action ID leaves a replacement action intact, matching-ID completion clearing, and replacement-ID completion preservation. A mismatched pending-action request keeps its production behavior: the atomic update throws before any transition, and no settlement runs.
5959

6060
Production completion also accepts a recovery-compatible `active` parent that still awaits the returning child, then clears the stale pointers. Normal model transitions never create that intermediate state, so it is covered by a focused reducer test rather than admitted as a generally valid reachable state.
6161

62-
Explicit resume compares the caller's expected parent with the authoritative child backlink under the child-file lock. Abandonment writes child detachment before parent release, so a stale approval is rejected even in that partial-write window. Representative reducer scenarios cover stale linked and standalone approvals; a two-store provider test covers abandonment after the parent precheck. Rehydrated interrupted tasks omit lineage from message metadata saves, preventing their stale construction snapshots from restoring severed links after rejection. This closes abandonment-before-resume, not every cross-host ordering: parent-only redelegation, reverse-order stale abandonment, and other live-task writers remain within the ownership/generation gaps below.
62+
Explicit resume validates both records under file locks. A linked child resume holds the parent file's advisory lock across the parent ownership check and the child activation, so a competing host cannot re-delegate the parent in between. The re-delegation write (`atomicUpdateParentDelegation`) re-reads the awaited child's on-disk record inside the parent-file lock and rejects the delegation unless that child is still interrupted. The two guards share one lock order, parent file before child file, so they close the competing resume and re-delegation interleavings in both directions. The child-file merge still compares the caller's expected parent with the authoritative child backlink. The resume rejects an abandonment that committed before it, even in its partial-write window.
63+
64+
Representative reducer scenarios cover stale linked and standalone approvals. Two-store tests cover abandonment after the parent precheck and both resume/re-delegation orderings. A declined resume returns before any store transition for both `resume_task` and `resume_completed_task`. This acceptance gate is deliberate and replaces the previous fall-through into the task loop. An accepted non-interrupted resume still runs without a store transition.
65+
66+
Rehydrated interrupted tasks omit lineage from message metadata saves, so their stale construction snapshots cannot restore severed links after rejection. Reverse-order stale abandonment and other live-task writers remain within the ownership/generation gaps below.
6367

6468
## Shared-store concurrency model
6569

@@ -145,6 +149,7 @@ The task delegation checker currently enforces:
145149
7. Active-child re-delegation, stale completion after ownership moves to another child, duplicate/late completion, and abandonment of a live child are rejected by the shared production guards.
146150
8. A rejected delegation settles only the exact matching pending `create_subtask` action. Settlement preserves status, lineage, and accounting, never clears a replacement or different-kind action, and never mutates a completed record. The settlement compare-and-clear reads the persisted record under the per-file lock, so it never clears from a stale host cache.
147151
9. A completion clears the completing child's pending action only when the completion event carries the exact matching action ID. A completion with no action ID or a different ID preserves the pending action.
152+
10. A linked interrupted child resumes only while the on-disk parent still awaits it, checked under the parent-file lock, and a re-delegation commits only while the awaited child's on-disk record is still interrupted, checked inside the parent-file lock. The rejected `resume-stale` action after a competing re-delegation leaves every record unchanged.
148153

149154
The completion persistence checker additionally enforces:
150155

‎scripts/check-task-lifecycle.ts‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ import {
77
completeDelegatedChild,
88
delegateTaskToChild,
99
interruptDelegatedChild,
10+
LifecycleTransitionError,
11+
resumeInterruptedChild,
1012
resumeInterruptedTask,
1113
settleRejectedCreateSubtaskAction,
1214
} from "../src/core/task-persistence/taskLifecycle"
@@ -40,7 +42,16 @@ interface WitnessContext {
4042
const MAX_DEPTH = 13
4143
const MAX_STATES = 10_000
4244
const actionIds = ["action-1", "action-2"] as const
43-
const expectedActions = ["delegate", "interrupt", "resume", "complete", "abandon", "stage", "settle-rejected"] as const
45+
const expectedActions = [
46+
"delegate",
47+
"interrupt",
48+
"resume",
49+
"resume-stale",
50+
"complete",
51+
"abandon",
52+
"stage",
53+
"settle-rejected",
54+
] as const
4455
const semanticLandmarks = {
4556
"interrupted-child-redelegation": (state: ModelState) =>
4657
state.parent?.status === "delegated" &&
@@ -193,9 +204,22 @@ function transitions(state: ModelState): Transition[] {
193204
(!parent.parentTaskId ||
194205
(lineageParent?.status === "delegated" && lineageParent.awaitingChildId === parent.id))
195206
if (resumeValid) {
207+
const resumed =
208+
lineageParent !== undefined
209+
? resumeInterruptedChild(lineageParent, parent)
210+
: resumeInterruptedTask(parent, undefined)
196211
result.push({
197212
name: `resume(${parentId})`,
198-
next: replace(state, { ...resumeInterruptedTask(parent, parent.parentTaskId), modelWasResumed: true }),
213+
next: replace(state, { ...resumed, modelWasResumed: true }),
214+
})
215+
} else if (parent.status === "interrupted" && parent.parentTaskId && lineageParent !== undefined) {
216+
// Representative rejected event: the parent re-delegated elsewhere,
217+
// so the store's parent-locked ownership check must reject the
218+
// resume and leave every record unchanged.
219+
assert.throws(() => resumeInterruptedChild(lineageParent, parent), LifecycleTransitionError)
220+
result.push({
221+
name: `resume-stale(${parentId})`,
222+
next: state,
199223
})
200224
}
201225
}
@@ -444,6 +468,10 @@ function runRepresentativeScenarios(): void {
444468
assert.equal(resumedNested.awaitingChildId, "child-b")
445469
const redelegated = delegateTaskToChild(delegated, "child-b", interruptedA.status)
446470
assert.throws(() => completeDelegatedChild(redelegated, interruptedA, "stale"), /not delegated to child/)
471+
// The parent-locked resume rejects a child whose parent re-delegated, and
472+
// accepts the same child while the parent still awaits it.
473+
assert.throws(() => resumeInterruptedChild(redelegated, interruptedA), /no longer awaits/)
474+
assert.equal(resumeInterruptedChild(delegated, interruptedA).status, "active")
447475

448476
const abandoned = abandonDelegatedChild(delegated, interruptedA)
449477
assert.throws(() => completeDelegatedChild(abandoned.parent, abandoned.child, "late"), /not delegated to child/)

0 commit comments

Comments
 (0)