Skip to content

Commit 2036123

Browse files
committed
fix(archive): wait for cleanup before WebSocket creation
🤖 Co-authored by GPT-6 in Codex via T3 Code
1 parent b997537 commit 2036123

3 files changed

Lines changed: 31 additions & 26 deletions

File tree

‎BRANCH_DETAILS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Archived conversations use cold storage instead of retaining full hot projection
88
- Cold restore preserves binary SQL values, rejects unsafe attachment entry names with a retryable failure, and atomically replaces attachment files before marking SQL rows restored. A malformed attachment leaves the cold bundle authoritative until repaired and retried. Restore pages chunk reads to bound memory, rejects unknown tables/chunk kinds, and intersects archived row columns with the current schema so older bundles remain recoverable after compatible migrations.
99
- Permanent thread deletion checks eligibility before writer quiescence under the storage tree lock. Obsolete jobs preserve live resources, remove only mismatched stale cold bundles before clearing their queue marker, and do nothing when no deletion marker remains. Permanent thread deletion removes the shell, event stream, command receipts, every thread-owned projection/runtime/checkpoint row, attachments, terminal history, provider logs and rotations, and any cold-archive manifest/chunks. Exact attachment ownership metadata remains available until external attachment and provider-log cleanup succeeds, after which the SQL and cold-chunk rows are removed; cross-thread plan references are cleared rather than leaving dangling ids, and the durable cleanup-queue entry is retained until filesystem cleanup and free-page reclamation succeed so interrupted deletes retry safely. Startup recovery also rediscovers soft-deleted shells that have not reached the cleanup queue, covering commits made while the lifecycle reactor was offline.
1010
- Project removal in `apps/web/src/components/settings/ProjectSettingsPanel.tsx` applies the archived-history warning and command options from `apps/web/src/components/settings/ProjectSettingsPanel.logic.ts` independently to each selected project member. Members with known live threads use forced deletion; when a member has no live shell threads, it requests archived-thread deletion without force because archived shells are excluded from normal navigation state. The server removes archived cold bundles but still rejects any unseen live thread. Project removal emits the same per-thread deletion events for archived shells, so already-cold threads pass through the durable lifecycle worker and lose their hot shell, archive manifest, and compressed chunks before cleanup completes. `apps/web/src/components/settings/ProjectSettingsPanel.logic.test.ts` covers standalone and grouped warning text and the per-member thread partitioning that selects each member's command options; `apps/server/src/orchestration/decider.delete.test.ts` covers the archived-only live-thread precondition; and `apps/server/src/orchestration/Layers/ThreadDeletionReactor.test.ts` covers the forced-project decider, lifecycle reactor, and cold-storage boundary together.
11-
- Archive/delete filesystem work runs through a durable background lifecycle queue. Thread creation is refused retryably while a soft-deleted shell or durable cleanup-queue entry remains, before any replacement event commits; the command id remains retryable after cleanup. The existing creation and first-prompt bootstrap acknowledgement waits only for same-ID lifecycle work before starting runtime resources. Unrelated bulk archives do not hold that creation wait; callers that require all lifecycle work to finish can still drain the whole queue. Archive and restore lifecycle work is serialized per project tree, with reference-counted idle-lock eviction after the final user or waiter. Archive eligibility and provider/terminal/log-writer shutdown run under that same tree lock, so a queued archive job that becomes stale after unarchive cannot stop the newly active session or delete its terminal history. Archive creation commits the complete compressed bundle, including its restore metadata, before the main-only destructive transaction deletes hot rows and marks the manifest `cleanup_pending`. This ordering avoids relying on atomic commits across attached databases when the hot database uses WAL. Destructive attachment/log cleanup must finish before the manifest becomes `cold`. Destructive archive transitions require terminal/log-writer shutdown before chunk creation and recheck the archived shell inside the hot-row deletion transaction; `archiveThread` normalizes typed quiesce failures to `ThreadColdStorageError` at the archive operation boundary. Provider shutdown also remains required while the projected session is `starting` or `running`; a missing provider binding is terminal for a settled or absent projected session, allowing legacy subagent shells that never owned a provider binding to move cold safely. Successful provider quiescence explicitly clears process-local background-liveness state, including the no-binding settled-shell path where no later `session.exited` event is guaranteed. The cold-storage boundary chunks child projections only after its caller-provided quiescence effect completes; pending-input coverage verifies that restored session and activity rows reflect stopped and resolved values while the lightweight hot shell retains a zero pending-input count. Failed lifecycle work requests one coalesced delayed rescan of the durable pending state instead of being dropped or introducing a permanent polling loop, enqueue interruptions release their deduplication reservation, and restart recovery preserves the same retry boundary. Filesystem failures other than a genuinely missing directory also keep retry state. Incomplete cleanup, including a `cleanup_pending` manifest whose archived shell has already been removed, and archived shells missing a manifest resume after restart without rebuilding an already durable bundle.
11+
- Archive/delete filesystem work runs through a durable background lifecycle queue. Thread creation is refused retryably while a soft-deleted shell or durable cleanup-queue entry remains, before any replacement event commits; the command id remains retryable after cleanup. WebSocket creation and first-prompt bootstrap wait for in-flight same-ID lifecycle work before dispatching creation, preserving draft retries without blocking the engine worker. A completed wait does not bypass the durable creation check when failed cleanup still has a marker. Unrelated bulk archives do not hold that creation wait; callers that require all lifecycle work to finish can still drain the whole queue. Archive and restore lifecycle work is serialized per project tree, with reference-counted idle-lock eviction after the final user or waiter. Archive eligibility and provider/terminal/log-writer shutdown run under that same tree lock, so a queued archive job that becomes stale after unarchive cannot stop the newly active session or delete its terminal history. Archive creation commits the complete compressed bundle, including its restore metadata, before the main-only destructive transaction deletes hot rows and marks the manifest `cleanup_pending`. This ordering avoids relying on atomic commits across attached databases when the hot database uses WAL. Destructive attachment/log cleanup must finish before the manifest becomes `cold`. Destructive archive transitions require terminal/log-writer shutdown before chunk creation and recheck the archived shell inside the hot-row deletion transaction; `archiveThread` normalizes typed quiesce failures to `ThreadColdStorageError` at the archive operation boundary. Provider shutdown also remains required while the projected session is `starting` or `running`; a missing provider binding is terminal for a settled or absent projected session, allowing legacy subagent shells that never owned a provider binding to move cold safely. Successful provider quiescence explicitly clears process-local background-liveness state, including the no-binding settled-shell path where no later `session.exited` event is guaranteed. The cold-storage boundary chunks child projections only after its caller-provided quiescence effect completes; pending-input coverage verifies that restored session and activity rows reflect stopped and resolved values while the lightweight hot shell retains a zero pending-input count. Failed lifecycle work requests one coalesced delayed rescan of the durable pending state instead of being dropped or introducing a permanent polling loop, enqueue interruptions release their deduplication reservation, and restart recovery preserves the same retry boundary. Filesystem failures other than a genuinely missing directory also keep retry state. Incomplete cleanup, including a `cleanup_pending` manifest whose archived shell has already been removed, and archived shells missing a manifest resume after restart without rebuilding an already durable bundle.
1212
- For unarchive commands targeting an existing archived shell, `apps/server/src/orchestration/Layers/OrchestrationEngine.ts` restores `cold` or `cleanup_pending` bundles before domain dispatch and treats an existing `restored` manifest as an in-flight or post-commit reservation without replaying its stale chunks over live rows or attachments. Active or absent targets reach the decider without claiming restore ownership, so domain rejections retain their rejected receipt and remain rejected on command-id retry. A transient restore failure re-archives any tree members restored before the failure and returns an error without writing a rejected command receipt, so the same command id can retry after storage recovers. A rejected or failed command re-archives the restored rows and files, while a successful command finalizes the cold bundle only when that request actually performed a restore; this prevents unrelated or already-hot unarchive commands from deleting archive data. The transaction and the in-memory restore-ownership handoff form one uninterruptible boundary, so cancellation cannot surface after SQL commits but before ownership transfers and roll back accepted data. Once SQL restoration commits, later publication or metrics failures cannot re-archive the restored data. Retrying an accepted unarchive receipt retries idempotent bundle finalization for that same thread. Unarchive events also queue lifecycle cleanup, and startup discovery includes restored manifests on active shells. That cleanup removes only the eligible thread's stale bundle before any provider or terminal shutdown, preserves process-owned restore reservations, and retries failures through the existing coalesced rescan. Finalization and archive-abort cleanup commit cold-row deletions before removing the main manifest, so a partial cleanup retains its discovery marker. A finalization failure does not change an already-committed unarchive result.
1313
- Unarchive reserves still-hot archived rows with a `restored` manifest before releasing the archive-tree lock. This prevents queued lifecycle work from moving those rows cold between the restore check and the unarchive command commit; command failure archives the reserved rows, while success removes the reservation. Process-local restore ownership distinguishes those live reservations from `restored` manifests abandoned by a crashed process, which startup lifecycle discovery moves cold again while the shell remains archived. If storage cannot restore or reserve the archived conversation, the command is rejected before an active-shell event can commit.
1414
- A `restored` manifest protects only the archive epoch that an unarchive command reserved. If post-commit bundle finalization fails and the user later archives the thread again, the newer shell timestamp lets lifecycle work replace the stale manifest and move the conversation cold.

‎apps/server/src/server.test.ts‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -12593,15 +12593,14 @@ it.layer(NodeServices.layer)("server router seam", (it) => {
1259312593
}).pipe(Effect.provide(NodeHttpServer.layerTest)),
1259412594
);
1259512595

12596-
it.effect("drains deletion cleanup through the re-created thread event", () =>
12596+
it.effect("waits for same-id deletion before direct and bootstrap creation dispatch", () =>
1259712597
Effect.gen(function* () {
1259812598
// A draft retry reuses the thread id its failed bootstrap deleted. The
1259912599
// deletion reactor stops sessions and closes terminals by that id, so
12600-
// both thread.create paths use the created event as a fence, then drain
12601-
// cleanup before handing the new incarnation to resource-owning work.
12600+
// both paths must wait before a replacement can publish its creation.
1260212601
const trace: Array<string> = [];
12603-
const drainRequested = yield* Deferred.make<void>();
12604-
const cleanupDone = yield* Deferred.make<void>();
12602+
let drainRequested = yield* Deferred.make<void>();
12603+
let cleanupDone = yield* Deferred.make<void>();
1260512604
yield* buildAppUnderTest({
1260612605
layers: {
1260712606
threadDeletionReactor: {
@@ -12614,6 +12613,7 @@ it.layer(NodeServices.layer)("server router seam", (it) => {
1261412613
}),
1261512614
},
1261612615
orchestrationEngine: {
12616+
latestSequence: Effect.succeed(41),
1261712617
dispatch: (command) =>
1261812618
Effect.sync(() => {
1261912619
trace.push(command.type);
@@ -12647,17 +12647,18 @@ it.layer(NodeServices.layer)("server router seam", (it) => {
1264712647
}),
1264812648
);
1264912649
yield* Deferred.await(drainRequested);
12650-
assert.deepEqual(trace, ["thread.create", "drain:1"]);
12650+
assert.deepEqual(trace, ["drain:41"]);
1265112651
yield* Deferred.succeed(cleanupDone, undefined);
1265212652
yield* Fiber.join(directCreate);
1265312653
}),
1265412654
),
1265512655
);
12656-
assert.deepEqual(trace, ["thread.create", "drain:1"]);
12656+
assert.deepEqual(trace, ["drain:41", "thread.create"]);
1265712657

12658-
// Cleanup is already released; the bootstrap path must still drain
12659-
// between creating the thread and starting its turn.
12658+
// A second connection's bootstrap retry waits on its own unfinished cleanup.
1266012659
trace.length = 0;
12660+
drainRequested = yield* Deferred.make<void>();
12661+
cleanupDone = yield* Deferred.make<void>();
1266112662
yield* Effect.scoped(
1266212663
withWsRpcClient(wsUrl, (client) =>
1266312664
Effect.gen(function* () {
@@ -12691,13 +12692,16 @@ it.layer(NodeServices.layer)("server router seam", (it) => {
1269112692
createdAt,
1269212693
}),
1269312694
);
12695+
yield* Deferred.await(drainRequested);
12696+
assert.deepEqual(trace, ["drain:41"]);
12697+
yield* Deferred.succeed(cleanupDone, undefined);
1269412698
yield* Fiber.join(bootstrapCreate);
1269512699
}),
1269612700
),
1269712701
);
1269812702
assert.deepEqual(trace, [
12703+
"drain:41",
1269912704
"thread.create",
12700-
"drain:1",
1270112705
"thread.message.user.append",
1270212706
"thread.turn.start",
1270312707
]);

‎apps/server/src/ws.ts‎

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1420,7 +1420,11 @@ const makeWsRpcLayer = (
14201420
}
14211421

14221422
if (bootstrap?.createThread) {
1423-
const created = yield* dispatchFromClient({
1423+
yield* threadDeletionReactor.drainThrough(
1424+
yield* orchestrationEngine.latestSequence,
1425+
command.threadId,
1426+
);
1427+
yield* dispatchFromClient({
14241428
type: "thread.create",
14251429
commandId: yield* serverCommandId("bootstrap-thread-create"),
14261430
threadId: command.threadId,
@@ -1433,12 +1437,7 @@ const makeWsRpcLayer = (
14331437
worktreePath: bootstrap.createThread.worktreePath,
14341438
createdAt: bootstrap.createThread.createdAt,
14351439
});
1436-
// The successful create is a fence in the engine command queue:
1437-
// every delete for the prior incarnation committed before it.
1438-
// Wait for that thread's cleanup before setup or turn start can
1439-
// own resources, without waiting for unrelated bulk archives.
14401440
createdThread = true;
1441-
yield* threadDeletionReactor.drainThrough(created.sequence, command.threadId);
14421441
// Persist the send now rather than with the turn: the thread is
14431442
// real from here on, so any client (or a reload) sees the message
14441443
// while the worktree is still being prepared. The turn start
@@ -1779,15 +1778,17 @@ const makeWsRpcLayer = (
17791778
const dispatchEffect =
17801779
normalizedCommand.type === "thread.turn.start" && normalizedCommand.bootstrap
17811780
? dispatchBootstrapTurnStart(normalizedCommand)
1782-
: dispatchFromClient(normalizedCommand).pipe(
1783-
Effect.tap(({ sequence }) =>
1784-
// Returning from thread.create is the handoff point at which
1785-
// clients may start resources for the new incarnation. Use
1786-
// its event sequence as the exact deletion-cleanup fence.
1787-
normalizedCommand.type === "thread.create"
1788-
? threadDeletionReactor.drainThrough(sequence, normalizedCommand.threadId)
1789-
: Effect.void,
1790-
),
1781+
: Effect.gen(function* () {
1782+
if (normalizedCommand.type === "thread.create") {
1783+
// Wait outside the command worker; durable cleanup failures
1784+
// still reach the engine's retryable creation gate.
1785+
yield* threadDeletionReactor.drainThrough(
1786+
yield* orchestrationEngine.latestSequence,
1787+
normalizedCommand.threadId,
1788+
);
1789+
}
1790+
return yield* dispatchFromClient(normalizedCommand);
1791+
}).pipe(
17911792
Effect.mapError((cause) =>
17921793
toDispatchCommandError(cause, "Failed to dispatch orchestration command"),
17931794
),

0 commit comments

Comments
 (0)