Skip to content

Commit 095fa0f

Browse files
committed
test: retain all drain failures and prove cleanup before disposal
1 parent 9351746 commit 095fa0f

2 files changed

Lines changed: 98 additions & 7 deletions

File tree

‎devlog/_plan/261004_fixture_isolation_stabilization/010_fixtures.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ In tests/update/update-restart-lease.test.ts:
99
1. MODIFY node:os import to namespace form and add Bun test spyOn. Before each case, capture environment and arrange a restorable homedir spy. sandbox() sets the spy to its case home before calling serviceStatePaths(). Replace platform-specific child HOME/USERPROFILE selection with box.home on every platform. Preserve the original OCX_REAL_HOME protection and token exclusion. Assert every parent candidate and the selected authority are within box.root, using path-relative containment; retain actual child-authority equality before service-child behavior. Pin/restore CODEX_SQLITE_HOME to the case if inherited override could escape fixture scope.
1010
2. MODIFY child bookkeeping to track actual Bun subprocesses and their drain completion; keep fake Node ChildProcess only at the production injection seam. Do not cast Bun objects to Node types for cleanup. Preserve childLogs behavior and original assertions.
1111
3. REPLACE the lease holder's `await Bun.sleep(13_000); lease.release();` with explicit stdin EOF protocol. Child acquires the real lease, writes a case-owned ready marker, then waits on its piped stdin. Closing stdin requests release; child calls release, writes a distinct released acknowledgment and exits normally. Parent keeps stdin open throughout the existing fail-closed assertions. The synchronous runService hook polls the ready marker under its existing bound, verifies holder ownership, then returns. Production reacquire 10s and stale-grace 30s remain unchanged.
12-
4. MODIFY teardown: request cooperative release for holders, require acknowledgment/exit0/exact lock absence before root removal, and await output drains. Bound every wait with existing watchdog/budget helpers and clear timers. If cooperation fails, kill/reap only that tracked fixture child under a separate bound, settle/cancel its drains, preserve the graceful failure and restore env/spies in finally. Remove only a proven-owned root after its children are reaped; retain it with diagnostics if safe cleanup cannot be proven. Never unlink/backdate a lock to obtain green.
12+
4. MODIFY teardown: request cooperative release for holders, require acknowledgment/exit0/exact lock absence before root removal, and await output drains. Bound every wait with existing watchdog/budget helpers and clear timers. If cooperation fails, kill/reap only that tracked fixture child under a separate bound, settle/cancel its drains while collecting every observed rejection by drain index, exclude only the exact cleanup-owned cancellation object (not an error name), preserve all genuine and graceful failures and restore env/spies in finally. Remove an ordinary proven-owned root only after every child is reaped, drains settled and the full cleanup protocol succeeded; retain it with diagnostics on an unexpected cleanup failure. An explicit negative control may dispose its owned root after reaping/draining only after verifying and consuming exactly its deliberately expected failures. Never unlink/backdate a lock to obtain green.
1313
5. ADD behavior controls in the same file: two cases yield different contained authority paths; parent/child authority matches with no service command; successful EOF release removes the exact lock and leaves no live child before root cleanup; ignored EOF/missing acknowledgment reaches bounded fallback and is reported as failed graceful cleanup, not success. Preserve all seven existing behavioral cases and assertions. Original holder→two successors sequence is the regression oracle.
1414

1515
## Injection snapshot: main scope

‎tests/update/update-restart-lease.test.ts‎

Lines changed: 97 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ type FixtureChild = {
7373
reaped: boolean;
7474
drained: boolean;
7575
forced: boolean;
76+
cleanupProven: boolean;
7677
released?: string;
7778
cleanup?: Promise<void>;
7879
failureObserved?: boolean;
@@ -162,7 +163,7 @@ async function bounded<T>(work: Promise<T>, label: string, ms = isolationBudgetM
162163

163164
function trackChild(box: Sandbox, child: FixtureChild["process"]): FixtureChild {
164165
const entry: FixtureChild = {
165-
box, process: child, output: [], abort: new AbortController(), drains: [], reaped: false, drained: false, forced: false,
166+
box, process: child, output: [], abort: new AbortController(), drains: [], reaped: false, drained: false, forced: false, cleanupProven: false,
166167
};
167168
children.push(entry);
168169
for (const stream of [child.stdout, child.stderr]) {
@@ -277,22 +278,39 @@ function cleanupChild(child: FixtureChild, graceMs = isolationBudgetMs(5_000)):
277278
}
278279
} catch (fallbackError) { failures.push(fallbackError); }
279280
} finally {
281+
const outcomes: Array<PromiseSettledResult<unknown> | undefined> = child.drains.map(() => undefined);
282+
const settled = Promise.all(child.drains.map((drain, index) => drain.then(
283+
value => { outcomes[index] = { status: "fulfilled", value }; },
284+
reason => { outcomes[index] = { status: "rejected", reason }; },
285+
)));
286+
const ownedCancellation = new Error("fixture-owned drain cancellation");
280287
try {
281288
await bounded(Promise.all(child.drains), "fixture child drains timed out");
282289
child.drained = true;
283290
} catch (error) {
284-
failures.push(error);
285-
child.abort.abort();
291+
// A drain rejection is collected by index below; only the wait's own error is added here.
292+
if (!outcomes.some(result => result?.status === "rejected" && result.reason === error)) failures.push(error);
293+
child.abort.abort(ownedCancellation);
286294
try {
287-
await bounded(Promise.allSettled(child.drains), "fixture child drain cancellation timed out");
295+
await bounded(settled, "fixture child drain cancellation timed out");
288296
child.drained = true;
289297
} catch (drainError) { failures.push(drainError); }
290298
}
299+
// Preserve late and same-object rejections from every drain, even after a bounded wait fails.
300+
for (const result of outcomes) {
301+
if (result?.status === "rejected" && result.reason !== ownedCancellation) failures.push(result.reason);
302+
}
291303
}
292304
if (failures.length) throw new AggregateError(failures, `fixture cleanup failed: ${failures.map(String).join("; ")}`);
305+
child.cleanupProven = true;
293306
})();
294307
}
295308

309+
function canRemoveSandbox(box: Sandbox): boolean {
310+
return children.filter(child => child.box === box).every(child =>
311+
child.reaped && child.drained && (child.cleanupProven || child.failureObserved === true));
312+
}
313+
296314
function expectOnlyCleanupFailure(error: unknown, message: string): void {
297315
expect(error).toBeInstanceOf(AggregateError);
298316
const failures = (error as AggregateError).errors;
@@ -393,7 +411,7 @@ afterEach(async () => {
393411
catch (error) { if (!child.failureObserved) failures.push(error); }
394412
}
395413
for (const box of sandboxes) {
396-
if (children.some(child => child.box === box && (!child.reaped || !child.drained))) {
414+
if (!canRemoveSandbox(box)) {
397415
failures.push(new Error(`retaining fixture with unconfirmed child cleanup: ${box.root}`));
398416
} else {
399417
try { removeTreeWithRetry(box.root); }
@@ -723,6 +741,8 @@ describe("lease fixture containment and cleanup controls", () => {
723741
expect(existsSync(box.lockDir)).toBe(true);
724742
await cleanupChild(holder);
725743
expect(holder.process.exitCode).toBe(0);
744+
expect(holder.cleanupProven).toBe(true);
745+
expect(canRemoveSandbox(box)).toBe(true);
726746
expect(holder.reaped && holder.drained).toBe(true);
727747
expect(readFileSync(holder.released!, "utf8")).toBe(String(holder.process.pid));
728748
expect(existsSync(box.lockDir)).toBe(false);
@@ -741,8 +761,76 @@ describe("lease fixture containment and cleanup controls", () => {
741761
"fixture child graceful exit timed out", "injected drain failure",
742762
]);
743763
expect(holder.reaped && holder.drained && holder.forced).toBe(true);
764+
expect(holder.cleanupProven).toBe(false);
765+
expect(canRemoveSandbox(box)).toBe(false);
744766
// This control consumes exactly its two deliberately injected failures, never arbitrary errors.
745767
holder.failureObserved = true;
768+
expect(canRemoveSandbox(box)).toBe(true);
769+
}, watchdogMs(20_000));
770+
771+
test("a delayed second drain rejection cannot hide behind the expected cleanup pair", async () => {
772+
const box = sandbox();
773+
const holder = spawnHolder(box, "ignore-eof");
774+
const late = new Promise<never>((_, reject) => {
775+
holder.abort.signal.addEventListener("abort", () => {
776+
void Promise.resolve().then(() => reject(new Error("unexpected delayed stdout failure")));
777+
}, { once: true });
778+
});
779+
holder.drains.push(Promise.reject(new Error("injected drain failure")), late);
780+
void Promise.allSettled(holder.drains);
781+
const error = await cleanupChild(holder, isolationBudgetMs(200)).catch(error => error);
782+
expect(error).toBeInstanceOf(AggregateError);
783+
const messages = error.errors.map((failure: Error) => failure.message);
784+
expect(() => expect(messages).toEqual([
785+
"fixture child graceful exit timed out", "injected drain failure",
786+
])).toThrow();
787+
expect(messages).toEqual([
788+
"fixture child graceful exit timed out", "injected drain failure", "unexpected delayed stdout failure",
789+
]);
790+
expect(holder.reaped && holder.drained && holder.forced).toBe(true);
791+
expect(canRemoveSandbox(box)).toBe(false);
792+
holder.failureObserved = true;
793+
}, watchdogMs(20_000));
794+
795+
test("drain cleanup excludes only its own cancellation reason", async () => {
796+
const box = sandbox();
797+
const holder = spawnHolder(box);
798+
const cancellation = new Promise<never>((_, reject) => {
799+
holder.abort.signal.addEventListener("abort", () => reject(holder.abort.signal.reason), { once: true });
800+
});
801+
const foreignAbort = new Promise<never>((_, reject) => {
802+
holder.abort.signal.addEventListener("abort", () => reject(new DOMException("unrelated stream abort", "AbortError")), { once: true });
803+
});
804+
holder.drains.push(Promise.reject(new Error("first drain failure")), cancellation, foreignAbort);
805+
void Promise.allSettled(holder.drains);
806+
const error = await cleanupChild(holder).catch(error => error);
807+
expect(error).toBeInstanceOf(AggregateError);
808+
expect(error.errors.map((failure: Error) => failure.message)).toEqual(["first drain failure", "unrelated stream abort"]);
809+
expect(holder.reaped && holder.drained).toBe(true);
810+
expect(holder.forced).toBe(false);
811+
expect(canRemoveSandbox(box)).toBe(false);
812+
holder.failureObserved = true;
813+
}, watchdogMs(20_000));
814+
815+
test("a drain timeout retains the later independent rejection", async () => {
816+
const box = sandbox();
817+
const holder = spawnHolder(box);
818+
const independent = new Promise<never>((_, reject) => {
819+
holder.abort.signal.addEventListener("abort", () => reject(new Error("late failure after drain timeout")), { once: true });
820+
});
821+
const cancellation = new Promise<never>((_, reject) => {
822+
holder.abort.signal.addEventListener("abort", () => reject(holder.abort.signal.reason), { once: true });
823+
});
824+
holder.drains.push(independent, cancellation);
825+
void Promise.allSettled(holder.drains);
826+
const error = await cleanupChild(holder).catch(error => error);
827+
expect(error).toBeInstanceOf(AggregateError);
828+
expect(error.errors.map((failure: Error) => failure.message)).toEqual([
829+
"fixture child drains timed out", "late failure after drain timeout",
830+
]);
831+
expect(holder.reaped && holder.drained).toBe(true);
832+
expect(canRemoveSandbox(box)).toBe(false);
833+
holder.failureObserved = true;
746834
}, watchdogMs(20_000));
747835

748836
for (const behavior of ["ignore-eof", "missing-ack"] as const) {
@@ -758,8 +846,11 @@ describe("lease fixture containment and cleanup controls", () => {
758846
expect(existsSync(box.lockDir)).toBe(behavior === "ignore-eof");
759847
expect(holder.forced).toBe(behavior === "ignore-eof");
760848
if (behavior === "missing-ack") expect(holder.process.exitCode).toBe(0);
761-
// Only this negative control consumes its expected rejection; ordinary teardown propagates it.
849+
expect(holder.cleanupProven).toBe(false);
850+
expect(canRemoveSandbox(box)).toBe(false);
851+
// Only this negative control consumes its expected rejection; ordinary teardown retains failures.
762852
holder.failureObserved = true;
853+
expect(canRemoveSandbox(box)).toBe(true);
763854
}, watchdogMs(20_000));
764855
}
765856
});

0 commit comments

Comments
 (0)