Skip to content

Commit 3e235a9

Browse files
committed
fix(apple-runner): let a start queued behind a settled close start fresh (#3220)
Review round on the fence found the over-refusal: a caller that merely QUEUED behind a close is not that teardown's retry, and must not wake to a canceled start once the close has settled. The lock task now re-routes such a start to a fresh admission once the fence has left the device — waiter interest follows the redirect — while a start woken WHILE the fence governs the device stays refused, exactly the mid-close retry the fence exists for. The settle clears the fence inside the session lock so a woken start never sees a half-settled teardown. The starting request also counts as an interested waiter on surfaces that pass no caller signal, so a joiner's cancellation can no longer outvote the live owner and stop its build — the protection the removed #3193 owner sniff carried, now by mechanism.
1 parent d8644c0 commit 3e235a9

5 files changed

Lines changed: 201 additions & 11 deletions

File tree

‎packages/platform-apple/src/runner/__tests__/runner-session-close-prep-fence.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,10 +213,49 @@ test('close during a cold build leaves the start no way to spawn a replacement b
213213

214214
assert.ok(first instanceof Error, 'the start whose build was killed failed');
215215
assert.ok(isRequestCanceledError(retried), 'the retry is refused as a canceled start');
216+
assert.equal(
217+
(retried as { details?: { runnerStartRetirementReason?: unknown } }).details
218+
?.runnerStartRetirementReason,
219+
'device_teardown',
220+
'the refusal carries the typed retirement reason, not just a cancellation',
221+
);
216222
assert.equal(mockBuildForTesting.mock.calls.length, 1, 'no replacement build was ever spawned');
217223
assert.equal(runnerPrepProcessChildren(device.id).length, 0, 'the killed build left the ledger');
218224
});
219225

226+
/**
227+
* The reviewer's counter-case to the fence: a caller that merely QUEUED behind the close — it
228+
* never owned the killed build and is not that teardown's retry — must not inherit the fence
229+
* once the close has settled. It wakes, re-routes to a fresh admission, and starts on its own.
230+
*/
231+
test('a start queued behind a close starts fresh once that close has settled', async () => {
232+
const device = { ...IOS_ADMISSION_SIMULATOR, id: 'runner-close-queued-sim' };
233+
const firstStart = ensureRunnerSession(device, {}).catch((error: unknown) => error);
234+
await vi.waitFor(() => assert.equal(builds.length, 1));
235+
236+
const closing = releaseIosRunnerOnClose(device.id, { retain: false });
237+
// Close has killed the prep child (proven by the empty ledger) and now waits on the session
238+
// lock; the fence is in effect. The killed build frees that lock, and this caller is already
239+
// queued behind close's own stop task, capturing the fenced admission while it still governs.
240+
await vi.waitFor(() => assert.equal(runnerPrepProcessChildren(device.id).length, 0));
241+
builds[0]!.kill();
242+
const queuedStart = ensureRunnerSession(device, {}).catch((error: unknown) => error);
243+
mockBuildForTesting.mockImplementationOnce(async () => {
244+
await seedBuiltRunner();
245+
return { exitCode: 0, stdout: '', stderr: '' } satisfies ExecResult;
246+
});
247+
248+
await closing;
249+
await firstStart;
250+
const started = await queuedStart;
251+
assert.ok(!(started instanceof Error), 'the queued caller started fresh: ' + String(started));
252+
assert.equal(
253+
mockBuildForTesting.mock.calls.length,
254+
2,
255+
'close spawned no replacement; the queued caller built its own runner',
256+
);
257+
});
258+
220259
/**
221260
* The nearest negative: with no close in sight, the next start after a killed build is a real
222261
* start and does build. A fence that simply refused the device after any kill would pass the

‎packages/platform-apple/src/runner/__tests__/runner-start-admission.test.ts‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
addRunnerStartWaiter,
77
cancelRunnerStartWaiter,
88
clearRunnerStartAdmissionAfterTeardown,
9+
rerouteSettledRunnerStart,
910
retireRunnerStartAdmissionForDevice,
1011
runnerStartAdmissionForDevice,
1112
runnerStartAdmitsPreparation,
@@ -133,3 +134,60 @@ test('a teardown with no start on the device still fences it until it settles',
133134
clearRunnerStartAdmissionAfterTeardown('admission-nested-sim');
134135
assert.equal(runnerStartAdmissionForDevice('admission-nested-sim').admitted, true);
135136
});
137+
138+
/**
139+
* A start that only QUEUED behind a teardown is not that teardown's retry: once the fence has
140+
* settled off the device, the woken start re-routes to a fresh admission and runs, instead of
141+
* failing forever for a close it never took part in (#3220 review). While the fence still
142+
* governs the device, nothing re-routes — that is the mid-close retry the fence exists for.
143+
*/
144+
test('a start woken after a settled teardown re-routes; one woken under the fence does not', () => {
145+
const device = 'admission-reroute-sim';
146+
const captured = runnerStartAdmissionForDevice(device);
147+
const waiter = new AbortController();
148+
addRunnerStartWaiter(captured, waiter.signal);
149+
150+
retireRunnerStartAdmissionForDevice(device);
151+
// Mid-close: the fence IS the device's admission, and the queued start stays refused.
152+
assert.equal(rerouteSettledRunnerStart(device, captured), undefined);
153+
154+
clearRunnerStartAdmissionAfterTeardown(device);
155+
const fresh = rerouteSettledRunnerStart(device, captured);
156+
assert.ok(fresh);
157+
assert.notEqual(fresh, captured);
158+
assert.equal(fresh.admitted, true);
159+
assert.equal(runnerStartAdmitsPreparation(device, fresh), true);
160+
161+
// The pre-reroute waiter follows the redirect: its cancel counts against the live admission.
162+
assert.equal(cancelRunnerStartWaiter(captured, waiter.signal), true);
163+
assert.equal(
164+
fresh.admitted,
165+
false,
166+
'the cancel closed the admission the work actually runs under',
167+
);
168+
});
169+
170+
/**
171+
* A re-route never steps into a newer teardown: a second close that fenced the device again
172+
* leaves the woken start refused rather than handing it an admission that is itself closed.
173+
*/
174+
test('a woken start is not re-routed into a newer fence', () => {
175+
const device = 'admission-reroute-fenced-sim';
176+
const captured = runnerStartAdmissionForDevice(device);
177+
retireRunnerStartAdmissionForDevice(device);
178+
clearRunnerStartAdmissionAfterTeardown(device);
179+
retireRunnerStartAdmissionForDevice(device);
180+
181+
assert.equal(rerouteSettledRunnerStart(device, captured), undefined);
182+
});
183+
184+
/** A last-waiter cancellation abandons the work; a start holding it is never re-routed. */
185+
test('a last-waiter retirement never re-routes', () => {
186+
const device = 'admission-reroute-cancel-sim';
187+
const captured = runnerStartAdmissionForDevice(device);
188+
const waiter = new AbortController();
189+
addRunnerStartWaiter(captured, waiter.signal);
190+
assert.equal(cancelRunnerStartWaiter(captured, waiter.signal), true);
191+
192+
assert.equal(rerouteSettledRunnerStart(device, captured), undefined);
193+
});

‎packages/platform-apple/src/runner/runner-session.ts‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -106,11 +106,21 @@ export async function ensureRunnerSession(
106106
admission.runnerStartAdmissionForDevice(device.id),
107107
);
108108
const start = withRunnerSessionLock(device.id, async () => {
109-
const { openRunnerStartBudget } = await import('./runner-start-budget.ts');
110-
const startAdmission = await startAdmissionReady;
111-
// A teardown that closed this start's admission while the start queued behind another is
112-
// refused here, before it adopts, boots, or builds (#3220).
113-
const { assertRunnerStartAdmitsPreparation } = await loadRunnerStartAdmission();
109+
const { openRunnerStartBudget, reserveRunnerStartOwnerInterest } =
110+
await import('./runner-start-budget.ts');
111+
const capturedAdmission = await startAdmissionReady;
112+
// A start woken after the teardown that fenced it has finished settling is a fresh start,
113+
// not that teardown's retry: it is re-routed to a new admission, keeping its waiter
114+
// interest live. A start woken WHILE the fence still governs the device is refused here,
115+
// before it adopts, boots, or builds (#3220).
116+
const {
117+
rerouteSettledRunnerStart,
118+
liveRunnerStartAdmission,
119+
assertRunnerStartAdmitsPreparation,
120+
} = await loadRunnerStartAdmission();
121+
const startAdmission = liveRunnerStartAdmission(
122+
rerouteSettledRunnerStart(device.id, capturedAdmission) ?? capturedAdmission,
123+
);
114124
assertRunnerStartAdmitsPreparation(device.id, startAdmission);
115125
// One budget for the whole start, opened once the lock is held so a start queued behind
116126
// another does not spend its clock waiting: the reuse check's toolchain probes, adoption and
@@ -120,6 +130,9 @@ export async function ensureRunnerSession(
120130
// still there for the retry (#2894). A start that outlives its caller is still bounded: once
121131
// the budget is spent the same signal ends it, the lock is released and the device is usable.
122132
const budget = openRunnerStartBudget(options);
133+
// The starting request counts as interested even on the surface that passes no `signal`, so
134+
// a joiner's cancellation can never outvote the live owner and stop its build (#3220).
135+
const releaseOwnerInterest = reserveRunnerStartOwnerInterest(startAdmission, options);
123136
try {
124137
const existing = runnerSessions.get(device.id);
125138
if (existing) {
@@ -136,6 +149,7 @@ export async function ensureRunnerSession(
136149
throw budget.exhausted.aborted ? budget.exhausted.reason : error;
137150
} finally {
138151
budget.close();
152+
releaseOwnerInterest();
139153
}
140154
});
141155
const [{ raceRunnerStartAgainstCaller }, startAdmission] = await Promise.all([
@@ -726,8 +740,14 @@ export async function stopIosRunnerSession(deviceId: string): Promise<void> {
726740
await stopRunnerSessionInternal(deviceId, undefined, { leaseLockHeld: true });
727741
await cleanupOwnedIosRunnerLease(deviceId);
728742
});
743+
// The settle clears the fence INSIDE the lock: a start queued behind this stop wakes
744+
// only to a device whose teardown has fully settled and re-routes to a fresh start,
745+
// never to a woken-before-clear window that reads as a mid-close retry (#3220).
746+
clearRunnerStartAdmissionAfterTeardown(deviceId);
729747
});
730748
} finally {
749+
// The error path settles too: a stop that throws mid-teardown must not leave the device
750+
// fenced past its own failure.
731751
clearRunnerStartAdmissionAfterTeardown(deviceId);
732752
}
733753
}

‎packages/platform-apple/src/runner/runner-start-admission.ts‎

Lines changed: 50 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,8 @@ export type RunnerStartAdmission = Readonly<{
4343

4444
type StartAdmission = RunnerStartAdmission & {
4545
readonly interestedWaiters: Set<AbortSignal>;
46+
/** Set once a settled-teardown re-route moves this start's work to a fresh admission. */
47+
redirectedTo: StartAdmission | undefined;
4648
/** Closes admission; true only when this call was the one that closed it. */
4749
close(reason: RunnerStartRetirementReason): boolean;
4850
};
@@ -101,6 +103,47 @@ export function clearAllRunnerStartAdmissionsAfterTeardown(): void {
101103
}
102104
}
103105

106+
/**
107+
* Re-routes a start that only *queued* behind a teardown: its captured admission was closed by
108+
* that teardown, and the teardown has since settled — its fence is no longer the device's
109+
* admission, so the queued caller runs as a fresh start under a new admission rather than as
110+
* the retired start's retry the fence exists for. Returns `undefined` when the captured
111+
* admission still governs the device — the teardown has not settled and a start woken under it
112+
* is the mid-close retry — when the closure was a last-waiter cancellation, which abandons the
113+
* work rather than deferring it, or when the device's current admission is itself fenced by a
114+
* newer teardown.
115+
*
116+
* The re-route leaves a redirect on the captured admission, so a caller whose interest was
117+
* registered before the re-route is followed to the fresh admission when it cancels or settles
118+
* (#3220).
119+
*/
120+
export function rerouteSettledRunnerStart(
121+
deviceId: string,
122+
captured: RunnerStartAdmission,
123+
): RunnerStartAdmission | undefined {
124+
if (captured.admitted || captured.retired !== 'device_teardown') return undefined;
125+
if (runnerStartAdmissionsByDevice.get(deviceId) === captured) return undefined;
126+
const own = captured as StartAdmission;
127+
const current = runnerStartAdmissionForDevice(deviceId) as StartAdmission;
128+
if (!current.admitted) return undefined;
129+
for (const waiter of own.interestedWaiters) {
130+
if (!waiter.aborted) current.interestedWaiters.add(waiter);
131+
}
132+
own.interestedWaiters.clear();
133+
own.redirectedTo = current;
134+
return current;
135+
}
136+
137+
/**
138+
* The admission a waiter's cancellation must act on: a re-routed start's work runs under the
139+
* fresh admission, so interest recorded on the captured one is followed to where it counts.
140+
*/
141+
export function liveRunnerStartAdmission(admission: RunnerStartAdmission): RunnerStartAdmission {
142+
let current = admission as StartAdmission;
143+
while (current.redirectedTo) current = current.redirectedTo;
144+
return current;
145+
}
146+
104147
/**
105148
* The gate the preparation-spawn seam and the session-publish point share. It is read
106149
* immediately BEFORE the subprocess is created: a check taken only when registering an
@@ -153,7 +196,7 @@ export function addRunnerStartWaiter(
153196
signal: AbortSignal,
154197
): 'interested' | 'already-spent' {
155198
if (signal.aborted) return 'already-spent';
156-
interestedWaitersOf(admission).add(signal);
199+
interestedWaitersOf(liveRunnerStartAdmission(admission)).add(signal);
157200
return 'interested';
158201
}
159202

@@ -174,14 +217,14 @@ export function cancelRunnerStartWaiter(
174217
admission: RunnerStartAdmission,
175218
signal: AbortSignal,
176219
): boolean {
177-
const own = admission as StartAdmission;
220+
const own = liveRunnerStartAdmission(admission) as StartAdmission;
178221
own.interestedWaiters.delete(signal);
179-
if (!admission.admitted || own.interestedWaiters.size > 0) return false;
222+
if (!own.admitted || own.interestedWaiters.size > 0) return false;
180223
// Nobody wants this start's outcome anymore, so the device is free to admit the next caller as
181224
// a fresh start. The fenced start keeps its own closed admission, and a teardown fence is
182225
// never lifted by a waiter: only the settled teardown clears it.
183-
if (runnerStartAdmissionsByDevice.get(admission.deviceId) === own) {
184-
runnerStartAdmissionsByDevice.delete(admission.deviceId);
226+
if (runnerStartAdmissionsByDevice.get(own.deviceId) === own) {
227+
runnerStartAdmissionsByDevice.delete(own.deviceId);
185228
}
186229
return own.close('last_waiter_canceled');
187230
}
@@ -196,7 +239,7 @@ export function releaseRunnerStartWaiter(
196239
admission: RunnerStartAdmission,
197240
signal: AbortSignal,
198241
): void {
199-
interestedWaitersOf(admission).delete(signal);
242+
interestedWaitersOf(liveRunnerStartAdmission(admission)).delete(signal);
200243
}
201244

202245
function createStartAdmission(deviceId: string): StartAdmission {
@@ -205,6 +248,7 @@ function createStartAdmission(deviceId: string): StartAdmission {
205248
return {
206249
deviceId,
207250
interestedWaiters,
251+
redirectedTo: undefined,
208252
get admitted() {
209253
return retired === undefined;
210254
},

‎packages/platform-apple/src/runner/runner-start-budget.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,35 @@ export async function raceRunnerStartAgainstCaller(
144144
});
145145
}
146146

147+
/**
148+
* The starting request's own interest in the start it opened, for the surface where the caller
149+
* hands over no `signal` at all: `raceRunnerStartAgainstCaller` registers nothing when there is
150+
* no signal to listen on, and without an owner entry here the start is invisible to the
151+
* last-waiter rule — a joiner canceling could then SIGTERM the live owner's build, which is the
152+
* exact protection the removed #3193 owner sniff used to provide. Keyed on the startup signal
153+
* (a cancelled request, never a deadline), so the owner's disconnect closes admission and stops
154+
* the build the same way a last waiter's does. Returns the release, called once the start
155+
* settles. A caller that did pass a signal already counts through the race; double-keying the
156+
* same caller would keep its cancellation from ever being the last one.
157+
*/
158+
export function reserveRunnerStartOwnerInterest(
159+
admission: RunnerStartAdmission,
160+
options: RunnerSessionOptions,
161+
): () => void {
162+
const signal = options.signal ? undefined : resolveRunnerStartupSignal(options);
163+
if (!signal || addRunnerStartWaiter(admission, signal) !== 'interested') return () => {};
164+
const onAbort = () => {
165+
if (cancelRunnerStartWaiter(admission, signal)) {
166+
void stopRunnerPrepProcesses(admission.deviceId);
167+
}
168+
};
169+
signal.addEventListener('abort', onAbort, { once: true });
170+
return () => {
171+
signal.removeEventListener('abort', onAbort);
172+
releaseRunnerStartWaiter(admission, signal);
173+
};
174+
}
175+
147176
/** A cancelled request killed its start on purpose; any other failure of a start nobody awaits is news. */
148177
function emitDetachedRunnerStartFailed(error: unknown): void {
149178
if (isRequestCanceledError(error)) return;

0 commit comments

Comments
 (0)