Skip to content

Commit bbd53d6

Browse files
authored
fix(ios): budget cold toolchain probes for the first-exec signature stall (callstack#2423)
* fix(ios): budget cold toolchain probes for the first-exec signature stall xcodebuild/xcrun toolchain probes in cache-identity.ts and runner-cache-metadata.ts were budgeted for a warm toolchain (10s/5s), below the ~18-19s syspolicyd signature-verification stall on the first exec after a fresh macOS host boots. Share one 30s floor constant between both call sites and retry once after a timeout while the deadline allows, since the second exec is instant. Closes callstack#2422 * fix(apple): duplicate the cold-toolchain-probe budget instead of a shared module toolchain-probe-budget.ts sat outside every platform-apple facade's eager closure, but runner-cache-metadata.ts (imported from it) sits inside all seven -- so the new import added one module to each, tripping the eager-closure-budgets no-growth gate (callstack#2422). Delete the shared module. cache-identity.ts keeps the canonical constant inline (it was already outside the gated closures); runner-cache-metadata.ts declares its own copy, guarded by a new unit test that asserts the two stay equal. * fix(apple): bound the cold toolchain probes by the owning request budget Three synchronous probes could each retry once at 30 s, so a wedged toolchain host blocked a request for ~180 s with no deadline and no cancellation check. The runner cache decision now takes the owning request's budget (remaining ms + abort signal) and builds one clock per fingerprint read: every attempt runs at min(per-call ceiling, remaining), the retry is skipped once the budget is spent, an exhausted budget fails the decision without starting another probe, and an aborted signal surfaces the cancellation instead of retrying. `ensureXctestrunArtifact` passes the build budget and signal, session reuse passes the startup budget and the request signal, and lease adoption passes the startup budget; a caller with neither is still capped at 45 s total, so the worst case falls from ~180 s to 45 s. Error codes, texts, and the probe hint are unchanged. Both retry classifiers now read the exec layer's structured timeout detail instead of matching "timed out after Nms" in the message. The predicate is exported once from host-kit's command surface and reaches `runner-cache-metadata.ts` through the Apple runner host port, so the file's eager closure is unchanged. Tests use a fake clock that only advances when a probe actually blocks for the timeout it was given, so the exhausted-budget and cancellation cases have to spend the budget to pass; both consumers also pin that an error saying "timed out after 10ms" without the structured detail is not retried. Refs callstack#2422 * fix(apple): spend one deadline across the toolchain probes and the step they precede The Apple runner cache decision runs up to three blocking toolchain probes before the step that needs the decision. Those probes were handed the phase's timeout and the step was then handed the same number again, so a cold-start probe stall added its 30 to 45 seconds on top of the phase budget instead of coming out of it. RunnerCacheProbeBudget now carries the phase's deadline rather than a timeout number, and each caller creates exactly one: - ensureXctestrunArtifact: the probes and xcodebuild read the same clock, and a phase with nothing left fails before the spawn. - ensureRunnerSession: the reuse probe spends from the startup clock, and the new session gets what it left. - tryAdoptRunnerSessionFromLease: the fingerprint probe spends from the caller's clock, and the adopted session gets the remainder. COLD_TOOLCHAIN_PROBE_TIMEOUT_MS now has one owner, core/config.ts. Snapshot source imports it; runner-cache-metadata reads it through the Apple runner host port, because core/config.ts is missing from one of the seven eager closures that evaluate that file and a direct import would grow it. createSnapshotSourceDeadline takes an injectable clock so a test can prove that a probe which blocked for its whole timeout leaves the retry only the remainder. * fix(apple): keep cancellation typed after a probe timeout and simplify the probe budget * fix(apple): check cancellation before the warm toolchain fingerprint cache * refactor(apple): guard each toolchain probe attempt in one place Fold the toolchain probe's three duplicated cancellation/budget guard sites (runToolchainProbe's pre-check, runToolchainProbeCommand's retry pre-check, and execToolchainProbeCommand's timeout computation) into one: attemptToolchainProbe checks cancellation and the remaining budget before every exec, first attempt and retry alike. The outer runToolchainProbe now rethrows cancellation and a spent budget instead of swallowing them into a probe failure, and only genuine probe errors become one. * fix(apple): keep cancellation typed when the final toolchain probe fails The guard fold left one gap: a request that aborts while the last probe is in flight and then fails with a non-timeout error has no next attempt whose guard could see the abort, so the catch classified it as an unreadable toolchain. The catch checks the signal again before classifying, as it did before the fold. * refactor(apple): own the toolchain probe budget in platform-apple and trim narration COLD_TOOLCHAIN_PROBE_TIMEOUT_MS moves from @agent-device/host-kit/command to runner/apple-runner-platform.ts, beside the SDK names the probes are run against. Both Apple toolchain probers import it directly, so the runner host port no longer carries a coldToolchainProbeTimeoutMs() accessor for a plain number. isCommandTimeoutError stays in host-kit, where the exec layer stamps the detail it reads. The comments that narrated control flow the code already shows are gone; the cold-start stall rationale (on the constant), the spawnSync cancellation limitation (on the probe clock) and one line per phase-deadline creation site remain.
1 parent dbfebd4 commit bbd53d6

19 files changed

Lines changed: 975 additions & 74 deletions

‎packages/host-kit/src/command.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ export {
77
execFailureDetails,
88
type ExecOptions,
99
type ExecResult,
10+
isCommandTimeoutError,
1011
isExecutablePath,
1112
requireExecSuccess,
1213
resolveExecutableOverridePath,

‎packages/host-kit/src/internal/exec.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import path from 'node:path';
55
import { flushDiagnosticsToSessionFile, withDiagnosticsScope } from './diagnostics.ts';
66
import {
77
coerceExecResult,
8+
isCommandTimeoutError,
89
requireExecSuccess,
910
runCmd,
1011
runCmdBackground,
@@ -424,3 +425,37 @@ test('coerceExecResult repairs loosely-typed provider results and keeps typed on
424425
} as unknown as ExecResult);
425426
assert.deepEqual(loose, { stdout: '', stderr: '42', exitCode: 1 });
426427
});
428+
429+
test('isCommandTimeoutError reads the structured timeout, not the message text', async () => {
430+
const killedAtTimeout = await runCmd(process.execPath, ['-e', 'setTimeout(() => {}, 10_000)'], {
431+
timeoutMs: 50,
432+
}).then(
433+
() => null,
434+
(error: unknown) => error,
435+
);
436+
assert.ok(isCommandTimeoutError(killedAtTimeout));
437+
438+
assert.ok(
439+
isCommandTimeoutError(
440+
(() => {
441+
try {
442+
runCmdSync(process.execPath, ['-e', 'setTimeout(() => {}, 10_000)'], { timeoutMs: 50 });
443+
return null;
444+
} catch (error) {
445+
return error;
446+
}
447+
})(),
448+
),
449+
);
450+
451+
// A tool that failed on its own and said "timed out" in its output: same
452+
// code, same wording, no timeout we imposed.
453+
assert.equal(
454+
isCommandTimeoutError(
455+
new AppError('COMMAND_FAILED', 'xcodebuild timed out after 10ms', { cmd: 'xcodebuild' }),
456+
),
457+
false,
458+
);
459+
assert.equal(isCommandTimeoutError(new Error('timed out after 10ms')), false);
460+
assert.equal(isCommandTimeoutError(undefined), false);
461+
});

‎packages/host-kit/src/internal/exec.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -592,6 +592,19 @@ function createTimeoutError(
592592
});
593593
}
594594

595+
/**
596+
* True only for the COMMAND_FAILED error this module raises when it kills a command at
597+
* its own `timeoutMs`. Callers classify with this rather than matching the message text,
598+
* so a command whose own output says "timed out" is not mistaken for one exec killed.
599+
*/
600+
export function isCommandTimeoutError(error: unknown): boolean {
601+
return (
602+
error instanceof AppError &&
603+
error.code === 'COMMAND_FAILED' &&
604+
typeof error.details?.timeoutMs === 'number'
605+
);
606+
}
607+
595608
function createExitError(
596609
executable: string,
597610
cmd: string,

‎packages/platform-apple/src/core/runner-host.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { publishFileSync, acquireProcessLock } from '@agent-device/host-kit/file
33
import { resolveIosSimulatorDeviceSetPath } from '@agent-device/kernel/device-isolation';
44

55
import {
6+
isCommandTimeoutError,
67
requireExecSuccess,
78
runCmdBackground,
89
runCmdStreaming,
@@ -55,6 +56,7 @@ export const appleRunnerHost: AppleRunnerHost = {
5556
runCmdSync,
5657
runCmdBackground,
5758
requireExecSuccess,
59+
isCommandTimeoutError,
5860
emitDiagnostic,
5961
withDiagnosticTimer,
6062
retryWithPolicy,

‎packages/platform-apple/src/runner/__tests__/runner-adoption.test.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ import fs from 'node:fs';
22
import path from 'node:path';
33
import { afterEach, beforeEach, expect, test, vi } from 'vitest';
44
import type { DeviceInfo } from '@agent-device/kernel/device';
5+
import { createRequestCanceledError, isRequestCanceledError } from '@agent-device/kernel/errors';
6+
import { resetAllProcessMemosForTests } from '@agent-device/kernel/ttl-memo';
57
import {
68
buildDetachedRunnerLease,
79
buildRunnerLease,
@@ -11,7 +13,9 @@ import {
1113
} from '../runner-lease.ts';
1214
import { isIosRunnerDetachEnabled, tryAdoptRunnerSessionFromLease } from '../runner-adoption.ts';
1315
import { sendRunnerCommandOnce } from '../runner-transport.ts';
16+
import { resolveExpectedRunnerCacheMetadata } from '../runner-xctestrun.ts';
1417
import { appleRunnerTestHost } from '../test-host.ts';
18+
import { appleToolchainProbeResult } from './apple-toolchain-fixtures.ts';
1519
import { mkdtempForTestSync } from './tmp-dir.ts';
1620

1721
vi.mock('../runner-transport.ts', async (importOriginal) => {
@@ -28,6 +32,7 @@ vi.mock('../runner-xctestrun.ts', async (importOriginal) => {
2832
});
2933

3034
const mockSendRunnerCommandOnce = vi.mocked(sendRunnerCommandOnce);
35+
const mockResolveExpectedRunnerCacheMetadata = vi.mocked(resolveExpectedRunnerCacheMetadata);
3136
const mockIsProcessAlive = vi.fn((_pid: number) => false);
3237
const mockReadProcessCommand = vi.fn((_pid: number): string | null => null);
3338
const mockReadProcessStartTime = vi.fn((_pid: number): string | null => 'test-process-start');
@@ -140,6 +145,54 @@ test('adoption succeeds for a live, matching, probe-healthy runner', async () =>
140145
expect(readStaleRunnerLease(simulator.id)).toBeNull();
141146
});
142147

148+
test('a request canceled during the fingerprint probe fails adoption instead of skipping it, cold or with the fingerprint cache warm', async () => {
149+
// The fingerprint check runs the same blocking toolchain probes a fresh
150+
// startup would, and it is handed the request's signal. A cancellation from
151+
// there is not an unresolvable derived path: swallowing it would let startup
152+
// walk on past a client that is already gone (#2422).
153+
const lease = writeStaleLease();
154+
mockIsProcessAlive.mockReturnValue(true);
155+
const request = new AbortController();
156+
request.abort();
157+
mockResolveExpectedRunnerCacheMetadata.mockImplementationOnce((_device, _projectRoot, budget) => {
158+
// The probe only cancels because the request's signal reached it.
159+
expect(budget?.signal?.aborted).toBe(true);
160+
throw createRequestCanceledError({ phase: 'apple_toolchain_probe' });
161+
});
162+
163+
await expect(
164+
tryAdoptRunnerSessionFromLease(simulator, { signal: request.signal }),
165+
).rejects.toSatisfy(isRequestCanceledError);
166+
expect(mockSendRunnerCommandOnce).not.toHaveBeenCalled();
167+
168+
// The mock above only proves adoption propagates whatever the fingerprint
169+
// call throws. Exercise the real production code path too: warm the actual
170+
// toolchain fingerprint memo (the module-level cache in
171+
// runner-cache-metadata.ts is a singleton the mock above never touches),
172+
// then adopt again with an already-aborted signal. Before the fix,
173+
// requireRunnerToolchainFingerprint returned the memoized value without
174+
// checking the signal, so a cache hit let adoption go on to probe uptime
175+
// and write the lease for an already-canceled request (#2422 round 4).
176+
resetAllProcessMemosForTests();
177+
const { resolveExpectedRunnerCacheMetadata: actualResolveExpectedRunnerCacheMetadata } =
178+
await vi.importActual<typeof import('../runner-xctestrun.ts')>('../runner-xctestrun.ts');
179+
appleRunnerTestHost.update({ runCmdSync: vi.fn(appleToolchainProbeResult) });
180+
actualResolveExpectedRunnerCacheMetadata(simulator);
181+
182+
const warmRequest = new AbortController();
183+
warmRequest.abort();
184+
mockResolveExpectedRunnerCacheMetadata.mockImplementationOnce(
185+
actualResolveExpectedRunnerCacheMetadata,
186+
);
187+
188+
await expect(
189+
tryAdoptRunnerSessionFromLease(simulator, { signal: warmRequest.signal }),
190+
).rejects.toSatisfy(isRequestCanceledError);
191+
expect(mockSendRunnerCommandOnce).not.toHaveBeenCalled();
192+
// Ownership was never transferred: the stale lease is untouched.
193+
expect(readStaleRunnerLease(simulator.id)?.ownerToken).toBe(lease.ownerToken);
194+
});
195+
143196
test('adoption is skipped for a recycled runner pid (start time mismatch)', async () => {
144197
// The live process on the leased pid started at a different time than the
145198
// lease recorded — pid recycled since the owner died (#1596). Adopting it
Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
import assert from 'node:assert/strict';
2+
import fs from 'node:fs';
3+
import path from 'node:path';
4+
import { afterEach, beforeEach, test, vi } from 'vitest';
5+
import { AppError } from '@agent-device/kernel/errors';
6+
import { resetAllProcessMemosForTests } from '@agent-device/kernel/ttl-memo';
7+
import { appleRunnerTestHost } from '../test-host.ts';
8+
import type { ExecOptions, ExecResult, ExecStreamOptions } from '../host.ts';
9+
import { ensureXctestrunArtifact } from '../runner-xctestrun.ts';
10+
import { appleToolchainProbeResult } from './apple-toolchain-fixtures.ts';
11+
import { MACOS_DEVICE } from './device-fixtures.ts';
12+
import { mkdtempForTestSync } from './tmp-dir.ts';
13+
14+
/**
15+
* The build phase spends one budget, not two (#2422).
16+
*
17+
* `ensureXctestrunArtifact` runs the cache decision's blocking toolchain probes
18+
* before it starts `xcodebuild`. Those probes used to be handed
19+
* `buildTimeoutMs` and the build was then handed the same number again, so a
20+
* cold-start probe stall added its 30 to 45 seconds on top of the phase budget
21+
* instead of coming out of it. These cases pin the shared deadline: the clock
22+
* moves only when a probe actually blocks for the timeout it was handed, so a
23+
* case that claims the budget was spent had to spend it.
24+
*/
25+
26+
const clock = { nowMs: 0 };
27+
const runCmdSync = vi.fn();
28+
const runCmdStreaming = vi.fn();
29+
let projectRoot: string;
30+
31+
beforeEach(() => {
32+
resetAllProcessMemosForTests();
33+
clock.nowMs = 0;
34+
projectRoot = mkdtempForTestSync('agent-device-runner-phase-root-');
35+
// `buildXctestrunArtifact` refuses to start a build without the runner project.
36+
fs.mkdirSync(
37+
path.join(projectRoot, 'apple', 'runner', 'AgentDeviceRunner', 'AgentDeviceRunner.xcodeproj'),
38+
{ recursive: true },
39+
);
40+
process.env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH = mkdtempForTestSync(
41+
'agent-device-runner-phase-derived-',
42+
);
43+
runCmdSync.mockReset().mockImplementation(appleToolchainProbeResult);
44+
runCmdStreaming
45+
.mockReset()
46+
.mockImplementation(async (): Promise<ExecResult> => ({ exitCode: 0, stdout: '', stderr: '' }));
47+
appleRunnerTestHost.update({
48+
runCmdSync,
49+
runCmdStreaming,
50+
findProjectRoot: () => projectRoot,
51+
readVersion: () => '0.0.0-test',
52+
deadlineFromTimeoutMs: (timeoutMs: number) => {
53+
const startedAtMs = clock.nowMs;
54+
const expiresAtMs = startedAtMs + Math.max(0, timeoutMs);
55+
return {
56+
remainingMs: () => Math.max(0, expiresAtMs - clock.nowMs),
57+
elapsedMs: () => Math.max(0, clock.nowMs - startedAtMs),
58+
isExpired: () => expiresAtMs - clock.nowMs <= 0,
59+
};
60+
},
61+
});
62+
});
63+
64+
afterEach(() => {
65+
delete process.env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH;
66+
});
67+
68+
test('a warm toolchain leaves the build the whole phase budget', async () => {
69+
await assert.rejects(
70+
ensureXctestrunArtifact(MACOS_DEVICE, { buildTimeoutMs: 120_000 }),
71+
missingXctestrun,
72+
);
73+
74+
assert.equal(clock.nowMs, 0);
75+
assert.equal(buildTimeoutMsGiven(), 120_000);
76+
});
77+
78+
test('a cold-start probe stall comes out of the build budget instead of being added to it', async () => {
79+
let xcodebuildProbes = 0;
80+
runCmdSync.mockImplementation((command: string, args: string[], options: ExecOptions) => {
81+
// The syspolicyd stall: the first `xcodebuild` exec blocks for the whole
82+
// timeout it was handed, the immediate next one answers at once.
83+
xcodebuildProbes += command === 'xcodebuild' ? 1 : 0;
84+
if (xcodebuildProbes === 1 && command === 'xcodebuild') {
85+
throw blockForWholeTimeout(command, options);
86+
}
87+
return appleToolchainProbeResult(command, args);
88+
});
89+
90+
await assert.rejects(
91+
ensureXctestrunArtifact(MACOS_DEVICE, { buildTimeoutMs: 120_000 }),
92+
missingXctestrun,
93+
);
94+
95+
// 30 s of stall, absorbed by the retry -- and charged to the phase, so the
96+
// build is given 90 s rather than a second full 120 s.
97+
assert.equal(clock.nowMs, 30_000);
98+
assert.equal(buildTimeoutMsGiven(), 90_000);
99+
});
100+
101+
test('a probe that spends the whole phase fails before xcodebuild is spawned', async () => {
102+
runCmdSync.mockImplementation((command: string, args: string[], options: ExecOptions) => {
103+
// The last of the three probes answers, but only after blocking for
104+
// everything the phase had left.
105+
if (args.includes('--show-sdk-build-version')) clock.nowMs += options.timeoutMs ?? 0;
106+
return appleToolchainProbeResult(command, args);
107+
});
108+
109+
await assert.rejects(
110+
ensureXctestrunArtifact(MACOS_DEVICE, { buildTimeoutMs: 30_000 }),
111+
(error: unknown) =>
112+
error instanceof AppError &&
113+
error.details?.reason === 'runner_phase_budget_exhausted' &&
114+
error.details?.phase === 'runner_xctestrun_build',
115+
);
116+
117+
assert.equal(clock.nowMs, 30_000);
118+
assert.equal(runCmdStreaming.mock.calls.length, 0);
119+
});
120+
121+
/** The fake build writes no `.xctestrun`; the budget it was handed is what these cases read. */
122+
function missingXctestrun(error: unknown): boolean {
123+
return error instanceof AppError && error.message === 'Failed to locate .xctestrun after build';
124+
}
125+
126+
function buildTimeoutMsGiven(): number | undefined {
127+
assert.equal(runCmdStreaming.mock.calls.length, 1);
128+
const [command, , options] = runCmdStreaming.mock.calls[0] as [
129+
string,
130+
string[],
131+
ExecStreamOptions,
132+
];
133+
assert.equal(command, 'xcodebuild');
134+
return options.timeoutMs;
135+
}
136+
137+
/** A probe that blocked for its whole timeout and was then killed, as the exec layer reports it. */
138+
function blockForWholeTimeout(command: string, options: ExecOptions): AppError {
139+
const timeoutMs = options.timeoutMs ?? 0;
140+
clock.nowMs += timeoutMs;
141+
return new AppError('COMMAND_FAILED', `${command} timed out after ${timeoutMs}ms`, {
142+
cmd: command,
143+
timeoutMs,
144+
});
145+
}

0 commit comments

Comments
 (0)