Skip to content

Commit 4758d27

Browse files
committed
test(apple-runner): drive a real runner death through the fake runner
runner-recovery-wiring: the notAccepted mutation test no longer scripts a second `ok` for the mutation. Nothing consumed it, and it suggested that a resend was expected. Its comment now says what the test models: status answers notAccepted, the way a runner whose journal did not survive a restart would. The path is status recovery, not the daemon's restart. The fake runner server gains an `exit` reply: it hangs up and stops listening, like a runner process that died mid-command. Before this, a test could only hang up on one request while the same server stayed up. Two new tests use a second server as the restarted runner: - press/fill (runner tap/type): the runner dies on the command. The mutation is sent once, is not sent to the restarted runner, fails with dispatched unknown, and the dead session is invalidated. A readText on the next request is served by the restarted runner. Mutation: declare tap and type with READ_ONLY_TRAITS (the resend trait) -> both rows fail. - get (runner readText): the runner dies mid-read. The connect loop gives up with a possibly-written POST (simctl curl exit 28), the daemon restarts the runner (runner_connect_failed_before_command_send), and the read is resent once and succeeds. Mutation: reduce canResendAfterRestart to `evidence.firstAttemptUnwritten` -> fails (no resend). With the real transport, a mutating command never reaches restartSessionAndRunCommand with a possibly-written first attempt. Mutations are sent by sendRunnerCommandOnce, and only the connect loop (waitForRunner, used for reads and the readiness preflight) stamps runner_connect_refused. So the mutating arm of canResendAfterRestart is covered only by the mocked lifecycle driver (ios-runner.transport.written-then-lost). A dying runner sends a mutation to status recovery, and a failed status probe rethrows the transport error without a runner_reply_lost reason.
1 parent be1b89f commit 4758d27

2 files changed

Lines changed: 104 additions & 19 deletions

File tree

‎packages/platform-apple/src/runner/__tests__/fake-runner-server.ts‎

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@ import type { AddressInfo } from 'node:net';
1313
export type FakeRunnerResponse =
1414
| { kind: 'ok'; data: Record<string, unknown> }
1515
| { kind: 'runnerError'; code: string; message: string }
16-
| { kind: 'hangUp' };
16+
| { kind: 'hangUp' }
17+
/** Hangs up and stops listening, as a runner process that died mid-command. */
18+
| { kind: 'exit' };
1719

1820
export type FakeRunnerRequest = {
1921
command: string;
@@ -44,6 +46,7 @@ export async function startFakeRunnerServer(
4446
: Object.fromEntries(Object.entries(script).map(([key, list]) => [key, [...list]]));
4547
const remaining = sequential ?? [];
4648
const requests: FakeRunnerRequest[] = [];
49+
let stopped: Promise<void> | undefined;
4750
const server = http.createServer((req, res) => {
4851
let raw = '';
4952
req.on('data', (chunk) => {
@@ -55,22 +58,13 @@ export async function startFakeRunnerServer(
5558
const next = byCommand
5659
? (byCommand[String(body.command ?? '')]?.shift() ?? { kind: 'ok' as const, data: {} })
5760
: remaining.shift();
58-
if (!next) {
59-
res.statusCode = 500;
60-
res.end(JSON.stringify({ ok: false, error: { message: 'fake runner script exhausted' } }));
61-
return;
62-
}
63-
if (next.kind === 'hangUp') {
61+
if (next?.kind === 'exit') {
6462
res.destroy();
63+
stopped ??= new Promise<void>((resolve) => server.close(() => resolve()));
64+
server.closeAllConnections();
6565
return;
6666
}
67-
if (next.kind === 'runnerError') {
68-
res.setHeader('content-type', 'application/json');
69-
res.end(JSON.stringify({ ok: false, error: { code: next.code, message: next.message } }));
70-
return;
71-
}
72-
res.setHeader('content-type', 'application/json');
73-
res.end(JSON.stringify({ ok: true, data: next.data }));
67+
writeFakeRunnerResponse(res, next);
7468
});
7569
});
7670
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
@@ -79,12 +73,34 @@ export async function startFakeRunnerServer(
7973
port,
8074
requests,
8175
close: () =>
76+
stopped ??
8277
new Promise<void>((resolve, reject) =>
8378
server.close((error) => (error ? reject(error) : resolve())),
8479
),
8580
};
8681
}
8782

83+
function writeFakeRunnerResponse(
84+
res: http.ServerResponse,
85+
next: Exclude<FakeRunnerResponse, { kind: 'exit' }> | undefined,
86+
): void {
87+
if (!next) {
88+
res.statusCode = 500;
89+
res.end(JSON.stringify({ ok: false, error: { message: 'fake runner script exhausted' } }));
90+
return;
91+
}
92+
if (next.kind === 'hangUp') {
93+
res.destroy();
94+
return;
95+
}
96+
res.setHeader('content-type', 'application/json');
97+
if (next.kind === 'runnerError') {
98+
res.end(JSON.stringify({ ok: false, error: { code: next.code, message: next.message } }));
99+
return;
100+
}
101+
res.end(JSON.stringify({ ok: true, data: next.data }));
102+
}
103+
88104
function parseBody(raw: string): Record<string, unknown> {
89105
try {
90106
const parsed: unknown = JSON.parse(raw);

‎packages/platform-apple/src/runner/__tests__/runner-recovery-wiring.test.ts‎

Lines changed: 74 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,11 @@ import {
1717
resolveExpectedRunnerCacheMetadata,
1818
} from '../runner-cache-metadata.ts';
1919
import { captureDiagnostics } from './runner-session-fixtures.ts';
20-
import { startFakeRunnerServer, type FakeRunnerServer } from './fake-runner-server.ts';
20+
import {
21+
startFakeRunnerServer,
22+
type FakeRunnerResponse,
23+
type FakeRunnerServer,
24+
} from './fake-runner-server.ts';
2125
import { resolveRunnerDetachDecision, RunnerCommandAccounting } from '../runner-session-types.ts';
2226
import { requireLifecycleSettlementRows } from './runner-swift-settlement-fixtures.ts';
2327

@@ -41,6 +45,7 @@ import { requireLifecycleSettlementRows } from './runner-swift-settlement-fixtur
4145
*/
4246

4347
let server: FakeRunnerServer | undefined;
48+
let restartedServer: FakeRunnerServer | undefined;
4449

4550
const { ensureRunnerSessionMock, invalidateRunnerSessionMock } = vi.hoisted(() => ({
4651
ensureRunnerSessionMock: vi.fn(),
@@ -93,6 +98,8 @@ const LOST_RESPONSE_MUTATION_ROWS = {
9398
afterEach(async () => {
9499
await server?.close();
95100
server = undefined;
101+
await restartedServer?.close();
102+
restartedServer = undefined;
96103
ensureRunnerSessionMock.mockReset();
97104
invalidateRunnerSessionMock.mockReset();
98105
});
@@ -161,13 +168,13 @@ test.each(Object.values(LOST_RESPONSE_MUTATION_ROWS))(
161168
},
162169
);
163170

164-
// #3074: the runner restarted between the send and the status probe, so its journal is empty and
165-
// `status` answers `notAccepted`. That is no proof the first send did not run.
171+
// #3074: `status` answers `notAccepted`, as a runner whose journal did not survive a restart between
172+
// the send and the probe would. That is no proof the first send did not run.
166173
test.each(Object.values(LOST_RESPONSE_MUTATION_ROWS))(
167-
'a $acceptanceCommand whose reply is lost and whose restarted runner answers notAccepted is sent once',
174+
'a $acceptanceCommand whose reply is lost and whose status answers notAccepted is sent once',
168175
async ({ runnerCommand, request }) => {
169176
server = await startFakeRunnerServer({
170-
[runnerCommand]: [{ kind: 'hangUp' }, { kind: 'ok', data: {} }],
177+
[runnerCommand]: [{ kind: 'hangUp' }],
171178
status: [{ kind: 'ok', data: { lifecycleState: 'notAccepted' } }],
172179
snapshot: [{ kind: 'ok', data: { nodes: [] } }],
173180
});
@@ -212,6 +219,68 @@ test('a read whose reply is lost is resent and succeeds', async () => {
212219
assert.equal(server.requests.filter((entry) => entry.command === 'snapshot').length, 2);
213220
});
214221

222+
// #3074: the runner process dies mid-command (the fake hangs up and stops listening), and the next
223+
// session the daemon gets is a new runner on a second server.
224+
async function runnerDiesOnCommand(
225+
runnerCommand: string,
226+
restartedScript: Record<string, FakeRunnerResponse[]>,
227+
): Promise<FakeRunnerServer> {
228+
server = await startFakeRunnerServer({ [runnerCommand]: [{ kind: 'exit' }] });
229+
restartedServer = await startFakeRunnerServer(restartedScript);
230+
ensureRunnerSessionMock
231+
.mockResolvedValueOnce(makeRunnerSession(server.port))
232+
.mockResolvedValueOnce(makeRunnerSession(restartedServer.port));
233+
// The simctl curl fallback of the connect loop timed out after its POST.
234+
appleRunnerTestHost.update({
235+
runXcrun: vi.fn(async () => ({ exitCode: 28, stdout: '', stderr: 'curl exited 28' })),
236+
});
237+
return restartedServer;
238+
}
239+
240+
function sendsOf(target: FakeRunnerServer, runnerCommand: string): number {
241+
return target.requests.filter((entry) => entry.command === runnerCommand).length;
242+
}
243+
244+
test.each(Object.values(LOST_RESPONSE_MUTATION_ROWS))(
245+
'a $acceptanceCommand whose runner dies mid-command is not sent to the restarted runner',
246+
async ({ runnerCommand, request }) => {
247+
const restarted = await runnerDiesOnCommand(runnerCommand, {
248+
readText: [{ kind: 'ok', data: { text: 'after' } }],
249+
});
250+
251+
await assert.rejects(runAppleRunnerCommand(IOS_SIMULATOR, { ...request }), (error: unknown) => {
252+
assert.ok(error instanceof AppError);
253+
assert.equal(error.details?.dispatched, 'unknown');
254+
return true;
255+
});
256+
assert.equal(invalidateRunnerSessionMock.mock.calls.length, 1, 'the dead runner is dropped');
257+
assert.deepEqual(
258+
await runAppleRunnerCommand(IOS_SIMULATOR, { command: 'readText', x: 5, y: 5 }),
259+
{ text: 'after' },
260+
);
261+
assert.equal(sendsOf(server!, runnerCommand), 1, `${runnerCommand} is dispatched once`);
262+
assert.equal(sendsOf(restarted, runnerCommand), 0, `${runnerCommand} is not resent`);
263+
},
264+
);
265+
266+
test('a get whose runner dies mid-read is resent once on the restarted runner', async () => {
267+
const restarted = await runnerDiesOnCommand('readText', {
268+
readText: [{ kind: 'ok', data: { text: 'hello' } }],
269+
});
270+
271+
assert.deepEqual(
272+
await runAppleRunnerCommand(IOS_SIMULATOR, { command: 'readText', x: 5, y: 5 }),
273+
{ text: 'hello' },
274+
);
275+
expect(invalidateRunnerSessionMock).toHaveBeenCalledTimes(1);
276+
expect(invalidateRunnerSessionMock).toHaveBeenCalledWith(
277+
expect.anything(),
278+
'runner_connect_failed_before_command_send',
279+
);
280+
assert.equal(sendsOf(server!, 'readText'), 1);
281+
assert.equal(sendsOf(restarted, 'readText'), 1, 'the read is resent once');
282+
});
283+
215284
// #2965: an inline `status` probe answers while the command it probes may still be executing, so its
216285
// own reply must not clear the mutation's outstanding charge. The handoff verdict is asserted through
217286
// `resolveRunnerDetachDecision` — the exact gate `detachRunnerSessionForShutdown` consults. Rows come

0 commit comments

Comments
 (0)