Skip to content

Commit 6739a58

Browse files
committed
refactor(apple-runner): gate the restart resend at the connect-failure call site
A mutation never reaches the connect loop, so the no-resend arm after a restart had no production route. The call site now restarts only when the first attempt provably wrote nothing or the command is read-only, and the arm and its error builder are gone. The written-then-lost row is driven through a fake runner that dies mid-command. A failed status probe keeps a transport reason under transportReason.
1 parent 1edcb78 commit 6739a58

7 files changed

Lines changed: 66 additions & 86 deletions

File tree

‎contracts/fixtures/dispatch-disclosure.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,7 @@
238238
{
239239
"id": "ios-runner.transport.written-then-lost",
240240
"producer": "ios-runner",
241-
"trigger": "a connect attempt posted a mutating command and then timed out (fetch deadline, simctl curl exit 28); the runner is restarted, and the command is not sent again because nothing proves the first send did not run",
241+
"trigger": "a mutating command was posted and the runner process died before replying; the status probe fails, the session is invalidated, and the command is not sent again (details.reason runner_reply_lost)",
242242
"dispatched": "unknown"
243243
},
244244
{

‎docs/adr/0011-interaction-guarantee-contract.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -189,17 +189,18 @@ repeat. A producer that runs several device inputs inside one bound operation
189189
Each row names its driver file by id prefix, and that file drives the real
190190
producer.
191191

192+
The Apple runner does not resend a mutating command whose first send may have
193+
run: a restart resends only a command the first attempt provably did not write,
194+
or a read-only one. A mutation whose runner dies mid-command fails with
195+
`reason: runner_reply_lost` and `dispatched: unknown` (row
196+
`ios-runner.transport.written-then-lost`).
197+
192198
Remaining gaps: a failure before the router's locked scope (session
193199
resolution, lock acquisition, lease and daemon-policy admission) never reaches
194200
the disclosure and carries no `dispatched`. It sends nothing, but a consumer
195201
must read the absent field as `unknown`. The public `agent-device/batch`
196202
`runBatch` has no ledger when a caller supplies its own `invoke`, so a batch run
197-
outside the daemon keeps the failing step's value. The row
198-
`ios-runner.transport.written-then-lost` proves the Apple runner's rule that a
199-
mutating command whose first send may have run is not resent after a restart,
200-
not a production route: over the real transport that arm is unreachable,
201-
because mutations go through `sendRunnerCommandOnce` and only the connect loop
202-
raises the restart trigger.
203+
outside the daemon keeps the failing step's value.
203204

204205
For `responseFields`, one `buildInteractionResponseData(...)` becomes the only
205206
construction site for interaction response payloads (this deletes the class of

‎packages/platform-apple/src/runner/__tests__/runner-command-recovery.test.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,13 +142,15 @@ test('notAccepted from a restarted runner fails the lost command as unknown, nam
142142
test('a failing status probe retains the invalidation and names the lost reply', async () => {
143143
const { result, invalidate, transportError } = await runRecovery({
144144
script: [{ kind: 'runnerError', code: 'COMMAND_FAILED', message: 'status probe exploded' }],
145+
transportError: new AppError('COMMAND_FAILED', 'socket hang up', { reason: 'socket_reset' }),
145146
});
146147

147148
await assert.rejects(result, (error: unknown) => {
148149
assert.ok(error instanceof AppError);
149150
assert.equal(error.message, transportError.message);
150151
assert.equal(error.cause, transportError);
151152
assert.equal(error.details?.reason, RUNNER_REPLY_LOST_REASON);
153+
assert.equal(error.details?.transportReason, 'socket_reset');
152154
assert.equal(error.details?.recovery, 'status_probe_failed');
153155
assert.equal(error.details?.dispatched, 'unknown');
154156
return true;

‎packages/platform-apple/src/runner/__tests__/runner-lifecycle-dispatch-disclosure.test.ts‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import {
1010
import { IOS_SIMULATOR } from './device-fixtures.ts';
1111
import { createTestRequestCancellation, makeRunnerSession } from './runner-session-fixtures.ts';
1212
import { appleRunnerTestHost } from '../test-host.ts';
13+
import { startFakeRunnerServer, type FakeRunnerServer } from './fake-runner-server.ts';
1314

1415
// contracts/fixtures/dispatch-disclosure.json, ios-runner pre-send and transport rows: each row
1516
// drives runAppleRunnerCommand through the real lifecycle and restart path with the session start
@@ -59,8 +60,12 @@ beforeEach(() => {
5960
});
6061
});
6162

62-
afterEach(() => {
63+
let fakeRunner: FakeRunnerServer | undefined;
64+
65+
afterEach(async () => {
6366
vi.unstubAllGlobals();
67+
await fakeRunner?.close();
68+
fakeRunner = undefined;
6469
});
6570

6671
async function tap(): Promise<unknown> {
@@ -164,17 +169,35 @@ const DRIVERS: Record<string, () => Promise<unknown>> = {
164169
curlExitCode: 7,
165170
}),
166171
'ios-runner.transport.written-then-lost': async () => {
172+
const runnerSession =
173+
await vi.importActual<typeof import('../runner-session.ts')>('../runner-session.ts');
174+
const runner = await startFakeRunnerServer({ tap: [{ kind: 'exit' }] });
175+
fakeRunner = runner;
176+
// The status probe finds no listener, over fetch or the simctl curl fallback.
177+
const { retryWithPolicy } = appleRunnerTestHost.defaults();
178+
appleRunnerTestHost.update({
179+
runXcrun: vi.fn(async () => ({ exitCode: 7, stdout: '', stderr: 'curl exited 7' })),
180+
retryWithPolicy: (task, policy, options) =>
181+
retryWithPolicy(task, { ...policy, baseDelayMs: 1, maxDelayMs: 1, jitter: 0 }, options),
182+
});
183+
mockEnsureRunnerSession.mockResolvedValueOnce(makeRunnerSession({ port: runner.port }));
184+
mockExecuteRunnerCommandWithSession.mockImplementation(
185+
runnerSession.executeRunnerCommandWithSession,
186+
);
167187
try {
168-
return await writtenThenLostThenRestarted(
169-
{ command: 'tap', x: 120, y: 240 },
170-
async () => ({}),
171-
);
188+
return await tap();
172189
} catch (error) {
173190
assert.ok(error instanceof AppError);
174191
assert.equal(error.details?.reason, RUNNER_REPLY_LOST_REASON);
175-
assert.equal(error.details?.runnerRestarted, true);
176-
assert.equal(sendsOnRestartedRunner(), 0, 'the tap is not sent again');
192+
assert.equal(error.details?.recovery, 'status_probe_failed');
193+
assert.equal(error.details?.runnerRestarted, undefined, 'the runner is not restarted');
177194
throw error;
195+
} finally {
196+
assert.equal(runner.requests.filter((entry) => entry.command === 'tap').length, 1);
197+
assert.deepEqual(
198+
mockInvalidateRunnerSession.mock.calls[0]?.[1],
199+
'transport_error_after_command_send',
200+
);
178201
}
179202
},
180203
'ios-runner.transport.read-only-written-then-lost': async () => {

‎packages/platform-apple/src/runner/runner-command-recovery.ts‎

Lines changed: 3 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -53,31 +53,6 @@ const RUNNER_STATUS_RECOVERY_TIMEOUT_MS = 3_000;
5353
*/
5454
export const RUNNER_REPLY_LOST_REASON = 'runner_reply_lost';
5555

56-
/**
57-
* The command was written and its reply lost, and the runner was restarted without resending it:
58-
* a restarted runner's journal cannot say whether the first send ran.
59-
*/
60-
export function buildRunnerRestartedWithoutResendError(
61-
command: RunnerCommand,
62-
transportError: AppError,
63-
options: AppleRunnerCommandOptions,
64-
): AppError {
65-
return new AppError(
66-
'COMMAND_FAILED',
67-
`Runner command "${command.command}" may have run before its reply was lost, so agent-device restarted the runner without sending it again.`,
68-
{
69-
command: command.command,
70-
commandId: command.commandId,
71-
reason: RUNNER_REPLY_LOST_REASON,
72-
recovery: 'runner_restarted_without_resend',
73-
hint: `Run snapshot -i to inspect the current UI and check whether "${command.command}" took effect before you retry it.`,
74-
logPath: options.logPath,
75-
transportError: transportError.message,
76-
},
77-
transportError,
78-
);
79-
}
80-
8156
export async function handleRunnerTransportErrorAfterCommandSend(params: {
8257
device: DeviceInfo;
8358
session: RunnerSession;
@@ -521,6 +496,9 @@ function buildStatusProbeFailedError(
521496
transportError.message,
522497
{
523498
...transportError.details,
499+
...(transportError.details?.reason === undefined
500+
? {}
501+
: { transportReason: transportError.details.reason }),
524502
command: command.command,
525503
commandId: command.commandId,
526504
reason: RUNNER_REPLY_LOST_REASON,

‎packages/platform-apple/src/runner/runner-error-classification.ts‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -115,10 +115,7 @@ type RunnerErrorVerdicts = {
115115
connectRetry?: boolean;
116116
/** Session-fatal classification: invalidate the cached runner session with this reason. */
117117
sessionFatalReason?: string;
118-
/**
119-
* Connect-shaped failure: restart the session. The command is resent only when the attempt
120-
* provably wrote nothing or the command is read-only.
121-
*/
118+
/** Connect-shaped failure before the command was sent: restart the session and replay. */
122119
restartBeforeSend?: boolean;
123120
/** Readiness preflight gave up before the command was written: restart the session and replay. */
124121
restartAfterReadinessPreflight?: boolean;
@@ -613,9 +610,9 @@ export function resolveRunnerFatalErrorReason(error: unknown): string | undefine
613610
}
614611

615612
/**
616-
* A connect-shaped failure: restart the runner session rather than probing a runner that never
617-
* accepted the connection. Whether the command is resent on the restarted runner is decided by
618-
* what proves the first attempt could not have run it.
613+
* A connect-shaped failure that surfaced before the command was sent: restart
614+
* the runner session and replay the command, rather than probing a runner
615+
* that never accepted the connection.
619616
*/
620617
export function shouldRestartRunnerBeforeCommandSend(error: unknown): boolean {
621618
return runnerErrorVerdict(error, 'restartBeforeSend') ?? false;

‎packages/platform-apple/src/runner/runner-lifecycle.ts‎

Lines changed: 19 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,7 @@ import type {
4242
AppleRunnerPrepareResult,
4343
} from './runner-provider.ts';
4444
import { markRunnerXctestrunArtifactBadForRun } from './runner-xctestrun.ts';
45-
import {
46-
buildRunnerRestartedWithoutResendError,
47-
handleRunnerTransportErrorAfterCommandSend,
48-
} from './runner-command-recovery.ts';
45+
import { handleRunnerTransportErrorAfterCommandSend } from './runner-command-recovery.ts';
4946
import { isReadOnlyRunnerCommand } from './runner-command-traits.ts';
5047
import {
5148
buildRunnerRecycleBudgetExhaustedError,
@@ -355,7 +352,12 @@ async function executeRunnerCommandAttempt(
355352
appErr,
356353
);
357354
}
358-
if (shouldRestartRunnerBeforeCommandSend(appErr) && session) {
355+
const firstAttemptUnwritten = isRunnerPreSendRefusal(appErr);
356+
if (
357+
shouldRestartRunnerBeforeCommandSend(appErr) &&
358+
session &&
359+
(firstAttemptUnwritten || isReadOnlyRunnerCommand(command))
360+
) {
359361
assertRunnerRequestActive(options.requestId);
360362
return await restartSessionAndRunCommand({
361363
device,
@@ -364,8 +366,7 @@ async function executeRunnerCommandAttempt(
364366
options,
365367
signal,
366368
restartReason: 'runner_connect_failed_before_command_send',
367-
firstAttemptError: appErr,
368-
firstAttemptUnwritten: isRunnerPreSendRefusal(appErr),
369+
firstAttemptUnwritten,
369370
});
370371
}
371372
if (session && shouldRestartRunnerAfterReadinessPreflight(appErr)) {
@@ -378,7 +379,6 @@ async function executeRunnerCommandAttempt(
378379
signal,
379380
restartReason: 'runner_readiness_preflight_failed_before_command_send',
380381
recoveredDiagnosticPhase: 'ios_runner_readiness_preflight_recovered',
381-
firstAttemptError: appErr,
382382
firstAttemptUnwritten: true,
383383
});
384384
}
@@ -411,8 +411,10 @@ async function restartSessionAndRunCommand(params: {
411411
| 'runner_connect_failed_before_command_send'
412412
| 'runner_readiness_preflight_failed_before_command_send';
413413
recoveredDiagnosticPhase?: string;
414-
firstAttemptError: AppError;
415-
/** The failed first attempt provably never wrote the command. */
414+
/**
415+
* The failed first attempt provably never wrote the command. When it may have, the replay can
416+
* double-send, and no failure of this restart may claim `no`.
417+
*/
416418
firstAttemptUnwritten: boolean;
417419
}): Promise<Record<string, unknown>> {
418420
const { device, command, options, signal, restartReason } = params;
@@ -434,13 +436,6 @@ async function restartSessionAndRunCommand(params: {
434436
throw markRunnerRestartError(error, params);
435437
});
436438
commitRunnerRecycle(recycleKey);
437-
if (!canResendAfterRestart(params)) {
438-
throw markRunnerRestartError(
439-
buildRunnerRestartedWithoutResendError(command, params.firstAttemptError, options),
440-
params,
441-
restartedSession,
442-
);
443-
}
444439
try {
445440
const recovered = await executeRunnerCommandWithSession(
446441
device,
@@ -485,20 +480,6 @@ async function restartSessionAndRunCommand(params: {
485480
}
486481
}
487482

488-
type RunnerRestartResendEvidence = Pick<
489-
Parameters<typeof restartSessionAndRunCommand>[0],
490-
'command' | 'firstAttemptUnwritten'
491-
>;
492-
493-
/**
494-
* Whether sending the command again on the restarted runner cannot run it twice: the first attempt
495-
* provably wrote nothing, or the command is read-only by its runner trait. A restarted runner's
496-
* journal is empty, so its `status` cannot prove the first send did not run.
497-
*/
498-
function canResendAfterRestart(evidence: RunnerRestartResendEvidence): boolean {
499-
return evidence.firstAttemptUnwritten || isReadOnlyRunnerCommand(evidence.command);
500-
}
501-
502483
function markRunnerRestartError(
503484
error: unknown,
504485
params: Pick<
@@ -525,23 +506,21 @@ function markRunnerRestartError(
525506
},
526507
error.cause ?? error,
527508
);
528-
return discloseRestartDispatch(marked, params, restartedSession);
509+
return discloseRestartDispatch(marked, params.firstAttemptUnwritten, restartedSession);
529510
}
530511

531512
/**
532-
* The transport fact, for reads and mutations alike: a first attempt that may have written the
533-
* command is `unknown`, whether or not a read was resent. After an unwritten first attempt, a
534-
* restart that never replayed is `no`, and a replay's own verdict stands; without one, only a
535-
* pre-send refusal is `no`. That a read repeats no app-visible action is the daemon router's rule
536-
* over the registry's `recordingEffect`, not this producer's.
513+
* A restart that never replayed says what the first attempt did; a replay after a first attempt
514+
* that may have written the command cannot claim `no` for the two sends together. After an unwritten
515+
* first attempt the replay's own verdict stands; without one, only a pre-send refusal is `no`.
537516
*/
538517
function discloseRestartDispatch(
539518
error: AppError,
540-
evidence: RunnerRestartResendEvidence,
519+
firstAttemptUnwritten: boolean,
541520
restartedSession: RunnerSession | undefined,
542521
): AppError {
543-
if (!evidence.firstAttemptUnwritten) return discloseDispatch(error, 'unknown');
544-
if (!restartedSession) return discloseDispatch(error, 'no');
522+
if (!restartedSession) return discloseDispatch(error, firstAttemptUnwritten ? 'no' : 'unknown');
523+
if (!firstAttemptUnwritten) return discloseDispatch(error, 'unknown');
545524
return discloseUnclassifiedDispatch(error, isRunnerPreSendRefusal(error) ? 'no' : 'unknown');
546525
}
547526

0 commit comments

Comments
 (0)