Skip to content

Commit 87c6fbf

Browse files
committed
fix(replay): gate #1385 launch-race retry on content-quality vs mechanism failure
Address review feedback on #1386: the bounded retry was retrying every thrown capture-failed, including permanent mechanism failures (e.g. a missing helper artifact) that a retry can never fix, at the cost of the full backoff budget before the identical divergence. Gate the retry on the same signal Android's helper capture path already emits for this exact distinction: retriable:true on a content-poor rejection (rejectAndroidHelperContentUnavailable), unset on a permanent one (androidSnapshotHelperUnavailableError). The non-throwing sparse-snapshot verdict still always retries. Mirrors #1381's isUnreadableCaptureContentError taxonomy for the wait keep-poll loop. Also update the two #1385 tests to use a retriable-tagged error and add a case proving a permanent failure fails on the first attempt.
1 parent fcc59cc commit 87c6fbf

3 files changed

Lines changed: 120 additions & 35 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -485,6 +485,16 @@ target-binding divergences reported before the device action. This is not genera
485485
> `verifyReplayActionTarget` races a launch this way — the post-failure diagnostic capture
486486
> (`buildReplayFailureDivergence`) and the post-resolution guard-mismatch capture both follow an
487487
> already-real failure, where retrying would only delay an already-decided divergence.
488+
>
489+
> The retry loop is further gated on the SAME content-quality-vs-mechanism-failure taxonomy #1381 draws
490+
> for the wait keep-poll loop (`isUnreadableCaptureContentError`): the non-throwing `sparse-snapshot`
491+
> verdict always retries (it is already a content-quality signal), but a thrown `capture-failed` only
492+
> retries when the underlying error is explicitly marked `retriable: true` — the SAME signal Android's
493+
> helper capture path already emits for a content-poor/system-window-only rejection
494+
> (`rejectAndroidHelperContentUnavailable`, `platforms/android/snapshot.ts`) and leaves unset for a
495+
> permanent one (the helper artifact itself missing, `androidSnapshotHelperUnavailableError`). A
496+
> mechanism failure — helper artifact missing, device offline — therefore still fails on the first
497+
> attempt rather than spending the full 12s retry budget on a foregone conclusion.
488498
489499
### 4. Divergence wire contract and replay-only resume
490500

‎src/daemon/handlers/__tests__/session-replay-target-verification-runtime.test.ts‎

Lines changed: 64 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ import { runReplayScriptFile } from '../session-replay-runtime.ts';
3030
import { SessionStore } from '../../session-store.ts';
3131
import type { DaemonRequest } from '../../types.ts';
3232
import { dispatchCommand } from '../../../core/dispatch.ts';
33+
import { AppError } from '../../../kernel/errors.ts';
3334
import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts';
3435

3536
const mockDispatchCommand = vi.mocked(dispatchCommand);
@@ -497,28 +498,42 @@ test('a target-binding divergence carries a real computed resume (step 5 wiring,
497498
});
498499

499500
// #1385: a step-2 press right after `open --relaunch` can capture while the
500-
// app is still launching/mounting — the pre-dispatch verification capture
501-
// throws (Android's "insufficient foreground app content" surfaces as a
502-
// thrown capture failure) or returns sparse on the first attempt(s), then
503-
// recovers once the app finishes mounting. The bounded retry in
504-
// `captureDivergenceObservation` (`retryLaunchRace`) must ride that out
505-
// instead of failing closed on the first unlucky capture.
506-
test('a transient pre-dispatch capture failure right after launch recovers within the bounded retry, and the action still dispatches', async () => {
501+
// app is still launching/mounting. The bounded retry in
502+
// `captureDivergenceObservation` (`retryLaunchRace`) only rides out a
503+
// content-quality verdict — the same `retriable: true` signal Android's
504+
// helper capture path already emits for a content-poor/system-window-only
505+
// rejection (`rejectAndroidHelperContentUnavailable`) — not a mechanism
506+
// failure (helper artifact missing, device offline), which must still fail
507+
// fast. `throwRetriableCaptureFailure`/`throwPermanentCaptureFailure` below
508+
// build the two shapes.
509+
function throwRetriableCaptureFailure(): never {
510+
throw new AppError(
511+
'COMMAND_FAILED',
512+
'Android snapshot helper returned insufficient foreground app content',
513+
{ retriable: true },
514+
);
515+
}
516+
517+
function throwPermanentCaptureFailure(): never {
518+
throw new AppError(
519+
'COMMAND_FAILED',
520+
'Android snapshot helper is unavailable: the bundled helper artifact was not found',
521+
);
522+
}
523+
524+
test('a transient content-quality capture failure right after launch recovers within the bounded retry, and the action still dispatches', async () => {
507525
const root = fs.mkdtempSync(
508526
path.join(os.tmpdir(), 'agent-device-replay-target-verify-launch-race-recover-'),
509527
);
510528
const { sessionStore, sessionName } = setupSession(root);
511529
const filePath = writeReplayFile(root, [SAVE_ANNOTATION, 'click id="save"']);
512530

513-
// First two captures fail closed (mirrors a mid-launch/mid-mount capture);
514-
// the third lands after the app settles and finds the recorded target.
531+
// First two captures fail closed with a content-quality (retriable) verdict
532+
// (mirrors a mid-launch/mid-mount capture); the third lands after the app
533+
// settles and finds the recorded target.
515534
mockDispatchCommand
516-
.mockImplementationOnce(async () => {
517-
throw new Error('Android snapshot helper returned insufficient foreground app content');
518-
})
519-
.mockImplementationOnce(async () => {
520-
throw new Error('Android snapshot helper returned insufficient foreground app content');
521-
})
535+
.mockImplementationOnce(async () => throwRetriableCaptureFailure())
536+
.mockImplementationOnce(async () => throwRetriableCaptureFailure())
522537
.mockResolvedValue({
523538
nodes: [
524539
{
@@ -551,16 +566,14 @@ test('a transient pre-dispatch capture failure right after launch recovers withi
551566
expect(mockDispatchCommand).toHaveBeenCalledTimes(3);
552567
});
553568

554-
test('a pre-dispatch capture failure that never recovers still fails closed as identity-unverifiable once the bounded retry is exhausted', async () => {
569+
test('a content-quality capture failure that never recovers still fails closed as identity-unverifiable once the bounded retry is exhausted', async () => {
555570
const root = fs.mkdtempSync(
556571
path.join(os.tmpdir(), 'agent-device-replay-target-verify-launch-race-exhausted-'),
557572
);
558573
const { sessionStore, sessionName } = setupSession(root);
559574
const filePath = writeReplayFile(root, [SAVE_ANNOTATION, 'click id="save"']);
560575

561-
mockDispatchCommand.mockImplementation(async () => {
562-
throw new Error('Android snapshot helper returned insufficient foreground app content');
563-
});
576+
mockDispatchCommand.mockImplementation(async () => throwRetriableCaptureFailure());
564577

565578
const invoked: DaemonRequest[] = [];
566579
const response = await runReplayScriptFile({
@@ -586,3 +599,35 @@ test('a pre-dispatch capture failure that never recovers still fails closed as i
586599
// Bounded: the retry gives up after its fixed delay list, not forever.
587600
expect(mockDispatchCommand.mock.calls.length).toBeGreaterThan(1);
588601
});
602+
603+
test('a permanent (non-content-quality) capture failure fails fast without retrying', async () => {
604+
const root = fs.mkdtempSync(
605+
path.join(os.tmpdir(), 'agent-device-replay-target-verify-launch-race-permanent-'),
606+
);
607+
const { sessionStore, sessionName } = setupSession(root);
608+
const filePath = writeReplayFile(root, [SAVE_ANNOTATION, 'click id="save"']);
609+
610+
// A missing helper artifact is not something a launch-race retry can fix —
611+
// it never carries `retriable: true`, so the capture must fail on the
612+
// FIRST attempt, not spend the retry budget on a foregone conclusion.
613+
mockDispatchCommand.mockImplementation(async () => throwPermanentCaptureFailure());
614+
615+
const invoked: DaemonRequest[] = [];
616+
const response = await runReplayScriptFile({
617+
req: baseReq({ positionals: [filePath] }),
618+
sessionName,
619+
logPath: path.join(root, 'daemon.log'),
620+
sessionStore,
621+
invoke: async (req) => {
622+
invoked.push(req);
623+
return { ok: true, data: {} };
624+
},
625+
});
626+
627+
expect(invoked.length).toBe(0);
628+
expect(response.ok).toBe(false);
629+
if (response.ok) return;
630+
const divergence = response.error.details?.divergence as Record<string, unknown>;
631+
expect(divergence.kind).toBe('identity-unverifiable');
632+
expect(mockDispatchCommand).toHaveBeenCalledTimes(1);
633+
});

‎src/daemon/handlers/session-replay-divergence.ts‎

Lines changed: 46 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import fs from 'node:fs';
22
import path from 'node:path';
33
import { sleep } from '../../utils/timeouts.ts';
4+
import { normalizeError } from '../../kernel/errors.ts';
45
import { markSessionPartialRefsIssued, setSessionSnapshot } from '../session-snapshot.ts';
56
import { isSparseSnapshotQualityVerdict } from '../../snapshot/snapshot-quality.ts';
67
import { displayLabel, formatRole } from '../../snapshot/snapshot-lines.ts';
@@ -248,22 +249,22 @@ export async function captureDivergenceObservation(params: {
248249
const { session, sessionName, sessionStore, logPath, action, retryLaunchRace = false } = params;
249250
const flags = divergenceCaptureFlags(action);
250251

251-
let observation = await captureDivergenceObservationAttempt({
252+
let attempt = await captureDivergenceObservationAttempt({
252253
session,
253254
sessionName,
254255
sessionStore,
255256
logPath,
256257
flags,
257258
});
258-
if (!retryLaunchRace) return observation;
259+
if (!retryLaunchRace) return attempt.observation;
259260

260261
const deadline = Date.now() + DIVERGENCE_CAPTURE_RETRY_DEADLINE_MS;
261262
for (const delayMs of DIVERGENCE_CAPTURE_RETRY_DELAYS_MS) {
262-
if (observation.state === 'available') break;
263+
if (attempt.observation.state === 'available' || !attempt.retryable) break;
263264
const remainingMs = deadline - Date.now();
264265
if (remainingMs <= 0) break;
265266
await sleep(Math.min(delayMs, remainingMs));
266-
observation = await captureDivergenceObservationAttempt({
267+
attempt = await captureDivergenceObservationAttempt({
267268
session,
268269
sessionName,
269270
sessionStore,
@@ -272,16 +273,36 @@ export async function captureDivergenceObservation(params: {
272273
});
273274
}
274275

275-
return observation;
276+
return attempt.observation;
276277
}
277278

279+
type DivergenceCaptureAttempt = {
280+
observation: DivergenceObservation;
281+
/**
282+
* Meaningful only when `observation.state === 'unavailable'`: whether this
283+
* particular failure is a content-quality verdict a launch-race retry
284+
* should ride out, versus a mechanism failure that will not resolve itself
285+
* (helper artifact missing, device offline). Mirrors #1381's
286+
* `isUnreadableCaptureContentError` taxonomy for the wait keep-poll loop —
287+
* content verdict retries, mechanism failure fails fast — via the SAME
288+
* signal Android's helper capture path already emits for exactly this
289+
* distinction: `retriable: true` on a content-poor/system-window-only
290+
* rejection (`rejectAndroidHelperContentUnavailable`,
291+
* `platforms/android/snapshot.ts`), left unset on a permanent one (the
292+
* helper artifact itself missing, `androidSnapshotHelperUnavailableError`).
293+
* The non-throwing `sparse-snapshot` verdict is always retryable — it is
294+
* already a content-quality signal, never a mechanism failure.
295+
*/
296+
retryable: boolean;
297+
};
298+
278299
async function captureDivergenceObservationAttempt(params: {
279300
session: SessionState;
280301
sessionName: string;
281302
sessionStore: SessionStore;
282303
logPath: string;
283304
flags: CommandFlags;
284-
}): Promise<DivergenceObservation> {
305+
}): Promise<DivergenceCaptureAttempt> {
285306
const { session, sessionName, sessionStore, logPath, flags } = params;
286307
try {
287308
const capture = await captureSnapshot({
@@ -293,9 +314,12 @@ async function captureDivergenceObservationAttempt(params: {
293314
const snapshot = capture.snapshot;
294315
if (isSparseSnapshotQualityVerdict(snapshot.snapshotQuality)) {
295316
return {
296-
state: 'unavailable',
297-
reason: 'sparse-snapshot',
298-
hint: 'The post-failure snapshot was sparse or unavailable; run snapshot -i to observe the current screen.',
317+
observation: {
318+
state: 'unavailable',
319+
reason: 'sparse-snapshot',
320+
hint: 'The post-failure snapshot was sparse or unavailable; run snapshot -i to observe the current screen.',
321+
},
322+
retryable: true,
299323
};
300324
}
301325
setSessionSnapshot(session, snapshot);
@@ -313,16 +337,22 @@ async function captureDivergenceObservationAttempt(params: {
313337
markSessionPartialRefsIssued(session, digestBodies);
314338
sessionStore.set(sessionName, session);
315339
return {
316-
state: 'available',
317-
nodes: snapshot.nodes,
318-
refsGeneration: session.snapshotGeneration ?? 0,
319-
appBundleId: session.appBundleId,
340+
observation: {
341+
state: 'available',
342+
nodes: snapshot.nodes,
343+
refsGeneration: session.snapshotGeneration ?? 0,
344+
appBundleId: session.appBundleId,
345+
},
346+
retryable: false,
320347
};
321348
} catch (error) {
322349
return {
323-
state: 'unavailable',
324-
reason: 'capture-failed',
325-
hint: `Post-failure snapshot capture failed (${error instanceof Error ? error.message : String(error)}); the original replay failure is unaffected.`,
350+
observation: {
351+
state: 'unavailable',
352+
reason: 'capture-failed',
353+
hint: `Post-failure snapshot capture failed (${error instanceof Error ? error.message : String(error)}); the original replay failure is unaffected.`,
354+
},
355+
retryable: normalizeError(error).retriable === true,
326356
};
327357
}
328358
}

0 commit comments

Comments
 (0)