Skip to content

Commit 96162ba

Browse files
committed
fix(replay): bounded retry on pre-dispatch capture racing app launch (#1385)
A step right after `open --relaunch` (e.g. a step-2 press) can have its pre-dispatch target-verification capture land while the app is still launching/mounting, producing a transient capture-failed/sparse-snapshot verdict that isn't a real divergence and fails the step closed before it ever dispatches. captureDivergenceObservation now takes an opt-in retryLaunchRace flag that bounds-retries that specific capture (fixed backoff, 12s deadline), mirroring wait's keep-polling landmark semantics (#1349) on this pre-dispatch path. Only verifyReplayActionTarget's gate opts in; the post-failure diagnostic capture and post-resolution guard-mismatch capture stay single-shot since they follow an already-real failure.
1 parent ab913c9 commit 96162ba

4 files changed

Lines changed: 186 additions & 5 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
## Status
44

55
Accepted (2026-07-10). Implemented on `main`, including the amendments folded into the decisions
6-
below (#1264, #1269, #1271 stage 2, #1280, #1349); only decision 5's replay benchmark extension
7-
remains deferred. The per-step landing record (PRs #1193-#1349) and the pre-acceptance migration
8-
plan live in this file's git history.
6+
below (#1264, #1269, #1271 stage 2, #1280, #1349, #1385); only decision 5's replay benchmark
7+
extension remains deferred. The per-step landing record (PRs #1193-#1349) and the pre-acceptance
8+
migration plan live in this file's git history.
99

1010
## Rules at a glance
1111

@@ -29,7 +29,9 @@ Normative summary, one entry per decision. The binding contracts, amendments, an
2929
`identity-unverifiable` divergence, never a silent pick. Amendments: non-unique ids demote to
3030
role+label (#1269); an identity-empty pressed container records its first labeled descendant,
3131
double-guarded fail-closed (#1280); `wait` landmark identity and `is` coverage dispatch on the
32-
`targetIdentityVerification` descriptor trait (#1349).
32+
`targetIdentityVerification` descriptor trait (#1349); the pre-dispatch verification capture
33+
bounds-retries a content-quality `capture-failed`/`sparse-snapshot` verdict on an app-launch race,
34+
opt-in per call site (#1385).
3335
4. **Divergence is a structured error, resumable by plan ordinal.** `ok:false` with code
3436
`REPLAY_DIVERGENCE` and `details.divergence` v1: `kind` (`action-failure` | `selector-miss` |
3537
`identity-mismatch` | `identity-unverifiable`), a bounded `screen` (same capture scope as plain
@@ -369,6 +371,20 @@ An old unannotated action remains executable without this check. All three diver
369371
target-binding divergences reported before the device action. This is not general outcome verification:
370372
`--verify` remains post-action change evidence with a different contract.
371373

374+
> **Amendment (#1385).** The capture this verification runs against (`captureDivergenceObservation`) is
375+
> itself the pre-dispatch gate a step right after `open --relaunch` races: the app can still be
376+
> launching/mounting when it lands, producing a transient content-quality verdict — `capture-failed`
377+
> (Android's snapshot helper returns "insufficient foreground app content" while the app is mounting, and
378+
> the capture path throws) or `sparse-snapshot` (iOS's private-AX fallback under load) — that is not a
379+
> real divergence, only an unlucky capture. This capture now retries with a bounded backoff (fixed delay
380+
> list, capped at a 12s wall-clock deadline) before falling through to `identity-unverifiable`, mirroring
381+
> the keep-polling semantics `wait`'s recorded-landmark identity verification (#1349) already applies on
382+
> its own (post-resolution) path. The retry is opt-in per call site (`retryLaunchRace`), not a change to
383+
> every `captureDivergenceObservation` caller: only the pre-dispatch verification gate in
384+
> `verifyReplayActionTarget` races a launch this way — the post-failure diagnostic capture
385+
> (`buildReplayFailureDivergence`) and the post-resolution guard-mismatch capture both follow an
386+
> already-real failure, where retrying would only delay an already-decided divergence.
387+
372388
### 4. Divergence wire contract and replay-only resume
373389

374390
**Divergence is a structured error, not success data.** The daemon returns `ok:false` with code

‎src/daemon/handlers/__tests__/session-replay-target-verification-runtime.test.ts‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,16 @@ vi.mock('../../../core/dispatch.ts', async (importOriginal) => {
1313
return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() };
1414
});
1515

16+
// #1385's pre-dispatch launch-race retry (captureDivergenceObservation's
17+
// `retryLaunchRace`) awaits a real `sleep` between attempts; stub it to a
18+
// no-op so the retry tests below exercise the retry BRANCH (still runs each
19+
// mocked capture attempt) without a real wall-clock wait (repo guidance: unit
20+
// tests must not wait real time).
21+
vi.mock('../../../utils/timeouts.ts', async (importOriginal) => {
22+
const actual = await importOriginal<typeof import('../../../utils/timeouts.ts')>();
23+
return { ...actual, sleep: vi.fn(async () => {}) };
24+
});
25+
1626
import fs from 'node:fs';
1727
import os from 'node:os';
1828
import path from 'node:path';
@@ -486,6 +496,97 @@ test('a target-binding divergence carries a real computed resume (step 5 wiring,
486496
expect(resume.reason).toBeUndefined();
487497
});
488498

499+
// #1385: a step-2 press right after `open --relaunch` can capture while the
500+
// app is still launching/mounting — the pre-dispatch verification capture
501+
// throws (Android's "insufficient foreground app content" surfaces as a
502+
// thrown capture failure) or returns sparse on the first attempt(s), then
503+
// recovers once the app finishes mounting. The bounded retry in
504+
// `captureDivergenceObservation` (`retryLaunchRace`) must ride that out
505+
// instead of failing closed on the first unlucky capture.
506+
test('a transient pre-dispatch capture failure right after launch recovers within the bounded retry, and the action still dispatches', async () => {
507+
const root = fs.mkdtempSync(
508+
path.join(os.tmpdir(), 'agent-device-replay-target-verify-launch-race-recover-'),
509+
);
510+
const { sessionStore, sessionName } = setupSession(root);
511+
const filePath = writeReplayFile(root, [SAVE_ANNOTATION, 'click id="save"']);
512+
513+
// First two captures fail closed (mirrors a mid-launch/mid-mount capture);
514+
// the third lands after the app settles and finds the recorded target.
515+
mockDispatchCommand
516+
.mockImplementationOnce(async () => {
517+
throw new Error('Android snapshot helper returned insufficient foreground app content');
518+
})
519+
.mockImplementationOnce(async () => {
520+
throw new Error('Android snapshot helper returned insufficient foreground app content');
521+
})
522+
.mockResolvedValue({
523+
nodes: [
524+
{
525+
index: 0,
526+
depth: 0,
527+
type: 'Button',
528+
identifier: 'save',
529+
label: 'Save',
530+
rect: { x: 10, y: 10, width: 40, height: 20 },
531+
},
532+
],
533+
truncated: false,
534+
backend: 'xctest',
535+
});
536+
537+
const invoked: DaemonRequest[] = [];
538+
const response = await runReplayScriptFile({
539+
req: baseReq({ positionals: [filePath] }),
540+
sessionName,
541+
logPath: path.join(root, 'daemon.log'),
542+
sessionStore,
543+
invoke: async (req) => {
544+
invoked.push(req);
545+
return { ok: true, data: {} };
546+
},
547+
});
548+
549+
expect(response.ok).toBe(true);
550+
expect(invoked.map((req) => req.command)).toEqual(['click']);
551+
expect(mockDispatchCommand).toHaveBeenCalledTimes(3);
552+
});
553+
554+
test('a pre-dispatch capture failure that never recovers still fails closed as identity-unverifiable once the bounded retry is exhausted', async () => {
555+
const root = fs.mkdtempSync(
556+
path.join(os.tmpdir(), 'agent-device-replay-target-verify-launch-race-exhausted-'),
557+
);
558+
const { sessionStore, sessionName } = setupSession(root);
559+
const filePath = writeReplayFile(root, [SAVE_ANNOTATION, 'click id="save"']);
560+
561+
mockDispatchCommand.mockImplementation(async () => {
562+
throw new Error('Android snapshot helper returned insufficient foreground app content');
563+
});
564+
565+
const invoked: DaemonRequest[] = [];
566+
const response = await runReplayScriptFile({
567+
req: baseReq({ positionals: [filePath] }),
568+
sessionName,
569+
logPath: path.join(root, 'daemon.log'),
570+
sessionStore,
571+
invoke: async (req) => {
572+
invoked.push(req);
573+
return { ok: true, data: {} };
574+
},
575+
});
576+
577+
expect(invoked.length).toBe(0);
578+
expect(response.ok).toBe(false);
579+
if (response.ok) return;
580+
expect(response.error.code).toBe('REPLAY_DIVERGENCE');
581+
const divergence = response.error.details?.divergence as Record<string, unknown>;
582+
expect(divergence.kind).toBe('identity-unverifiable');
583+
const cause = divergence.cause as { code: string; message: string };
584+
expect(cause.code).toBe('IDENTITY_UNVERIFIABLE');
585+
expect(cause.message).toContain('capture-failed');
586+
// Bounded: the retry gives up after its fixed delay list, not forever.
587+
expect(mockDispatchCommand.mock.calls.length).toBeGreaterThan(1);
588+
});
589+
489590
// ---------------------------------------------------------------------------
490591
// #1349: wait's post-resolution landmark verification. An annotated selector
491592
// wait must NEVER enter the generic pre-dispatch resolution path — polling is

‎src/daemon/handlers/session-replay-divergence.ts‎

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import fs from 'node:fs';
22
import path from 'node:path';
3+
import { sleep } from '../../utils/timeouts.ts';
34
import { markSessionPartialRefsIssued, setSessionSnapshot } from '../session-snapshot.ts';
45
import { isSparseSnapshotQualityVerdict } from '../../snapshot/snapshot-quality.ts';
56
import { displayLabel, formatRole } from '../../snapshot/snapshot-lines.ts';
@@ -216,15 +217,72 @@ export function toReplayRepairHintCapture(
216217
* (`collectSettleChromeRefs`) and the meaningful-target filter stay layered ON
217218
* TOP of this full capture as FILTERS, never as a narrower scoping.
218219
*/
220+
// #1385: a `capture-failed` / `sparse-snapshot` verdict on the PRE-DISPATCH
221+
// target-verification capture (`verifyReplayActionTarget`) right after an
222+
// `open --relaunch` step is often just the app still launching/mounting, not
223+
// a real divergence — the SAME transition `wait`'s keep-polling landmark
224+
// verification (#1349) rides out on its own (post-resolution) path. Bounded
225+
// by BOTH a wall-clock deadline and a fixed delay-list length, so a
226+
// mocked-instant `sleep` in tests cannot turn this into a busy-loop — it just
227+
// runs the list once (mirrors the Android freshness retry shape in
228+
// `snapshot-capture.ts`).
229+
//
230+
// Opt-in via `retryLaunchRace`, NOT the default for every caller: the
231+
// post-failure diagnostic capture (`buildReplayFailureDivergence`) and the
232+
// post-resolution guard-mismatch capture follow an ALREADY-REAL failure, not
233+
// a launch race, and plenty of unit tests stub a throwing `snapshot` dispatch
234+
// there expecting an immediate `capture-failed` result (repo convention: unit
235+
// tests must not wait real time). Retrying unconditionally would force real
236+
// wall-clock waits onto every one of those.
237+
const DIVERGENCE_CAPTURE_RETRY_DEADLINE_MS = 12_000;
238+
const DIVERGENCE_CAPTURE_RETRY_DELAYS_MS = [300, 500, 800, 1200, 2000, 3000, 4000] as const;
239+
219240
export async function captureDivergenceObservation(params: {
220241
session: SessionState;
221242
sessionName: string;
222243
sessionStore: SessionStore;
223244
logPath: string;
224245
action: ReplayReportAction;
246+
retryLaunchRace?: boolean;
225247
}): Promise<DivergenceObservation> {
226-
const { session, sessionName, sessionStore, logPath, action } = params;
248+
const { session, sessionName, sessionStore, logPath, action, retryLaunchRace = false } = params;
227249
const flags = divergenceCaptureFlags(action);
250+
251+
let observation = await captureDivergenceObservationAttempt({
252+
session,
253+
sessionName,
254+
sessionStore,
255+
logPath,
256+
flags,
257+
});
258+
if (!retryLaunchRace) return observation;
259+
260+
const deadline = Date.now() + DIVERGENCE_CAPTURE_RETRY_DEADLINE_MS;
261+
for (const delayMs of DIVERGENCE_CAPTURE_RETRY_DELAYS_MS) {
262+
if (observation.state === 'available') break;
263+
const remainingMs = deadline - Date.now();
264+
if (remainingMs <= 0) break;
265+
await sleep(Math.min(delayMs, remainingMs));
266+
observation = await captureDivergenceObservationAttempt({
267+
session,
268+
sessionName,
269+
sessionStore,
270+
logPath,
271+
flags,
272+
});
273+
}
274+
275+
return observation;
276+
}
277+
278+
async function captureDivergenceObservationAttempt(params: {
279+
session: SessionState;
280+
sessionName: string;
281+
sessionStore: SessionStore;
282+
logPath: string;
283+
flags: CommandFlags;
284+
}): Promise<DivergenceObservation> {
285+
const { session, sessionName, sessionStore, logPath, flags } = params;
228286
try {
229287
const capture = await captureSnapshot({
230288
device: session.device,

‎src/daemon/handlers/session-replay-target-verification.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,12 +317,18 @@ export async function verifyReplayActionTarget(params: {
317317
return { verified: false, response: await buildRecordedUnverifiableResponse() };
318318
}
319319

320+
// #1385: this is the pre-dispatch gate a step right after `open --relaunch`
321+
// can race — the app may still be launching/mounting when this capture
322+
// lands, producing a transient `capture-failed` / `sparse-snapshot`
323+
// verdict that is not a real divergence. Bounded retry rides out that
324+
// transition instead of failing closed on the first unlucky capture.
320325
const observation = await captureDivergenceObservation({
321326
session,
322327
sessionName,
323328
sessionStore,
324329
logPath,
325330
action,
331+
retryLaunchRace: true,
326332
});
327333
if (observation.state !== 'available') {
328334
return {

0 commit comments

Comments
 (0)