Skip to content

Commit 5ef52ad

Browse files
authored
fix(replay): bounded retry on pre-dispatch capture racing app launch (#1385) (#1386)
* fix(replay): bounded retry on pre-dispatch capture racing app launch (#1385) A step right after `open --relaunch` (e.g. a step-2 press) can have its pre-dispatch target-verification capture land while the app is still launching/mounting, producing a transient capture-failed/sparse-snapshot verdict that isn't a real divergence and fails the step closed before it ever dispatches. captureDivergenceObservation now takes an opt-in retryLaunchRace flag that bounds-retries that specific capture (fixed backoff, 12s deadline), mirroring wait's keep-polling landmark semantics (#1349) on this pre-dispatch path. Only verifyReplayActionTarget's gate opts in; the post-failure diagnostic capture and post-resolution guard-mismatch capture stay single-shot since they follow an already-real failure. * fix(replay): gate #1385 launch-race retry on content-quality vs mechanism failure Address review feedback on #1386: the bounded retry was retrying every thrown capture-failed, including permanent mechanism failures (e.g. a missing helper artifact) that a retry can never fix, at the cost of the full backoff budget before the identical divergence. Gate the retry on the same signal Android's helper capture path already emits for this exact distinction: retriable:true on a content-poor rejection (rejectAndroidHelperContentUnavailable), unset on a permanent one (androidSnapshotHelperUnavailableError). The non-throwing sparse-snapshot verdict still always retries. Mirrors #1381's isUnreadableCaptureContentError taxonomy for the wait keep-poll loop. Also update the two #1385 tests to use a retriable-tagged error and add a case proving a permanent failure fails on the first attempt. * fix(replay): narrow #1385 retry to a real content-quality taxonomy, anchor the deadline Address second review pass on #1386: P1 - `retriable === true` was too broad: Android's adb layer (adb-executor.ts) marks genuine transport mechanism failures retriable too (connection_dropped, device_offline, server_version_mismatch - an unchanged retry of the SAME adb command can succeed there), so those consumed the launch-race retry budget despite being exactly the mechanism failures the ADR says must fail fast. Replace the check with the narrow discriminator: the thrown error's androidSnapshotHelperFailureReason must be one of the three literal content verdicts rejectAndroidHelperContentUnavailable attaches (empty-helper-output, system-window-only, content-poor-app-window) - never the free-form message androidSnapshotHelperCaptureError sets for a crashed/timed-out helper, and never present at all for a permanently missing helper artifact. Added a regression test for an adb mechanism failure marked retriable at the transport level, proving it still fails on the first attempt. P2 - the 12s deadline started AFTER the first (unbounded) capture attempt, so the effective wall-clock cost was "first capture + 12s of retries", undocumented and untested as such. Anchor the deadline before the first attempt instead, and rewrite the comments/ADR to state precisely what is and is not bounded: the delay list caps attempt count, the deadline caps sleep time from entry, neither caps an individual capture's own duration. Added a fake-timer test proving the deadline (not just the delay array's length) is what stops retries when captures themselves consume real time. * test(replay): add targeted iOS sparse-snapshot retry regressions Address the reviewer's second finding on #1386: the Android capture-failed retry path had recover/exhaustion regressions, but the iOS sparse-snapshot verdict branch (isSparseSnapshotQualityVerdict, always-retryable, no thrown error involved) had no test of its own - only incidental coverage by way of the Android-shaped tests. Add the mirror pair: a sparse verdict that recovers within the bounded retry and dispatches, and one that never recovers and fails closed as identity-unverifiable once the retry is exhausted. * fix(replay): re-consume shared #1381 predicate after rebase reintroduced local one The branch's rebase onto main re-resolved the #1381 conflict without carrying forward the earlier fix, reintroducing a local ANDROID_CAPTURE_CONTENT_QUALITY_REASONS/isCaptureContentQualityFailure duplicate of the shared isUnreadableCaptureContentError predicate (src/snapshot/snapshot-quality.ts, landed by #1381). Two parallel copies of the same taxonomy can silently drift. Drop the local duplicate and its Android-platform-type import; consume the shared predicate directly, matching what ADR 0012 already documents.
1 parent 9b48e11 commit 5ef52ad

5 files changed

Lines changed: 510 additions & 18 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
## Status
44

55
Accepted (2026-07-10). Implemented on `main`, including the amendments folded into the decisions
6-
below (#1264, #1269, #1271 stage 2, #1280, #1349); only decision 5's replay benchmark extension
7-
remains deferred. The per-step landing record (PRs #1193-#1349) and the pre-acceptance migration
8-
plan live in this file's git history.
6+
below (#1264, #1269, #1271 stage 2, #1280, #1349, #1385); only decision 5's replay benchmark
7+
extension remains deferred. The per-step landing record (PRs #1193-#1349) and the pre-acceptance
8+
migration plan live in this file's git history.
99

1010
## Rules at a glance
1111

@@ -29,7 +29,9 @@ Normative summary, one entry per decision. The binding contracts, amendments, an
2929
`identity-unverifiable` divergence, never a silent pick. Amendments: non-unique ids demote to
3030
role+label (#1269); an identity-empty pressed container records its first labeled descendant,
3131
double-guarded fail-closed (#1280); `wait` landmark identity and `is` coverage dispatch on the
32-
`targetIdentityVerification` descriptor trait (#1349).
32+
`targetIdentityVerification` descriptor trait (#1349); the pre-dispatch verification capture
33+
bounds-retries a content-quality `capture-failed`/`sparse-snapshot` verdict on an app-launch race,
34+
opt-in per call site (#1385).
3335
4. **Divergence is a structured error, resumable by plan ordinal.** `ok:false` with code
3436
`REPLAY_DIVERGENCE` and `details.divergence` v1: `kind` (`action-failure` | `selector-miss` |
3537
`identity-mismatch` | `identity-unverifiable`), a bounded `screen` (same capture scope as plain
@@ -369,6 +371,41 @@ An old unannotated action remains executable without this check. All three diver
369371
target-binding divergences reported before the device action. This is not general outcome verification:
370372
`--verify` remains post-action change evidence with a different contract.
371373

374+
> **Amendment (#1385).** The capture this verification runs against (`captureDivergenceObservation`) is
375+
> itself the pre-dispatch gate a step right after `open --relaunch` races: the app can still be
376+
> launching/mounting when it lands, producing a transient content-quality verdict — `capture-failed`
377+
> (Android's snapshot helper returns "insufficient foreground app content" while the app is mounting, and
378+
> the capture path throws) or `sparse-snapshot` (iOS's private-AX fallback under load) — that is not a
379+
> real divergence, only an unlucky capture. This capture now retries with a bounded backoff (fixed
380+
> 7-entry delay list, 12s DELAY-ONLY budget — see below) before falling through to `identity-unverifiable`,
381+
> mirroring the keep-polling semantics `wait`'s recorded-landmark identity verification (#1349) already
382+
> applies on its own (post-resolution) path. The retry is opt-in per call site (`retryLaunchRace`), not a
383+
> change to every `captureDivergenceObservation` caller: only the pre-dispatch verification gate in
384+
> `verifyReplayActionTarget` races a launch this way — the post-failure diagnostic capture
385+
> (`buildReplayFailureDivergence`) and the post-resolution guard-mismatch capture both follow an
386+
> already-real failure, where retrying would only delay an already-decided divergence.
387+
>
388+
> The retry loop is further gated on the SAME content-quality-vs-mechanism-failure taxonomy #1381 draws
389+
> for the wait keep-poll loop (`isUnreadableCaptureContentError`): the non-throwing `sparse-snapshot`
390+
> verdict always retries (it is already a content-quality signal), but a thrown `capture-failed` only
391+
> retries when the underlying error's `androidSnapshotHelperFailureReason` is one of the three literal
392+
> codes `rejectAndroidHelperContentUnavailable` (`platforms/android/snapshot.ts`) attaches to a
393+
> content-poor/system-window-only rejection — `empty-helper-output`, `system-window-only`,
394+
> `content-poor-app-window` (mirroring `AndroidHelperContentRecoveryDecision['reason']`,
395+
> `platforms/android/snapshot-content-recovery.ts`). This is deliberately narrower than the error's own
396+
> generic `retriable` flag: Android's adb layer separately marks true mechanism failures retriable too
397+
> (`connection_dropped`, `device_offline`, `server_version_mismatch` — an unchanged retry of the SAME adb
398+
> command can succeed there), and a helper artifact permanently missing
399+
> (`androidSnapshotHelperUnavailableError`) carries neither signal. A mechanism failure therefore still
400+
> fails on the first attempt rather than spending the retry budget on a foregone conclusion, regardless of
401+
> what its own `retriable` flag says.
402+
>
403+
> The 12s budget is a DELAY-ONLY bound, anchored at the first capture attempt: it caps how long this loop
404+
> sleeps between retries, not how long any individual capture attempt itself may run (a capture already
405+
> carries its own platform-level timeouts this loop does not shorten or re-implement). The fixed-length
406+
> delay array is a separate, independent bound on attempt COUNT, so a mocked-instant `sleep` in unit tests
407+
> cannot turn this into a real-time busy-loop.
408+
372409
### 4. Divergence wire contract and replay-only resume
373410

374411
**Divergence is a structured error, not success data.** The daemon returns `ok:false` with code

‎src/daemon/handlers/__tests__/session-replay-divergence.test.ts‎

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,16 @@ vi.mock('../../../core/dispatch.ts', async (importOriginal) => {
1414
// in `snapshot-capture.ts` (`capturePostActionAwareSnapshot`) awaits; making it
1515
// instant does not change control flow — the loop still runs, retries, and
1616
// re-captures — so the test still proves two dispatches and use of the retried
17-
// tree. No other test in this file triggers a sleep path.
17+
// tree. The #1385 `retryLaunchRace` retry loop (`captureDivergenceObservation`)
18+
// awaits this SAME `sleep`, further down in this file.
1819
vi.mock('../../../utils/timeouts.ts', async (importOriginal) => {
1920
const actual = await importOriginal<typeof import('../../../utils/timeouts.ts')>();
2021
return { ...actual, sleep: vi.fn(async () => {}) };
2122
});
2223

2324
import { dispatchCommand } from '../../../core/dispatch.ts';
2425
import type { RawSnapshotNode } from '../../../kernel/snapshot.ts';
26+
import { AppError } from '../../../kernel/errors.ts';
2527
import {
2628
makeAndroidSession,
2729
makeIosSession,
@@ -32,7 +34,10 @@ import {
3234
walkNonRawAndroidFixture,
3335
} from '../../../__tests__/test-utils/android-ui-hierarchy-fixtures.ts';
3436
import { SessionStore } from '../../session-store.ts';
35-
import { buildReplayFailureDivergence } from '../session-replay-divergence.ts';
37+
import {
38+
buildReplayFailureDivergence,
39+
captureDivergenceObservation,
40+
} from '../session-replay-divergence.ts';
3641

3742
const mockDispatchCommand = vi.mocked(dispatchCommand);
3843

@@ -1135,3 +1140,56 @@ test.each([
11351140
expect(screen.refs.some((ref) => ref.label === 'Battery charging, 100 percent.')).toBe(false);
11361141
expect(screen.refs.some((ref) => ref.label === 'Wifi signal full.')).toBe(false);
11371142
});
1143+
1144+
// #1385 P2: the retry deadline is a DELAY-ONLY budget, not a per-attempt
1145+
// capture timeout — this loop does not itself bound how long a single
1146+
// capture attempt runs, only how much SLEEP time it spends between attempts
1147+
// (measured from the first attempt). This test proves that bound is real
1148+
// wall-clock behavior, not merely "run the fixed 7-entry delay list": each
1149+
// mocked capture attempt here advances the (fake) system clock by 5s to
1150+
// stand in for a slow capture, so the 12s deadline is exhausted after just 3
1151+
// attempts — far short of the 8 attempts (1 + 7 delays) the array alone would
1152+
// allow — proving the deadline, not the array length, is what stopped it.
1153+
test('captureDivergenceObservation retryLaunchRace: the 12s deadline bounds retries even when captures themselves consume wall-clock time', async () => {
1154+
vi.useFakeTimers();
1155+
try {
1156+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-deadline-'));
1157+
const sessionStore = new SessionStore(path.join(root, 'sessions'));
1158+
const sessionName = 'default';
1159+
sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' }));
1160+
1161+
mockDispatchCommand.mockImplementation(async () => {
1162+
vi.setSystemTime(new Date(Date.now() + 5_000));
1163+
throw new AppError(
1164+
'COMMAND_FAILED',
1165+
'Android snapshot helper returned insufficient foreground app content',
1166+
{ androidSnapshotHelperFailureReason: 'content-poor-app-window', retriable: true },
1167+
);
1168+
});
1169+
1170+
const action = {
1171+
ts: 0,
1172+
command: 'click',
1173+
positionals: ['label="Save"'],
1174+
flags: {},
1175+
result: { selectorChain: ['label="Save"', 'id="save"'] },
1176+
};
1177+
1178+
const observation = await captureDivergenceObservation({
1179+
session: sessionStore.get(sessionName)!,
1180+
sessionName,
1181+
sessionStore,
1182+
logPath: path.join(root, 'daemon.log'),
1183+
action,
1184+
retryLaunchRace: true,
1185+
});
1186+
1187+
expect(observation.state).toBe('unavailable');
1188+
// 1 initial attempt + 2 retries: the 3rd retry's pre-sleep remaining-budget
1189+
// check sees the deadline already passed (3 x 5s = 15s > 12s) and breaks
1190+
// BEFORE a 4th capture — not the 8 attempts the delay array alone permits.
1191+
expect(mockDispatchCommand).toHaveBeenCalledTimes(3);
1192+
} finally {
1193+
vi.useRealTimers();
1194+
}
1195+
});

0 commit comments

Comments
 (0)