Skip to content

Commit 88ba8cd

Browse files
committed
fix(interaction): wait polls ride out content-unreadable captures (live-validated on Android)
Live ADR 0016 validation on a Pixel emulator showed a destination-guard wait replayed immediately after a navigation press deterministically dies: the first poll's capture lands mid-transition and the Android helper's 'insufficient foreground app content' verdict threw out of the polling loop. iOS already yields the same state as a sparse verdict with no matches, so the loop kept polling there — this makes wait semantics platform-consistent. A content-verdict capture failure (isUnreadableCaptureContentError) now counts as a no-match poll for selector and text waits; a wait whose screen never became readable rethrows the last capture verdict at the deadline, so persistent breakage keeps its diagnosis. Other capture failures still throw immediately.
1 parent 5eee73d commit 88ba8cd

4 files changed

Lines changed: 193 additions & 23 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,12 @@ A recorded `id` never matches a node without that id.
402402
> first ancestry mismatch in `mismatches`) BEFORE the wait reports success, while a selector that
403403
> never matched at all remains the ordinary wait timeout (`action-failure` — that failure needs a
404404
> state repair, not an identity repair, and its `repairHint` routing is already correct).
405+
> A wait poll also rides out a capture that judged the current screen UNREADABLE (the Android
406+
> helper's content verdicts, `isUnreadableCaptureContentError`; iOS surfaces the same state as a
407+
> sparse verdict with no matches): live Android validation showed a guard wait replayed right after
408+
> a navigation press deterministically dies on its first mid-transition capture otherwise. A wait
409+
> whose screen never became readable rethrows the last capture verdict at the deadline instead of
410+
> masking it as a generic timeout.
405411
> Record time writes evidence in a **landmark mode**: the step-5 self-check is identity-set
406412
> MEMBERSHIP rather than isolation (mirroring what replay will actually verify), and a matched node
407413
> with no id and no label after #1269 demotion records **no annotation** — a role-only landmark

‎src/commands/interaction/runtime/selector-read.test.ts‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -635,3 +635,106 @@ test('runtime wait without a recorded landmark returns the satisfying match for
635635
assert.equal(result.node?.label, 'Screen X');
636636
assert.equal(result.preActionNodes?.length, 2);
637637
});
638+
639+
// ---------------------------------------------------------------------------
640+
// Wait polls ride out captures that judged the screen unreadable (the
641+
// mid-transition Android helper content verdicts) instead of aborting the
642+
// wait — the live-validated destination-guard gap from #1349's PR review.
643+
// ---------------------------------------------------------------------------
644+
645+
function unreadableCaptureError() {
646+
return new AppError(
647+
'COMMAND_FAILED',
648+
'Android snapshot helper returned insufficient foreground app content',
649+
{
650+
androidSnapshotHelperFailureReason: 'content-poor-app-window',
651+
retriable: true,
652+
},
653+
);
654+
}
655+
656+
function waitDeviceWithCaptures(captures: Array<() => ReturnType<typeof landmarkScreen>>) {
657+
let call = 0;
658+
return createAgentDevice({
659+
backend: {
660+
platform: 'android',
661+
captureSnapshot: async () => {
662+
const produce = captures[Math.min(call, captures.length - 1)]!;
663+
call += 1;
664+
return { snapshot: produce() };
665+
},
666+
} satisfies AgentDeviceBackend,
667+
artifacts: createLocalArtifactAdapter(),
668+
sessions: createMemorySessionStore([{ name: 'default' }]),
669+
policy: localCommandPolicy(),
670+
clock: createFakeClock(),
671+
});
672+
}
673+
674+
test('runtime wait rides out an unreadable mid-transition capture and succeeds on the next poll', async () => {
675+
const device = waitDeviceWithCaptures([
676+
() => {
677+
throw unreadableCaptureError();
678+
},
679+
() => landmarkScreen('Detail Screen'),
680+
]);
681+
682+
const result = await device.selectors.wait({
683+
session: 'default',
684+
target: { kind: 'selector', selector: 'label="Screen X"', timeoutMs: 5000 },
685+
});
686+
687+
assert.equal(result.kind, 'selector');
688+
if (result.kind !== 'selector') throw new Error('unreachable');
689+
assert.equal(result.waitedMs >= 300, true);
690+
});
691+
692+
test('runtime wait rethrows the capture verdict when the screen never became readable', async () => {
693+
const device = waitDeviceWithCaptures([
694+
() => {
695+
throw unreadableCaptureError();
696+
},
697+
]);
698+
699+
await assert.rejects(
700+
device.selectors.wait({
701+
session: 'default',
702+
target: { kind: 'selector', selector: 'label="Screen X"', timeoutMs: 1000 },
703+
}),
704+
/insufficient foreground app content/,
705+
);
706+
});
707+
708+
test('runtime wait keeps the plain timeout when readable polls simply never matched', async () => {
709+
const empty = () => makeSnapshotState([{ index: 0, depth: 0, type: 'Other', label: 'Loading' }]);
710+
const device = waitDeviceWithCaptures([
711+
empty,
712+
() => {
713+
throw unreadableCaptureError();
714+
},
715+
]);
716+
717+
await assert.rejects(
718+
device.selectors.wait({
719+
session: 'default',
720+
target: { kind: 'selector', selector: 'label="Screen X"', timeoutMs: 1000 },
721+
}),
722+
/wait timed out for selector/,
723+
);
724+
});
725+
726+
test('runtime wait still fails immediately on a non-content capture failure', async () => {
727+
const device = waitDeviceWithCaptures([
728+
() => {
729+
throw new AppError('COMMAND_FAILED', 'adb device offline');
730+
},
731+
]);
732+
733+
await assert.rejects(
734+
device.selectors.wait({
735+
session: 'default',
736+
target: { kind: 'selector', selector: 'label="Screen X"', timeoutMs: 5000 },
737+
}),
738+
/adb device offline/,
739+
);
740+
});

‎src/commands/interaction/runtime/selector-read.ts‎

Lines changed: 67 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import type { SnapshotNode } from '../../../kernel/snapshot.ts';
88
import { findNodeByRef, normalizeRef } from '../../../kernel/snapshot.ts';
99
import {
1010
isSparseSnapshotQualityVerdict,
11+
isUnreadableCaptureContentError,
1112
type SnapshotQualityVerdict,
1213
} from '../../../snapshot/snapshot-quality.ts';
1314
import type { AgentDeviceRuntime, CommandContext } from '../../../runtime-contract.ts';
@@ -574,29 +575,34 @@ async function waitForSelector(
574575
// must not abort a wait whose job is to wait through it, so the loop keeps
575576
// polling; only the deadline turns this into the fail-closed refusal.
576577
let landmarkMismatch: WaitLandmarkMismatchEvidence | undefined;
578+
const unreadable = createUnreadablePollTracker();
577579
while (now(runtime) - start < timeout) {
578580
// Presence-only poll: skip scroll-hint derivation (#1270), same as waitForFindMatch.
579-
const capture = await captureSelectorSnapshot(runtime, options, {
580-
updateSession: true,
581-
includeHiddenContentHints: false,
582-
...capturePolicy,
583-
});
584-
const nodes = capture.snapshot.nodes;
585-
const matchList = listSelectorChainMatches(nodes, chain, {
586-
platform: runtime.backend.platform,
587-
});
588-
if (matchList) {
589-
const landmark = resolveLandmarkMatch(nodes, matchList, recordedLandmark);
590-
if (landmark.kind === 'satisfied') {
591-
return {
592-
kind: 'selector',
593-
selector: matchList.selector.raw,
594-
waitedMs: now(runtime) - start,
595-
node: landmark.node,
596-
preActionNodes: nodes,
597-
};
581+
const capture = await unreadable.attempt(() =>
582+
captureSelectorSnapshot(runtime, options, {
583+
updateSession: true,
584+
includeHiddenContentHints: false,
585+
...capturePolicy,
586+
}),
587+
);
588+
if (capture) {
589+
const nodes = capture.snapshot.nodes;
590+
const matchList = listSelectorChainMatches(nodes, chain, {
591+
platform: runtime.backend.platform,
592+
});
593+
if (matchList) {
594+
const landmark = resolveLandmarkMatch(nodes, matchList, recordedLandmark);
595+
if (landmark.kind === 'satisfied') {
596+
return {
597+
kind: 'selector',
598+
selector: matchList.selector.raw,
599+
waitedMs: now(runtime) - start,
600+
node: landmark.node,
601+
preActionNodes: nodes,
602+
};
603+
}
604+
landmarkMismatch = landmark.evidence;
598605
}
599-
landmarkMismatch = landmark.evidence;
600606
}
601607
await sleep(runtime, POLL_INTERVAL_MS);
602608
}
@@ -607,9 +613,43 @@ async function waitForSelector(
607613
{ reason: WAIT_LANDMARK_MISMATCH_REASON, ...landmarkMismatch },
608614
);
609615
}
616+
unreadable.rethrowIfNeverReadable();
610617
throw new AppError('COMMAND_FAILED', `wait timed out for selector: ${selectorExpression}`);
611618
}
612619

620+
/**
621+
* A wait poll rides out a capture that judged the current screen unreadable
622+
* (`isUnreadableCaptureContentError` — the mid-transition state a wait exists
623+
* to wait through; iOS surfaces the same state as a sparse verdict with no
624+
* matches). Any other capture failure still throws immediately, and a wait
625+
* whose screen NEVER became readable rethrows the last content verdict at the
626+
* deadline so persistent breakage keeps its capture diagnosis instead of a
627+
* generic timeout.
628+
*/
629+
function createUnreadablePollTracker(): {
630+
attempt: <T>(capture: () => Promise<T>) => Promise<T | undefined>;
631+
rethrowIfNeverReadable: () => void;
632+
} {
633+
let sawReadableCapture = false;
634+
let lastUnreadableError: unknown;
635+
return {
636+
attempt: async <T>(capture: () => Promise<T>): Promise<T | undefined> => {
637+
try {
638+
const result = await capture();
639+
sawReadableCapture = true;
640+
return result;
641+
} catch (error) {
642+
if (!isUnreadableCaptureContentError(error)) throw error;
643+
lastUnreadableError = error;
644+
return undefined;
645+
}
646+
},
647+
rethrowIfNeverReadable: () => {
648+
if (!sawReadableCapture && lastUnreadableError !== undefined) throw lastUnreadableError;
649+
},
650+
};
651+
}
652+
613653
type LandmarkMatchOutcome =
614654
| { kind: 'satisfied'; node: SnapshotNode }
615655
| { kind: 'identity-mismatch'; evidence: WaitLandmarkMismatchEvidence };
@@ -659,13 +699,17 @@ async function waitForText(
659699
): Promise<WaitCommandResult> {
660700
const timeout = timeoutMs ?? DEFAULT_TIMEOUT_MS;
661701
const start = now(runtime);
702+
const unreadable = createUnreadablePollTracker();
662703
while (now(runtime) - start < timeout) {
663-
const found = runtime.backend.findText
664-
? (await runtime.backend.findText(toBackendContext(runtime, options), text)).found
665-
: await snapshotContainsText(runtime, options, text);
704+
const found = await unreadable.attempt(async () =>
705+
runtime.backend.findText
706+
? (await runtime.backend.findText(toBackendContext(runtime, options), text)).found
707+
: await snapshotContainsText(runtime, options, text),
708+
);
666709
if (found) return { kind: 'text', text, waitedMs: now(runtime) - start };
667710
await sleep(runtime, POLL_INTERVAL_MS);
668711
}
712+
unreadable.rethrowIfNeverReadable();
669713
throw new AppError('COMMAND_FAILED', `wait timed out for text: ${text}`);
670714
}
671715

‎src/snapshot/snapshot-quality.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,3 +116,20 @@ function collapsedLeafWarnings(
116116
}
117117
return warnings;
118118
}
119+
120+
/**
121+
* True when a thrown capture failure is a CONTENT verdict — the capture
122+
* mechanism worked but judged the current screen unreadable (the Android
123+
* helper's content-recovery refusals, `androidSnapshotHelperFailureReason`) —
124+
* rather than a mechanism failure. A polling wait rides these out exactly
125+
* like a sparse-verdict capture that returned no match (iOS already yields a
126+
* verdict instead of throwing): a mid-transition screen is the state a wait
127+
* exists to wait through. One-shot reads keep failing loudly, and a wait
128+
* whose screen never becomes readable rethrows the capture failure at its
129+
* deadline instead of masking it as a generic timeout.
130+
*/
131+
export function isUnreadableCaptureContentError(error: unknown): boolean {
132+
if (!error || typeof error !== 'object') return false;
133+
const details = (error as { details?: Record<string, unknown> }).details;
134+
return typeof details?.androidSnapshotHelperFailureReason === 'string';
135+
}

0 commit comments

Comments
 (0)