Skip to content

Commit a78b6bb

Browse files
authored
feat(wait): carry a per-poll timeline in timeout failures (#2344)
* feat(wait): carry a per-poll timeline in timeout failures A wait timeout said `reason`, `readableCaptures`, and `waitedMs`, so a failure could not say where its budget went: the runs behind #2343 spent a 10s budget on one poll (5.8s runner findText on a fresh app, 3.4s of target discovery, a fallback cancelled at the deadline) and reported the same `wait_capture_stalled` as a dead runner. The failure details now carry `captures` and `polls[]`, one entry per poll with `startedMs` on the wait's own clock, `durationMs`, and a typed outcome (readable, unreadable, deadline, runner-restart), next to the unchanged reason and the request-log link. Long waits keep the first five and last twenty-five polls so the response stays compact. * docs(wait): name the polling timeout paths that carry the poll timeline Review follow-up: wait --stable uses its own error builder and a never-readable strict absence preserves its predicate failure, so the timeline is documented for the polling timeout paths that emit it. * feat(wait): carry the poll evidence on the replay landmark-mismatch refusal Review follow-up. A replayed selector wait refused for a recorded landmark mismatch threw without the captures/polls evidence, and when its final poll ended in a runner restart the refusal hid that outcome. The refusal now carries the same failure evidence a timeout does, next to its mismatch details; two regressions cover a mismatch followed by a deadline-cancelled capture and by a runner restart. Docs and changelog name the refusal alongside the polling timeout paths.
1 parent ff59309 commit a78b6bb

6 files changed

Lines changed: 195 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22

33
## Unreleased
44

5+
- Added: polling `wait` timeouts (`wait <selector>`, `wait text`, `wait @ref`, and `wait absent`
6+
after a readable capture) carry a per-poll timeline in `error.details` (`captures`, `polls[]`
7+
with `startedMs`, `durationMs`, and a typed `outcome`: readable, unreadable, deadline,
8+
runner-restart) next to the unchanged `reason`, so a failure says where its budget went without
9+
opening the request log. Long waits keep the first five and last twenty-five polls. The replay
10+
landmark-mismatch refusal carries the same poll evidence next to its mismatch details; `wait
11+
--stable` timeouts and a never-readable strict absence keep their existing diagnostics.
512
- Fixed: the iOS Simulator AX snapshot route bounds how long a capture waits for app discovery
613
and stops starting a discovery per capture. Discovery (`simctl launchctl list` through xcrun)
714
takes seconds on a loaded host; a capture now waits at most 1.5s for the one in-flight

‎src/commands/interaction/runtime/wait-polling.test.ts‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,61 @@ test('poll delay observes both runtime and command cancellation', async () => {
4141
await assert.rejects(sleeping, reason);
4242
}
4343
});
44+
45+
test('failure evidence carries every poll on the wait clock with its typed outcome', async () => {
46+
let currentMs = 0;
47+
const runtime = {
48+
clock: { now: () => currentMs, sleep: async (_durationMs: number) => {} },
49+
} as AgentDeviceRuntime;
50+
const unreadable = new Error('content verdict');
51+
const polling = createWaitPolling(runtime, {}, 60, SELECTOR_PIPELINE_POLICIES.wait, {
52+
isUnreadableError: (error) => error === unreadable,
53+
});
54+
55+
await polling.capture(async () => {
56+
currentMs += 20;
57+
return 'seen';
58+
});
59+
await polling.capture(async () => {
60+
currentMs += 10;
61+
throw unreadable;
62+
});
63+
// The third capture outlives the remaining real-time budget: the deadline cancels it.
64+
const last = await polling.capture(async () => {
65+
await new Promise((resolve) => setTimeout(resolve, 120));
66+
currentMs += 40;
67+
return 'late';
68+
});
69+
70+
assert.equal(last.timedOut, true);
71+
const evidence = polling.failureEvidence();
72+
assert.equal(evidence.captures, 3);
73+
assert.equal(evidence.readableCaptures, 1);
74+
assert.deepEqual(evidence.polls, [
75+
{ startedMs: 0, durationMs: 20, outcome: 'readable' },
76+
{ startedMs: 20, durationMs: 10, outcome: 'unreadable' },
77+
{ startedMs: 30, durationMs: 40, outcome: 'deadline' },
78+
]);
79+
});
80+
81+
test('a long wait keeps its first polls and its last polls in the failure evidence', async () => {
82+
let currentMs = 0;
83+
const runtime = {
84+
clock: { now: () => currentMs, sleep: async (_durationMs: number) => {} },
85+
} as AgentDeviceRuntime;
86+
const polling = createWaitPolling(runtime, {}, 100_000, SELECTOR_PIPELINE_POLICIES.wait);
87+
for (let index = 0; index < 40; index += 1) {
88+
await polling.capture(async () => {
89+
currentMs += 100;
90+
return index;
91+
});
92+
}
93+
94+
const evidence = polling.failureEvidence();
95+
assert.equal(evidence.captures, 40);
96+
assert.equal(evidence.polls.length, 30);
97+
assert.equal(evidence.polls[0]?.startedMs, 0);
98+
assert.equal(evidence.polls[4]?.startedMs, 400);
99+
assert.equal(evidence.polls[5]?.startedMs, 1_500);
100+
assert.equal(evidence.polls.at(-1)?.startedMs, 3_900);
101+
});

‎src/commands/interaction/runtime/wait-polling.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,30 @@ export const DEFAULT_WAIT_TIMEOUT_MS = SELECTOR_PIPELINE_POLICIES.wait.poll.defa
1717

1818
export type WaitPollDeadline = 'capture-stalled' | 'capture-truncated' | 'runner-restart-exhausted';
1919

20+
/**
21+
* How one poll ended: a readable capture, an unreadable content verdict the wait rode out, the
22+
* deadline cancelling the capture in flight, or that cancellation carrying runner-restart
23+
* evidence. Whether a readable capture matched is the caller's verdict, not the poll's.
24+
*/
25+
export type WaitPollOutcome = 'readable' | 'unreadable' | 'deadline' | 'runner-restart';
26+
27+
/** One poll on the wait's own clock: when it started after the wait began and how long it ran. */
28+
export type WaitPollRecord = {
29+
startedMs: number;
30+
durationMs: number;
31+
outcome: WaitPollOutcome;
32+
};
33+
34+
/** Keeps a long wait's failure compact: the first polls carry the cold-start cost, the last the end. */
35+
const WAIT_POLL_TIMELINE_HEAD = 5;
36+
const WAIT_POLL_TIMELINE_TAIL = 25;
37+
2038
export type WaitFailureEvidence = {
2139
timeoutMs: number;
2240
readableCaptures: number;
41+
/** Every poll attempted, readable or not. */
42+
captures: number;
43+
polls: WaitPollRecord[];
2344
waitedMs: number;
2445
runnerRestarted?: true;
2546
runnerRestartReason?: string;
@@ -77,12 +98,16 @@ export function createWaitPolling(
7798
const timeoutMs = requestedTimeoutMs ?? budget.defaultTimeoutMs;
7899
const startedAtMs = now(runtime);
79100
const unreadable = createUnreadablePollTracker(classification.isUnreadableError);
101+
const polls: WaitPollRecord[] = [];
80102
let timeoutEvidence: Partial<WaitFailureEvidence> = {};
81103
const remainingMs = () => Math.max(0, timeoutMs - (now(runtime) - startedAtMs));
82104

83105
return {
84106
capture: async <T>(capture: (signal: AbortSignal) => Promise<T>) => {
85107
let captureWasReadable = false;
108+
const startedMs = now(runtime) - startedAtMs;
109+
const recordPoll = (outcome: WaitPollOutcome) =>
110+
polls.push({ startedMs, durationMs: now(runtime) - startedAtMs - startedMs, outcome });
86111
const result = await runWithinWaitDeadline(
87112
runtime,
88113
options,
@@ -96,9 +121,11 @@ export function createWaitPolling(
96121
);
97122
if (!result.timedOut) {
98123
if (captureWasReadable) unreadable.recordReadableCapture();
124+
recordPoll(captureWasReadable ? 'readable' : 'unreadable');
99125
return result;
100126
}
101127
const runnerRestart = runnerRestartTimeoutEvidence(result.error);
128+
recordPoll(runnerRestart ? 'runner-restart' : 'deadline');
102129
timeoutEvidence = runnerRestart ?? {};
103130
// A capture that only becomes readable after its deadline is not evidence for this wait.
104131
// Count only captures that completed before runWithinWaitDeadline returned a timeout.
@@ -119,6 +146,8 @@ export function createWaitPolling(
119146
failureEvidence: (): WaitFailureEvidence => ({
120147
timeoutMs,
121148
readableCaptures: unreadable.readableCaptures(),
149+
captures: polls.length,
150+
polls: compactPollTimeline(polls),
122151
waitedMs: now(runtime) - startedAtMs,
123152
...timeoutEvidence,
124153
}),
@@ -131,6 +160,11 @@ export function createWaitPolling(
131160
};
132161
}
133162

163+
function compactPollTimeline(polls: readonly WaitPollRecord[]): WaitPollRecord[] {
164+
if (polls.length <= WAIT_POLL_TIMELINE_HEAD + WAIT_POLL_TIMELINE_TAIL) return [...polls];
165+
return [...polls.slice(0, WAIT_POLL_TIMELINE_HEAD), ...polls.slice(-WAIT_POLL_TIMELINE_TAIL)];
166+
}
167+
134168
function waitCaptureStalledError(message: string, evidence: WaitFailureEvidence): AppError {
135169
return new AppError('COMMAND_FAILED', message, {
136170
reason: WAIT_REASONS.captureStalled,

‎src/commands/interaction/runtime/wait-selector.test.ts‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,98 @@ test('runtime wait fails closed at the deadline when only impostors matched the
207207
assert.equal(observed.label, 'Screen X');
208208
const ancestry = error.details?.observedAncestry as Array<{ role: string; label?: string }>;
209209
assert.equal(ancestry[0]?.label, 'List Screen');
210+
assertReadablePollEvidence(error);
211+
});
212+
213+
/** The refusal carries the same poll evidence a plain timeout would. */
214+
function assertReadablePollEvidence(error: AppError): void {
215+
const details = error.details ?? {};
216+
const polls = details.polls as Array<{ outcome: string }>;
217+
assert.ok(polls.length >= 1);
218+
assert.ok(polls.every((poll) => poll.outcome === 'readable'));
219+
assert.equal(details.captures, polls.length);
220+
assert.equal(details.readableCaptures, polls.length);
221+
assert.equal(typeof details.waitedMs, 'number');
222+
}
223+
224+
/**
225+
* An impostor capture, then a capture that outlives the deadline: the refusal still names the
226+
* landmark mismatch, and its poll timeline shows the deadline cutting the last capture short,
227+
* with the runner-restart evidence that capture carried.
228+
*/
229+
async function landmarkRefusalAfter(
230+
finalCapture: () => Promise<ReturnType<typeof landmarkScreen>>,
231+
): Promise<AppError> {
232+
const recorded = recordedLandmarkFor(landmarkScreen('Detail Screen'));
233+
let call = 0;
234+
const impostor = landmarkScreen('List Screen');
235+
const device = createAgentDevice({
236+
backend: {
237+
platform: 'ios',
238+
captureSnapshot: async () => {
239+
call += 1;
240+
if (call === 1) return { snapshot: impostor };
241+
return { snapshot: await finalCapture() };
242+
},
243+
} satisfies AgentDeviceBackend,
244+
artifacts: createLocalArtifactAdapter(),
245+
sessions: createMemorySessionStore([{ name: 'default', snapshot: impostor }]),
246+
policy: localCommandPolicy(),
247+
clock: createFakeClock(100),
248+
});
249+
const error = await device.selectors
250+
.wait({
251+
session: 'default',
252+
target: {
253+
kind: 'selector',
254+
selector: 'label="Screen X"',
255+
timeoutMs: 400,
256+
recordedLandmark: recorded,
257+
},
258+
})
259+
.then(
260+
() => undefined,
261+
(error: unknown) => error,
262+
);
263+
assert.ok(error instanceof AppError);
264+
assert.equal(error.details?.reason, WAIT_LANDMARK_MISMATCH_REASON);
265+
return error;
266+
}
267+
268+
test('landmark refusal after a deadline-cancelled capture keeps the poll timeline', async () => {
269+
const error = await landmarkRefusalAfter(async () => {
270+
await new Promise((resolve) => setTimeout(resolve, 600));
271+
return landmarkScreen('List Screen');
272+
});
273+
274+
const polls = error.details?.polls as Array<{ outcome: string }>;
275+
assert.deepEqual(
276+
polls.map((poll) => poll.outcome),
277+
['readable', 'deadline'],
278+
);
279+
assert.equal(error.details?.readableCaptures, 1);
280+
assert.equal(error.details?.captures, 2);
281+
});
282+
283+
test('landmark refusal after a runner restart keeps the restart outcome', async () => {
284+
const error = await landmarkRefusalAfter(async () => {
285+
await new Promise((resolve) => setTimeout(resolve, 600));
286+
throw new AppError('COMMAND_FAILED', 'runner restarted', {
287+
runnerRestarted: true,
288+
runnerRestartReason: 'runner_readiness_preflight_failed_before_command_send',
289+
});
290+
});
291+
292+
const polls = error.details?.polls as Array<{ outcome: string }>;
293+
assert.deepEqual(
294+
polls.map((poll) => poll.outcome),
295+
['readable', 'runner-restart'],
296+
);
297+
assert.equal(error.details?.runnerRestarted, true);
298+
assert.equal(
299+
error.details?.runnerRestartReason,
300+
'runner_readiness_preflight_failed_before_command_send',
301+
);
210302
});
211303

212304
test('runtime wait with a recorded landmark keeps the plain timeout when the selector never matched', async () => {

‎src/commands/interaction/runtime/wait-selector.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,10 +119,12 @@ export async function waitForSelector<Runtime extends SelectorWaitRuntime>(
119119
await polling.sleepUntilNextPoll();
120120
}
121121
if (deadline !== 'capture-stalled' && landmarkMismatch) {
122+
// The refusal keeps the poll evidence a plain timeout would carry, including a runner
123+
// restart on the final poll: the mismatch is the verdict, not the whole story of the wait.
122124
throw new AppError(
123125
'COMMAND_FAILED',
124126
`wait matched selector ${selectorExpression} but no candidate carried the recorded landmark identity`,
125-
{ reason: WAIT_LANDMARK_MISMATCH_REASON, ...landmarkMismatch },
127+
{ reason: WAIT_LANDMARK_MISMATCH_REASON, ...polling.failureEvidence(), ...landmarkMismatch },
126128
);
127129
}
128130
throw waitTimeoutError(`wait timed out for selector: ${selectorExpression}`, polling, deadline);

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,7 @@ agent-device alert dismiss
419419
- Because `wait @ref` is text-based after resolution, duplicate labels can match a different element than the original ref target.
420420
- `wait` shares the selector/snapshot resolution flow used by `click`, `fill`, `get`, and `is`.
421421
- Wait failures carry a structured `error.details.reason` in `--json` output: `wait_target_absent` proves a positive wait never found a match; `wait_target_present` means strict `wait absent` reached its deadline with valid captures that still contained matches; `predicate_failed` means strict `wait absent` could not prove absence because no valid capture arrived, with the final observation/diagnostic preserved; `wait_capture_stalled` means no readable capture arrived and is retriable; `wait_deadline_exceeded` means a later capture consumed the remaining budget after an earlier readable capture; `wait_landmark_identity_mismatch` is a replay destination-guard refusal; and `wait_stable_timeout` means the UI did not settle. Use `readableCaptures`, `waitedMs`, `matches`, and `firstMatch` instead of parsing error text. `firstMatch` carries identity/text evidence only; absence failures do not claim visibility or rect evidence.
422+
- Polling wait timeouts (`wait <selector>`, `wait text`, `wait @ref`, and `wait absent` once a readable capture has been seen) also carry `captures` (every poll attempted), `readableCaptures`, and `polls`, one entry per poll with `startedMs` on the wait's own clock, `durationMs`, and `outcome` (`readable`, `unreadable`, `deadline`, or `runner-restart`), so a timeout says where its budget went; long waits keep the first five and last twenty-five polls. A replayed selector wait refused for a recorded landmark mismatch (`wait_landmark_identity_mismatch`) carries the same poll evidence next to its mismatch details. `wait --stable` timeouts and a never-readable strict absence keep their own diagnostics. `logPath` links the full request log.
422423
- `alert` inspects or handles system alerts on iOS simulator, macOS desktop, and Android native/runtime permission dialogs.
423424
- `alert` without an action is equivalent to `alert get`.
424425
- `accept` and `dismiss` are sent once on every platform. A lost or unconfirmed response is reported as an error and never replayed; run `alert get` before acting again.

0 commit comments

Comments
 (0)