Skip to content

Commit 056d014

Browse files
committed
fix(wait): keep the poll timeline on a wait exhausted by retriable refusals
Review finding on #2493: a wait that spent its whole budget being refused threw the last refusal raw, so the common all-RUNNER_BUSY case carried no captures, waitedMs or polls and could not show where its budget went — contradicting the evidence this PR documents. The mirror gap existed on the other exhaustion shape: when the deadline cancelled the final poll, the wait reported a generic stall and dropped the runner code and retry details instead. Both shapes now raise one error that keeps the producer's code, message, hint and retry details and carries the wait's own evidence, with reason wait_capture_stalled and the original as its cause. A content verdict is still preserved untouched, since it already describes the capture it came from, and whether it outranks the stall verdict stays the caller's policy (wait absent). Live-verified against a genuinely wedged simulator runner: COMMAND_FAILED, retriable true, runnerErrorCode RUNNER_BUSY, reason wait_capture_stalled, captures 6, readableCaptures 0, waitedMs 8041, polls retriable,retriable,retriable,retriable,retriable,deadline.
1 parent 5276cbf commit 056d014

4 files changed

Lines changed: 197 additions & 35 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,10 @@
1919
caller to retry. A poll refused with a failure its producer marked retriable is now ridden out
2020
like an unreadable capture: the wait keeps polling to its deadline, records the poll as
2121
`retriable` in its timeout evidence, and surfaces the refusal only if no readable capture ever
22-
completed. A wedged runner (`RUNNER_WEDGED`) is not retriable and still ends the wait at once.
22+
completed. That surfaced refusal keeps the producer's code, message and retry details and now
23+
also carries the wait's own `captures`, `readableCaptures`, `waitedMs` and `polls`, so a budget
24+
spent entirely on refusals is distinguishable from one immediate refusal. A wedged runner
25+
(`RUNNER_WEDGED`) is not retriable and still ends the wait at once.
2326
- Fixed: a runner failure recovered from the lifecycle journal after its transport response was
2427
lost is now classified exactly like the same failure on a live response. `RUNNER_BUSY` reached
2528
callers as a bare `RUNNER_BUSY` wire code without the `retriable` flag on that path, while the

‎src/commands/interaction/runtime/wait-polling.test.ts‎

Lines changed: 105 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,14 +145,20 @@ test('a retriable producer refusal is ridden out and named in the poll timeline'
145145
]);
146146
});
147147

148-
test('a wait spent entirely on retriable refusals fails with the refusal, not a generic timeout', async () => {
148+
/**
149+
* The exhaustion case must stay distinguishable from a single immediate refusal — that difference
150+
* is the whole point of riding the refusal out — so the wait's own timeline has to survive onto the
151+
* error alongside the producer's code, message and retry details.
152+
*/
153+
test('a wait spent entirely on retriable refusals keeps both the refusal and its poll timeline', async () => {
149154
let currentMs = 0;
150155
const runtime = {
151156
clock: { now: () => currentMs, sleep: async (_durationMs: number) => {} },
152157
} as AgentDeviceRuntime;
153158
const busy = new AppError('COMMAND_FAILED', 'runner is still finishing a previous command', {
154159
runnerErrorCode: 'RUNNER_BUSY',
155160
retriable: true,
161+
hint: 'Wait a few seconds and retry.',
156162
});
157163
const polling = createWaitPolling(runtime, {}, 10_000, SELECTOR_PIPELINE_POLICIES.wait);
158164

@@ -165,7 +171,104 @@ test('a wait spent entirely on retriable refusals fails with the refusal, not a
165171

166172
assert.throws(
167173
() => waitTimeoutError('wait timed out for text: Jump to form', polling, undefined),
168-
busy,
174+
(error: unknown) => {
175+
assert.ok(error instanceof AppError);
176+
assert.equal(error.cause, busy);
177+
const details = error.details ?? {};
178+
assert.deepEqual(
179+
{
180+
code: error.code,
181+
message: error.message,
182+
runnerErrorCode: details.runnerErrorCode,
183+
retriable: details.retriable,
184+
hint: details.hint,
185+
reason: details.reason,
186+
captures: details.captures,
187+
readableCaptures: details.readableCaptures,
188+
waitedMs: details.waitedMs,
189+
timeoutMs: details.timeoutMs,
190+
polls: details.polls,
191+
},
192+
{
193+
code: busy.code,
194+
message: busy.message,
195+
runnerErrorCode: 'RUNNER_BUSY',
196+
retriable: true,
197+
hint: 'Wait a few seconds and retry.',
198+
reason: 'wait_capture_stalled',
199+
captures: 3,
200+
readableCaptures: 0,
201+
waitedMs: 300,
202+
timeoutMs: 10_000,
203+
polls: [
204+
{ startedMs: 0, durationMs: 100, outcome: 'retriable' },
205+
{ startedMs: 100, durationMs: 100, outcome: 'retriable' },
206+
{ startedMs: 200, durationMs: 100, outcome: 'retriable' },
207+
],
208+
},
209+
);
210+
return true;
211+
},
212+
);
213+
});
214+
215+
test('a content verdict that never became readable is preserved exactly as its producer wrote it', async () => {
216+
let currentMs = 0;
217+
const runtime = {
218+
clock: { now: () => currentMs, sleep: async (_durationMs: number) => {} },
219+
} as AgentDeviceRuntime;
220+
const unreadable = new AppError('COMMAND_FAILED', 'capture was unreadable', {
221+
observation: 'unreadable',
222+
});
223+
const polling = createWaitPolling(runtime, {}, 10_000, SELECTOR_PIPELINE_POLICIES.wait, {
224+
isUnreadableError: (error) => error === unreadable,
225+
});
226+
227+
await polling.capture(async () => {
228+
currentMs += 100;
229+
throw unreadable;
230+
});
231+
232+
assert.throws(
233+
() => waitTimeoutError('wait timed out', polling, undefined),
234+
(error: unknown) => error === unreadable,
235+
);
236+
});
237+
238+
test('a deadline-cancelled poll after only refusals still reports the refusal with its timeline', async () => {
239+
let currentMs = 0;
240+
const runtime = {
241+
clock: { now: () => currentMs, sleep: async (_durationMs: number) => {} },
242+
} as AgentDeviceRuntime;
243+
const busy = new AppError('COMMAND_FAILED', 'runner is still finishing a previous command', {
244+
runnerErrorCode: 'RUNNER_BUSY',
245+
retriable: true,
246+
});
247+
const polling = createWaitPolling(runtime, {}, 60, SELECTOR_PIPELINE_POLICIES.wait);
248+
249+
await polling.capture(async () => {
250+
currentMs += 20;
251+
throw busy;
252+
});
253+
// The last poll outlives the remaining real-time budget, so the deadline cancels it and the wait
254+
// ends on `capture-stalled` rather than on an exhausted poll loop.
255+
const last = await polling.capture(async () => {
256+
await new Promise((resolve) => setTimeout(resolve, 120));
257+
currentMs += 40;
258+
return 'late';
259+
});
260+
assert.equal(last.timedOut, true);
261+
262+
assert.throws(
263+
() => waitTimeoutError('wait timed out', polling, last.timedOut ? last.deadline : undefined),
264+
(error: unknown) => {
265+
assert.ok(error instanceof AppError);
266+
assert.equal(error.details?.runnerErrorCode, 'RUNNER_BUSY');
267+
assert.equal(error.details?.retriable, true);
268+
assert.equal(error.details?.readableCaptures, 0);
269+
assert.equal(error.details?.captures, 2);
270+
return true;
271+
},
169272
);
170273
});
171274

‎src/commands/interaction/runtime/wait-polling.ts‎

Lines changed: 87 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { AppError, type AppErrorDetails } from '@agent-device/kernel/errors';
1+
import { AppError, asAppError, type AppErrorDetails } from '@agent-device/kernel/errors';
22
import { WAIT_REASONS, type WaitReason } from '@agent-device/contracts/wait';
33
import { isUnreadableCaptureContentError } from '@agent-device/contracts/android-snapshot-quality';
44
import { selectorPollBudget } from '@agent-device/selectors/selector-pipeline';
@@ -75,19 +75,26 @@ export type WaitPollingClassification = {
7575
preserveUnreadableOnStall?: boolean;
7676
};
7777

78+
/** The failure a poll hit instead of an observation, and how the tracker classified it. */
79+
type UnobservedPollCause = {
80+
outcome: Extract<WaitPollOutcome, 'unreadable' | 'retriable'>;
81+
error: unknown;
82+
};
83+
7884
type UnobservedPollTracker = {
7985
attempt: <T>(capture: () => Promise<T>) => Promise<T | undefined>;
80-
/** Why the most recent poll produced no observation, for that poll's timeline entry. */
81-
lastUnobservedOutcome: () => Extract<WaitPollOutcome, 'unreadable' | 'retriable'> | undefined;
86+
/** The last poll's cause — also names that poll's timeline outcome. */
87+
lastUnobservedCause: () => UnobservedPollCause | undefined;
8288
recordReadableCapture: () => void;
8389
readableCaptures: () => number;
84-
rethrowIfNeverReadable: () => void;
90+
/** The cause every poll hit, or undefined once any capture was readable. */
91+
neverReadableCause: () => UnobservedPollCause | undefined;
8592
};
8693

8794
type WaitFailurePolling = {
8895
failureEvidence: () => WaitFailureEvidence;
8996
preserveUnreadableOnStall?: boolean;
90-
rethrowIfNeverReadable: () => void;
97+
neverReadableCause: () => UnobservedPollCause | undefined;
9198
};
9299

93100
/**
@@ -129,9 +136,7 @@ export function createWaitPolling(
129136
);
130137
if (!result.timedOut) {
131138
if (captureWasReadable) unobserved.recordReadableCapture();
132-
recordPoll(
133-
captureWasReadable ? 'readable' : (unobserved.lastUnobservedOutcome() ?? 'unreadable'),
134-
);
139+
recordPoll(completedPollOutcome(captureWasReadable, unobserved.lastUnobservedCause()));
135140
return result;
136141
}
137142
const runnerRestart = runnerRestartTimeoutEvidence(result.error);
@@ -141,15 +146,7 @@ export function createWaitPolling(
141146
// Count only captures that completed before runWithinWaitDeadline returned a timeout.
142147
return {
143148
timedOut: true as const,
144-
// A poll is a backend stall when no completed capture established a readable observation;
145-
// the poll index is not evidence. This remains true after one or more unreadable content
146-
// verdicts followed by a capture that consumes the remaining budget.
147-
deadline:
148-
runnerRestart !== undefined
149-
? ('runner-restart-exhausted' as const)
150-
: unobserved.readableCaptures() === 0
151-
? ('capture-stalled' as const)
152-
: ('capture-truncated' as const),
149+
deadline: timedOutDeadline(runnerRestart !== undefined, unobserved.readableCaptures()),
153150
};
154151
},
155152
hasTimeRemaining: () => remainingMs() > 0,
@@ -162,14 +159,34 @@ export function createWaitPolling(
162159
...timeoutEvidence,
163160
}),
164161
preserveUnreadableOnStall: classification.preserveUnreadableOnStall,
165-
rethrowIfNeverReadable: unobserved.rethrowIfNeverReadable,
162+
neverReadableCause: unobserved.neverReadableCause,
166163
sleepUntilNextPoll: async () =>
167164
await sleepWithWaitCancellation(runtime, options, Math.min(budget.intervalMs, remainingMs())),
168165
timeoutMs,
169166
waitedMs: () => now(runtime) - startedAtMs,
170167
};
171168
}
172169

170+
/** How a poll that finished inside its deadline is recorded: what it produced, or why it did not. */
171+
function completedPollOutcome(
172+
readable: boolean,
173+
cause: UnobservedPollCause | undefined,
174+
): WaitPollOutcome {
175+
if (readable) return 'readable';
176+
return cause?.outcome ?? 'unreadable';
177+
}
178+
179+
/**
180+
* Why the deadline, not the target, ended the wait. A poll is a backend stall when no completed
181+
* capture established a readable observation; the poll index is not evidence. This remains true
182+
* after one or more unreadable content verdicts followed by a capture that consumes the remaining
183+
* budget.
184+
*/
185+
function timedOutDeadline(runnerRestarted: boolean, readableCaptures: number): WaitPollDeadline {
186+
if (runnerRestarted) return 'runner-restart-exhausted';
187+
return readableCaptures === 0 ? 'capture-stalled' : 'capture-truncated';
188+
}
189+
173190
function compactPollTimeline(polls: readonly WaitPollRecord[]): WaitPollRecord[] {
174191
if (polls.length <= WAIT_POLL_TIMELINE_HEAD + WAIT_POLL_TIMELINE_TAIL) return [...polls];
175192
return [...polls.slice(0, WAIT_POLL_TIMELINE_HEAD), ...polls.slice(-WAIT_POLL_TIMELINE_TAIL)];
@@ -210,6 +227,49 @@ function waitTargetAbsentError(message: string, evidence: WaitFailureEvidence):
210227
});
211228
}
212229

230+
/**
231+
* Hands back the cause every poll hit when none produced a readable observation, because it says
232+
* more than a generic timeout. The two kinds of cause need different treatment:
233+
*
234+
* - A content verdict (sparse, truncated, unreadable) already describes the capture it came from,
235+
* so it is preserved exactly as the producer wrote it.
236+
* - A retriable refusal describes a single poll. On its own it cannot distinguish one immediate
237+
* refusal from a whole budget spent being refused — the distinction this wait exists to make —
238+
* so it is re-raised with the producer's code, message and retry details intact plus the poll
239+
* timeline that shows the exhaustion.
240+
*
241+
* `preserveContentVerdict` is the caller's policy for the first kind only (`wait absent` keeps its
242+
* typed observation ahead of the wait's own stall verdict); a refusal is preserved for every wait.
243+
*/
244+
function rethrowNeverReadableCause(
245+
polling: WaitFailurePolling,
246+
evidence: WaitFailureEvidence,
247+
preserveContentVerdict: boolean,
248+
): void {
249+
const cause = polling.neverReadableCause();
250+
if (cause === undefined) return;
251+
// A refusal names a condition the caller can act on, so it is preserved however the budget ran
252+
// out — polled away between refusals, or cancelled mid-capture by the deadline.
253+
if (cause.outcome === 'retriable') throw waitRetriableRefusalError(cause.error, evidence);
254+
if (preserveContentVerdict) throw cause.error;
255+
}
256+
257+
function waitRetriableRefusalError(cause: unknown, evidence: WaitFailureEvidence): AppError {
258+
const refusal = asAppError(cause);
259+
return new AppError(
260+
refusal.code,
261+
refusal.message,
262+
{
263+
// `wait_capture_stalled` by definition: no readable capture established an observation
264+
// before the timeout. A reason the producer already set is more specific, so it wins.
265+
reason: WAIT_REASONS.captureStalled satisfies WaitReason,
266+
...refusal.details,
267+
...evidence,
268+
},
269+
refusal,
270+
);
271+
}
272+
213273
export function waitTimeoutError(
214274
message: string,
215275
polling: WaitFailurePolling,
@@ -220,12 +280,14 @@ export function waitTimeoutError(
220280
return waitRunnerRestartExhaustedError(message, evidence);
221281
}
222282
if (deadline === 'capture-stalled') {
223-
if (polling.preserveUnreadableOnStall) polling.rethrowIfNeverReadable();
283+
// Whether a content verdict outranks the stall verdict is the caller's policy; a refusal is
284+
// preserved either way.
285+
rethrowNeverReadableCause(polling, evidence, polling.preserveUnreadableOnStall === true);
224286
return waitCaptureStalledError(message, evidence);
225287
}
226288
if (deadline === 'capture-truncated') return waitDeadlineExceededError(message, evidence);
227289

228-
polling.rethrowIfNeverReadable();
290+
rethrowNeverReadableCause(polling, evidence, true);
229291
return evidence.readableCaptures === 0
230292
? waitCaptureStalledError(message, evidence)
231293
: waitTargetAbsentError(message, evidence);
@@ -278,30 +340,24 @@ function createUnobservedPollTracker(
278340
isUnreadableError: (error: unknown) => boolean = isUnreadableCaptureContentError,
279341
): UnobservedPollTracker {
280342
let readableCaptureCount = 0;
281-
let lastUnobservedError: unknown;
282-
let lastUnobservedOutcome: Extract<WaitPollOutcome, 'unreadable' | 'retriable'> | undefined;
343+
let lastCause: UnobservedPollCause | undefined;
283344
return {
284345
attempt: async <T>(capture: () => Promise<T>): Promise<T | undefined> => {
285346
try {
286347
return await capture();
287348
} catch (error) {
288-
if (isUnreadableError(error)) lastUnobservedOutcome = 'unreadable';
289-
else if (isRetriablePollFailure(error)) lastUnobservedOutcome = 'retriable';
349+
if (isUnreadableError(error)) lastCause = { outcome: 'unreadable', error };
350+
else if (isRetriablePollFailure(error)) lastCause = { outcome: 'retriable', error };
290351
else throw error;
291-
lastUnobservedError = error;
292352
return undefined;
293353
}
294354
},
295-
lastUnobservedOutcome: () => lastUnobservedOutcome,
355+
lastUnobservedCause: () => lastCause,
296356
recordReadableCapture: () => {
297357
readableCaptureCount += 1;
298358
},
299359
readableCaptures: () => readableCaptureCount,
300-
rethrowIfNeverReadable: () => {
301-
if (readableCaptureCount === 0 && lastUnobservedError !== undefined) {
302-
throw lastUnobservedError;
303-
}
304-
},
360+
neverReadableCause: () => (readableCaptureCount === 0 ? lastCause : undefined),
305361
};
306362
}
307363

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -421,7 +421,7 @@ agent-device alert dismiss
421421
- Because `wait @ref` is text-based after resolution, duplicate labels can match a different element than the original ref target.
422422
- `wait` shares the selector/snapshot resolution flow used by `click`, `fill`, `get`, and `is`.
423423
- Wait failures carry a structured `error.details.reason` in `--json` output: `wait_target_absent` proves a positive wait never found a match; `wait_target_present` means strict `wait absent` reached its deadline with valid captures that still contained matches; `predicate_failed` means strict `wait absent` could not prove absence because no valid capture arrived, with the final observation/diagnostic preserved; `wait_capture_stalled` means no readable capture arrived and is retriable; `wait_deadline_exceeded` means a later capture consumed the remaining budget after an earlier readable capture; `wait_landmark_identity_mismatch` is a replay destination-guard refusal; and `wait_stable_timeout` means the UI did not settle. Use `readableCaptures`, `waitedMs`, `matches`, and `firstMatch` instead of parsing error text. `firstMatch` carries identity/text evidence only; absence failures do not claim visibility or rect evidence.
424-
- Polling wait timeouts (`wait <selector>`, `wait text`, `wait @ref`, and `wait absent` once a readable capture has been seen) also carry `captures` (every poll attempted), `readableCaptures`, and `polls`, one entry per poll with `startedMs` on the wait's own clock, `durationMs`, and `outcome` (`readable`, `unreadable`, `retriable` for a poll the producer refused with a failure it marked retriable, `deadline`, or `runner-restart`), so a timeout says where its budget went; long waits keep the first five and last twenty-five polls. A replayed selector wait refused for a recorded landmark mismatch (`wait_landmark_identity_mismatch`) carries the same poll evidence next to its mismatch details. `wait --stable` timeouts and a never-readable strict absence keep their own diagnostics. `logPath` links the full request log.
424+
- Polling wait timeouts (`wait <selector>`, `wait text`, `wait @ref`, and `wait absent` once a readable capture has been seen) also carry `captures` (every poll attempted), `readableCaptures`, and `polls`, one entry per poll with `startedMs` on the wait's own clock, `durationMs`, and `outcome` (`readable`, `unreadable`, `retriable` for a poll the producer refused with a failure it marked retriable, `deadline`, or `runner-restart`), so a timeout says where its budget went; long waits keep the first five and last twenty-five polls. A replayed selector wait refused for a recorded landmark mismatch (`wait_landmark_identity_mismatch`) carries the same poll evidence next to its mismatch details. A wait that never saw a readable capture reports the cause its polls hit instead of a generic timeout: a content verdict is preserved as its producer wrote it, while a refusal the producer marked retriable keeps its code, message and retry details **and** carries the poll evidence above, so an exhausted budget stays distinguishable from a single immediate refusal. `wait --stable` timeouts and a never-readable strict absence keep their own diagnostics. `logPath` links the full request log.
425425
- `alert` inspects or handles system alerts on iOS simulator, macOS desktop, and Android native/runtime permission dialogs.
426426
- `alert` without an action is equivalent to `alert get`.
427427
- `accept` and `dismiss` are sent once on every platform. A lost or unconfirmed response is reported as an error and never replayed; run `alert get` before acting again.

0 commit comments

Comments
 (0)