Skip to content

Commit b9b410a

Browse files
authored
feat(interaction): report readiness wait on a successful press, click, or longpress (#3081)
* feat(interaction): report the readiness wait on a successful press, click, or longpress A step that waited for its target and then acted now returns data.readiness with polls and waitedMs, present only when more than one capture was needed. The runtime selector poll reports the wait on the resolved target, the shared interaction response builder attaches it without touching the warning, and a replay step's pre-dispatch gate hands its own wait to the dispatch through internal.replayReadinessWait so one step reports one combined readiness. Tests and the mutation that fails each: - press-target-readiness.test.ts "third poll" (provider scenario, polls 3): resolution.ts drops `readiness = ready.readiness`. - press-target-readiness.test.ts "first capture hits" and selector-readiness.test.ts "first capture ... no readiness": the `observed.polls.length > 1` guard becomes `> 0`. - selector-readiness.test.ts "polls until the target appears": same drop as the third-poll scenario. - interaction-touch-response.test.ts carried-wait cases: the builder stops combining `params.carriedReadiness`; the first case also pins the prior warning staying composed. - session-replay-target-verification-runtime.test.ts gate cases: the adapter stops passing gateWait to applyReplayReadinessWait; or the gate records a wait at `polls > 0`. * docs: document data.readiness on a press that had to wait * refactor(replay): drop the carried gate wait; replay evidence stays the readiness diagnostic * test(interaction): pin data.readiness on the client and MCP success paths
1 parent 06d3e81 commit b9b410a

11 files changed

Lines changed: 157 additions & 3 deletions

File tree

‎packages/contracts/src/interaction.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,8 +181,18 @@ export type ResolvedInteractionTarget =
181181
resolution?: ResolutionDisclosure;
182182
recordingTarget?: RecordingTargetOverride;
183183
preAction?: SurfaceScopedNodes;
184+
readiness?: ReadinessWaitEvidence;
184185
};
185186

187+
/**
188+
* The wait a successful press/click/longpress spent before its target appeared. Reported only when
189+
* more than one capture was needed, so its presence means the step would have failed without the wait.
190+
*/
191+
export type ReadinessWaitEvidence = {
192+
polls: number;
193+
waitedMs: number;
194+
};
195+
186196
/**
187197
* A post-action capture that describes a DIFFERENT surface than the pre-action baseline (#2438): an
188198
* in-place iOS system surface (a web sign-in or Apple Pay sheet, hosted out of the app's process)
@@ -354,6 +364,7 @@ type TouchResponseDataBase = {
354364
evidence?: InteractionEvidence;
355365
settle?: SettleObservation;
356366
resolution?: ResolutionDisclosure;
367+
readiness?: ReadinessWaitEvidence;
357368
cost?: ResponseCost;
358369
/** Direct iOS Maestro coordinate-fallback signals. */
359370
maestroNonHittableCoordinateFallbackAllowed?: boolean;

‎src/__tests__/client-interactions-readiness.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,17 @@ test('readinessTimeoutMs on press/click/longpress reaches the request flags', as
1717
assert.equal(setup.calls[1]?.flags?.readinessTimeoutMs, 1_500);
1818
assert.equal(setup.calls[2]?.flags?.readinessTimeoutMs, 900);
1919
});
20+
21+
test('a press that waited returns the daemon data.readiness untouched', async () => {
22+
const readiness = { polls: 2, waitedMs: 180 };
23+
const setup = createTransport(async () => ({ ok: true, data: { ref: '@e1', readiness } }));
24+
const client = createAgentDeviceClient(setup.config, { transport: setup.transport });
25+
26+
const pressed = await client.interactions.press({ selector: 'label=Foo' });
27+
const clicked = await client.interactions.click({ selector: 'label=Foo' });
28+
const longPressed = await client.interactions.longPress({ selector: 'label=Foo' });
29+
30+
for (const result of [pressed, clicked, longPressed]) {
31+
assert.deepEqual((result as { readiness?: unknown }).readiness, readiness);
32+
}
33+
});

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { readinessScheduleFor } from '@agent-device/selectors/selector-pipeline-
55
import {
66
attemptSelectorResolution,
77
pollForSelectorReadiness,
8+
type SelectorReadinessWait,
89
selectorInteractionFailure,
910
} from './selector-readiness.ts';
1011
import {
@@ -138,6 +139,7 @@ async function resolveSelectorInteractionTarget(
138139
const selectorExpression = target.selector;
139140
let capture: InteractionSnapshot;
140141
let resolved: SelectorResolution | null;
142+
let readiness: SelectorReadinessWait | undefined;
141143
const readinessSchedule = readinessScheduleFor(params.pipeline.poll, params.readinessTimeoutMs);
142144
if (!readinessSchedule) {
143145
const attempt = await attemptSelectorResolution(runtime, options, selectorExpression, params);
@@ -162,6 +164,7 @@ async function resolveSelectorInteractionTarget(
162164
);
163165
capture = ready.capture;
164166
resolved = ready.resolved;
167+
readiness = ready.readiness;
165168
}
166169
// #1542: see the ref-target twin in ref-target-resolution.ts.
167170
const selected = resolved;
@@ -188,6 +191,7 @@ async function resolveSelectorInteractionTarget(
188191
kind: 'selector',
189192
point,
190193
target: { kind: 'selector', selector: resolved.selector },
194+
...(readiness ? { readiness } : {}),
191195
...describeResolvedInteractionNode(
192196
runtime,
193197
visibleNode,

‎src/commands/interaction/runtime/selector-readiness.test.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,12 +60,28 @@ test('runtime press with readinessTimeoutMs polls until the target appears', asy
6060

6161
assert.equal(result.kind, 'selector');
6262
assert.equal(result.node?.label, 'Continue');
63+
assert.equal(result.kind === 'selector' ? result.readiness?.polls : undefined, 2);
6364
assert.ok(
6465
captures >= 4,
6566
`expected at least 4 captures before the target appeared, got ${captures}`,
6667
);
6768
});
6869

70+
test('runtime press that resolves on the first capture reports no readiness despite a budget', async () => {
71+
const device = createInteractionDevice(makeSnapshotState([CONTINUE_BUTTON]), {
72+
clock: createFakeClock(),
73+
tap: async () => ({ ok: true }),
74+
});
75+
76+
const result = await device.interactions.press(selector('label=Continue'), {
77+
session: 'default',
78+
readinessTimeoutMs: 2_000,
79+
});
80+
81+
assert.equal(result.kind, 'selector');
82+
assert.equal('readiness' in result, false);
83+
});
84+
6985
test('runtime press caps a readinessTimeoutMs larger than the row maxTimeoutMs at 2_000ms', async () => {
7086
const device = createInteractionDevice(makeSnapshotState([]), {
7187
clock: createFakeClock(),

‎src/commands/interaction/runtime/selector-readiness.ts‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,13 @@ export type SelectorResolutionAttempt = {
4444
export type ResolvedSelectorAttempt = {
4545
capture: InteractionSnapshot;
4646
resolved: SelectorResolution;
47+
/** The wait that preceded the hit; absent when the first capture resolved. */
48+
readiness?: SelectorReadinessWait;
4749
};
4850

51+
/** What a successful readiness wait reports: present only when more than one capture was needed. */
52+
export type SelectorReadinessWait = Pick<SelectorReadinessDetails, 'polls' | 'waitedMs'>;
53+
4954
/** The readiness poll's evidence, attached to a target-not-found failure only (never to a refusal). */
5055
export type SelectorReadinessDetails = {
5156
polls: number;
@@ -272,7 +277,14 @@ export async function pollForSelectorReadiness(
272277
...(runtime.clock ? { clock: runtime.clock } : {}),
273278
phase: 'interaction_target_readiness',
274279
});
275-
if (observed.kind === 'done') return observed.result;
280+
if (observed.kind === 'done') {
281+
return observed.polls.length > 1
282+
? {
283+
...observed.result,
284+
readiness: { polls: observed.polls.length, waitedMs: observed.waitedMs },
285+
}
286+
: observed.result;
287+
}
276288
if (observed.kind === 'failed') throw withSparseReadiness(observed.error, observed);
277289
throw await readinessExhaustedFailure(runtime, selectorExpression, params, observed);
278290
}

‎src/daemon/interaction/internal/__tests__/interaction-touch-response.test.ts‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,11 @@ import { test, expect, vi, beforeEach } from 'vitest';
22
import { attachRefs } from '@agent-device/kernel/snapshot';
33
import { makeSessionStore } from '../../../../__tests__/test-utils/store-factory.ts';
44
import { handleInteractionCommands } from '../../index.ts';
5-
import { transformTouchResponseData } from '../interaction-touch-response.ts';
5+
import {
6+
buildInteractionResponseData,
7+
transformTouchResponseData,
8+
} from '../interaction-touch-response.ts';
9+
import type { PressCommandResult } from '@agent-device/contracts/interaction';
610
import {
711
getRuntimeBindings,
812
mockFillPoint,
@@ -82,6 +86,41 @@ test('commandless longpress response omits internal interaction diagnostics', ()
8286
).toEqual({ completedSteps: 1 });
8387
});
8488

89+
function selectorResult(readiness?: { polls: number; waitedMs: number }): PressCommandResult {
90+
return {
91+
kind: 'selector',
92+
point: { x: 10, y: 20 },
93+
target: { kind: 'selector', selector: 'label=Continue' },
94+
node: { ref: 'e1', index: 0, type: 'Button', label: 'Continue' },
95+
selectorChain: ['label=Continue'],
96+
warning: 'earlier warning',
97+
...(readiness ? { readiness } : {}),
98+
};
99+
}
100+
101+
const WAITED_SELECTOR_RESULT = selectorResult({ polls: 3, waitedMs: 400 });
102+
103+
test('the response builder reports the resolved wait and keeps the prior warning', () => {
104+
const { responseData, result } = buildInteractionResponseData({
105+
source: { kind: 'runtime', result: WAITED_SELECTOR_RESULT },
106+
referenceFrame: undefined,
107+
staleRefsWarning: 'stale refs',
108+
});
109+
110+
expect(responseData.readiness).toEqual({ polls: 3, waitedMs: 400 });
111+
expect(responseData.warning).toBe('earlier warning stale refs');
112+
expect(result.readiness).toBeUndefined();
113+
});
114+
115+
test('the response builder omits readiness when nothing waited', () => {
116+
const { responseData } = buildInteractionResponseData({
117+
source: { kind: 'runtime', result: selectorResult() },
118+
referenceFrame: undefined,
119+
});
120+
121+
expect('readiness' in responseData).toBe(false);
122+
});
123+
85124
test('press @ref --verify surfaces evidence through the interactionResultExtra allowlist', async () => {
86125
const sessionStore = makeSessionStore();
87126
const sessionName = 'verify-press';

‎src/daemon/interaction/internal/interaction-touch-response.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,7 @@ export function buildInteractionResponseData(params: {
234234
visualization.warning = warning;
235235
responseData.warning = warning;
236236
}
237+
if ('readiness' in result && result.readiness) responseData.readiness = result.readiness;
237238
return { result: visualization, responseData, ...recordedTargetCapture(result) };
238239
}
239240

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { createAgentDeviceClient } from '../../agent-device-client.ts';
4+
import { createTransport } from '../../__tests__/client-transport-fixture.ts';
5+
import { createCommandToolExecutor } from '../command-tools.ts';
6+
7+
test('MCP press returns the daemon data.readiness in structuredContent', async () => {
8+
const readiness = { polls: 2, waitedMs: 180 };
9+
const setup = createTransport(async () => ({ ok: true, data: { ref: '@e1', readiness } }));
10+
const executor = createCommandToolExecutor({
11+
createClient: () => createAgentDeviceClient(setup.config, { transport: setup.transport }),
12+
});
13+
14+
const result = await executor.execute('press', {
15+
target: { kind: 'selector', selector: 'label=Continue' },
16+
});
17+
18+
assert.equal(result.isError, false);
19+
assert.deepEqual(result.structuredContent?.readiness, readiness);
20+
});

‎test/integration/provider-scenarios/press-target-readiness.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,41 @@ test('press waits for a selector missing on the first two captures, then taps on
142142
);
143143
});
144144

145+
test('press whose target appears on the third poll reports readiness with polls 3 on the success response', async () => {
146+
await withPressReadinessDaemon(
147+
[
148+
snapshotEntry(APPLICATION_ONLY_NODES),
149+
snapshotEntry(APPLICATION_ONLY_NODES),
150+
snapshotEntry(APPLICATION_ONLY_NODES),
151+
snapshotEntry(APPLICATION_ONLY_NODES),
152+
repeatSnapshotEntry(CONTINUE_BUTTON_NODES),
153+
tapEntry(200, 322),
154+
],
155+
async (daemon) => {
156+
const press = await daemon.callCommand('press', ['label=Continue'], {
157+
readinessTimeoutMs: 2_000,
158+
});
159+
const data = assertRpcOk(press);
160+
const readiness = data.readiness as { polls: number; waitedMs: number } | undefined;
161+
assert.equal(readiness?.polls, 3);
162+
assert.equal(typeof readiness?.waitedMs, 'number');
163+
},
164+
);
165+
});
166+
167+
test('press whose first capture hits carries no readiness field even with a readiness budget', async () => {
168+
await withPressReadinessDaemon(
169+
[snapshotEntry(CONTINUE_BUTTON_NODES), tapEntry(200, 322)],
170+
async (daemon) => {
171+
const press = await daemon.callCommand('press', ['label=Continue'], {
172+
readinessTimeoutMs: 2_000,
173+
});
174+
const data = assertRpcOk(press);
175+
assert.equal('readiness' in data, false);
176+
},
177+
);
178+
});
179+
145180
test('press fails with the standard no-match error, carrying readiness evidence, when the target never appears', async () => {
146181
await withPressReadinessDaemon([repeatSnapshotEntry(APPLICATION_ONLY_NODES)], async (daemon) => {
147182
// No fake clock is wired into this daemon-composed runtime path (AgentDeviceRuntime.clock is

‎website/docs/docs/client-api.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ await client.interactions.press({
297297
});
298298
```
299299

300-
The wait is capped at 2 seconds and covers only a target that has not appeared. When the target is still missing after the wait, the error carries `error.details.readiness` with `waitedMs`, `polls`, and `end` (`expired` or `stalled`). A capture that shows an empty accessibility tree ends the wait at once with `capture_sparse` and `readiness.end: sparse`. A covered, off-screen, or ambiguous target fails at once, and a screen that stays unreadable for the whole wait fails with its own error; neither carries `readiness`. `readinessTimeoutMs` is not an MCP tool argument and has no CLI flag.
300+
The wait is capped at 2 seconds and covers only a target that has not appeared. When the target is still missing after the wait, the error carries `error.details.readiness` with `waitedMs`, `polls`, and `end` (`expired` or `stalled`). A capture that shows an empty accessibility tree ends the wait at once with `capture_sparse` and `readiness.end: sparse`. When the command had to wait and then succeeded, the result carries `data.readiness` with `polls` and `waitedMs`. A command that found its target on the first look has no `readiness` field. A covered, off-screen, or ambiguous target fails at once, and a screen that stays unreadable for the whole wait fails with its own error; neither carries `readiness`. `readinessTimeoutMs` is not an MCP tool argument and has no CLI flag.
301301

302302
Vega OS client support is currently VVD-only and covers device discovery, app open/close, `back`, `home`, and `tvRemote`. Physical Fire TV, capture, selector, install, logging, and performance methods report unsupported for Vega targets.
303303

0 commit comments

Comments
 (0)