Skip to content

Commit 0495ee1

Browse files
authored
fix(ios): mark zero-area nodes unmeasured and point their press refusal at scroll --until (callstack#4495)
* fix(ios): mark zero-area nodes unmeasured and point their press refusal at scroll --until A scroll view can keep its scrolled-out content in the tree with a zero-area frame (Flutter does on iOS). The Simulator AX bridge fold kept such a node as if it were on screen, and pressing one was refused with no recovery. The fold now marks such a node [unmeasured]. A target with no usable frame is still refused; when a scrollable sibling is listed before it, which is how Flutter publishes a scroll view and its content, the refusal names scroll --until with no direction, since the frame gives none. Fixes callstack#4451 * fix(ios): quote the selector, name snapshot -i and skip scroll bars in the unmeasured recovery The hint serializes the selector as a double-quoted argument, so one holding an apostrophe survives. The ref form names snapshot -i, as the off-screen ref hint does. A scroll bar listed before the target is no longer read as its scroll container. The help and docs say where the mark is added: every iOS snapshot presented from an acquired tree, not only the Simulator AX bridge. * fix(ios): quote the scroll --until selector for the shell and scope the unmeasured mark to regular snapshots The recovery hint wrapped the selector with JSON.stringify, and inside double quotes a shell still expands $, backticks and !. It now uses a POSIX single-quote wrap. The help and snapshots.md say the [unmeasured] mark belongs to regular snapshots: --raw returns before the fold. * fix(snapshot): mark unmeasured nodes in the shared snapshot assembly and quote scroll hints with shellQuote The unmeasured mark moves from the iOS acquired-tree fold to buildSnapshotState, next to the covered annotation, so every producer gets it: the XCTest runner, the Simulator AX bridge, the provider trees, Android and desktop. Regular snapshots only, as before. The zero-area recovery takes its selector already quoted from the command-runtime caller, which uses the kernel's shellQuote; the selectors entry cannot import it without growing its eager closure. The off-screen hint uses the same helper, so a selector with an apostrophe no longer breaks its command. scrollContainerRef is the bare ref body, like details.ref. * fix(selectors): quote the recovery selector only when needed and bar it from ref targets The zero-area recovery quotes its selector with shellQuoteIfNeeded, as the moving-target gate does, so a bare selector stays bare. The ref arm of the failure type forbids shellQuotedSelector, so a ref target cannot be handed the selector form of the hint.
1 parent f91377e commit 0495ee1

9 files changed

Lines changed: 402 additions & 9 deletions

File tree

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
import { expect, test } from 'vitest';
2+
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
3+
import type { SnapshotCaptureProvenance } from '@agent-device/contracts/snapshot-model';
4+
import { buildSnapshotState } from './snapshot-state.ts';
5+
6+
/**
7+
* A Flutter screen on iOS at the top of its content (#4451): the scroll view and its content are
8+
* siblings, and the content below the fold keeps a zero-area frame at the origin.
9+
*/
10+
function scrolledOutButtonTree(): RawSnapshotNode[] {
11+
return [
12+
{ index: 0, depth: 0, type: 'Application', rect: { x: 0, y: 0, width: 402, height: 874 } },
13+
{
14+
index: 1,
15+
depth: 1,
16+
parentIndex: 0,
17+
type: 'ScrollView',
18+
rect: { x: 0, y: 118, width: 402, height: 756 },
19+
},
20+
{
21+
index: 2,
22+
depth: 1,
23+
parentIndex: 0,
24+
type: 'StaticText',
25+
label: 'Tall block',
26+
rect: { x: 157, y: 546, width: 86, height: 27 },
27+
},
28+
{
29+
index: 3,
30+
depth: 1,
31+
parentIndex: 0,
32+
type: 'Button',
33+
label: 'View all',
34+
rect: { x: 0, y: 0, width: 0, height: 0 },
35+
},
36+
{ index: 4, depth: 1, parentIndex: 0, type: 'StaticText', label: 'No frame' },
37+
];
38+
}
39+
40+
function hintsByNode(nodes: readonly RawSnapshotNode[]) {
41+
return nodes.map((node) => [node.label ?? node.type, node.presentationHints]);
42+
}
43+
44+
const PROVENANCES: SnapshotCaptureProvenance[] = [
45+
{ backend: 'xctest', producer: 'apple-runner' },
46+
{ backend: 'android', producer: 'android-uiautomator' },
47+
];
48+
49+
test.each(PROVENANCES)(
50+
'marks a node published with a zero-area rect as unmeasured ($producer)',
51+
(provenance) => {
52+
const state = buildSnapshotState({ nodes: scrolledOutButtonTree(), ...provenance }, undefined);
53+
54+
expect(hintsByNode(state.nodes)).toEqual([
55+
['Application', undefined],
56+
['ScrollView', undefined],
57+
['Tall block', undefined],
58+
['View all', ['unmeasured']],
59+
// A node that reports no frame at all made no measurement to doubt.
60+
['No frame', undefined],
61+
]);
62+
},
63+
);
64+
65+
test('a raw snapshot keeps the reported nodes without the unmeasured mark', () => {
66+
const state = buildSnapshotState(
67+
{ nodes: scrolledOutButtonTree(), backend: 'xctest', producer: 'apple-runner' },
68+
{ snapshotRaw: true },
69+
);
70+
71+
expect(state.nodes.flatMap((node) => node.presentationHints ?? [])).toEqual([]);
72+
});

‎packages/capture-kit/src/snapshot-state.ts‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import {
2222
type SnapshotViewportSize,
2323
type SnapshotQualityNote,
2424
} from '@agent-device/kernel/snapshot';
25+
import { isPositiveFiniteRect } from '@agent-device/kernel/rect';
2526
import {
2627
snapshotRefMinting,
2728
type SnapshotBackend,
@@ -104,7 +105,9 @@ export function buildSnapshotState(
104105
const nodes = attachRefs(
105106
snapshotRaw
106107
? scopedNodes
107-
: annotateCoveredSnapshotNodes(scopedNodes, overlayOptionsForBackend(data?.backend)),
108+
: annotateUnmeasuredSnapshotNodes(
109+
annotateCoveredSnapshotNodes(scopedNodes, overlayOptionsForBackend(data?.backend)),
110+
),
108111
snapshotRefMinting(data),
109112
);
110113
const textInputGeometry = readSnapshotTextInputGeometry(data.nodes);
@@ -136,6 +139,25 @@ export function buildSnapshotState(
136139
};
137140
}
138141

142+
/**
143+
* The hint a node carries when the rect it is published with has no area. That is not evidence
144+
* that the node is off screen, but nothing confirms that it is on screen either: a scroll view
145+
* keeps its scrolled-out content in the tree with a zero-area frame (#4451), as SpringBoard keeps
146+
* another page's icons (#3841).
147+
*/
148+
const UNMEASURED_PRESENTATION_HINT = 'unmeasured';
149+
150+
function annotateUnmeasuredSnapshotNodes(nodes: RawSnapshotNode[]): RawSnapshotNode[] {
151+
return nodes.map((node) =>
152+
node.rect && !isPositiveFiniteRect(node.rect)
153+
? {
154+
...node,
155+
presentationHints: [...(node.presentationHints ?? []), UNMEASURED_PRESENTATION_HINT],
156+
}
157+
: node,
158+
);
159+
}
160+
139161
function annotateBackendReplacementSurfaces(
140162
owner: object & { backend?: SnapshotBackend },
141163
nodes: RawSnapshotNode[],

‎packages/command-runtime/src/commands/interaction/runtime/resolution-touch-point.test.ts‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import assert from 'node:assert/strict';
22
import { test } from 'vitest';
33
import type { Point } from '@agent-device/kernel/snapshot';
4+
import { attachSnapshotNodeInteractionBounds } from '@agent-device/kernel/rect-center';
5+
import { buildSnapshotState } from '@agent-device/capture-kit/snapshot-state';
46
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
57
import {
68
buildUiHierarchySnapshot,
@@ -120,12 +122,114 @@ test('runtime press refuses a selector match with a zero-area rect and dispatche
120122
assert.equal((error as { code?: string }).code, 'COMMAND_FAILED');
121123
assert.equal(details?.reason, 'target_bounds_invalid');
122124
assert.equal(details?.dispatched, 'no');
125+
// No scroll container holds the icon, so no scroll is offered as the recovery.
126+
assert.equal(details?.scrollContainerRef, undefined);
127+
assert.equal(details?.hint, undefined);
123128
return true;
124129
},
125130
);
126131
assert.deepEqual(taps, []);
127132
});
128133

134+
/**
135+
* A Flutter screen on iOS at the top of its content (#4451), as the Simulator AX bridge presents it:
136+
* the scroll view and its content are siblings, and the content below the fold keeps a 0x0 rect.
137+
*/
138+
function scrolledOutButtonSnapshot(label = 'View all') {
139+
return makeSnapshotState([
140+
{
141+
index: 0,
142+
depth: 0,
143+
type: 'Application',
144+
rect: { x: 0, y: 0, width: 402, height: 874 },
145+
hittable: false,
146+
},
147+
{
148+
index: 1,
149+
depth: 1,
150+
parentIndex: 0,
151+
type: 'ScrollView',
152+
role: 'FlutterSemanticsScrollView',
153+
rect: { x: 0, y: 118, width: 402, height: 756 },
154+
hittable: true,
155+
},
156+
{
157+
index: 2,
158+
depth: 1,
159+
parentIndex: 0,
160+
type: 'StaticText',
161+
role: 'FlutterSemanticsObject',
162+
label: 'Tall block',
163+
rect: { x: 157, y: 546, width: 86, height: 27 },
164+
hittable: true,
165+
},
166+
{
167+
index: 3,
168+
depth: 1,
169+
parentIndex: 0,
170+
type: 'Button',
171+
role: 'FlutterSemanticsObject',
172+
label,
173+
rect: { x: 0, y: 0, width: 0, height: 0 },
174+
hittable: false,
175+
},
176+
]);
177+
}
178+
179+
test('runtime press on a zero-area target beside a scroll container names the scroll that reveals it', async () => {
180+
const taps: Point[] = [];
181+
const device = createInteractionDevice(scrolledOutButtonSnapshot(), {
182+
tap: async (_context, point) => {
183+
taps.push(point);
184+
return { ok: true };
185+
},
186+
});
187+
188+
await assert.rejects(
189+
() => device.interactions.press(selector('label="View all"'), { session: 'default' }),
190+
(error: unknown) => {
191+
const details = (error as { details?: Record<string, unknown> }).details;
192+
assert.equal(details?.reason, 'target_bounds_invalid');
193+
assert.equal(details?.dispatched, 'no');
194+
assert.equal(details?.scrollContainerRef, 'e2');
195+
assert.match(String(details?.hint), /scroll <direction> --until 'label="View all"'/);
196+
return true;
197+
},
198+
);
199+
await assert.rejects(
200+
() => device.interactions.press(ref('@e4'), { session: 'default' }),
201+
(error: unknown) => {
202+
const details = (error as { details?: Record<string, unknown> }).details;
203+
assert.equal(details?.reason, 'target_bounds_invalid');
204+
assert.equal(details?.scrollContainerRef, 'e2');
205+
// A scroll expires the ref frame, so the ref recovery steers to a selector instead.
206+
assert.match(String(details?.hint), /retry with a selector/);
207+
assert.match(String(details?.hint), /re-run snapshot -i/);
208+
assert.doesNotMatch(String(details?.hint), /--until/);
209+
return true;
210+
},
211+
);
212+
assert.deepEqual(taps, []);
213+
});
214+
215+
test('the scroll a zero-area target is pointed at survives the shell', async () => {
216+
// The hint is copied into a shell: double quotes would expand the `$`, and a bare single-quote
217+
// wrap would end at the apostrophe.
218+
const device = createInteractionDevice(scrolledOutButtonSnapshot("Today's $5 deal"), {
219+
tap: async () => ({ ok: true }),
220+
});
221+
222+
await assert.rejects(
223+
() => device.interactions.press(selector('label="Today\'s $5 deal"'), { session: 'default' }),
224+
(error: unknown) => {
225+
const details = (error as { details?: Record<string, unknown> }).details;
226+
assert.equal(details?.reason, 'target_bounds_invalid');
227+
assert.match(String(details?.hint), /--until 'label="Today'\\''s \$5 deal"'/);
228+
return true;
229+
},
230+
);
231+
});
232+
129233
test('runtime press on a Compose button dispatches its center only once its role node is folded', async () => {
130234
const tree = parseUiHierarchyTree(`<hierarchy>
131235
<node class="android.view.View" resource-id="compose-container" bounds="[53,1856][1027,1982]"
@@ -166,3 +270,98 @@ test('runtime press on a Compose button dispatches its center only once its role
166270
dispatched: 'no',
167271
});
168272
});
273+
274+
test('runtime press offers no scroll for a zero-area target listed before the only scroll container', async () => {
275+
const state = scrolledOutButtonSnapshot();
276+
const [application, scrollView, , button] = state.nodes;
277+
const device = createInteractionDevice(
278+
makeSnapshotState([application!, { ...button!, index: 1 }, { ...scrollView!, index: 2 }]),
279+
{ tap: async () => ({ ok: true }) },
280+
);
281+
282+
await assert.rejects(
283+
() => device.interactions.press(selector('label="View all"'), { session: 'default' }),
284+
(error: unknown) => {
285+
const details = (error as { details?: Record<string, unknown> }).details;
286+
assert.equal(details?.reason, 'target_bounds_invalid');
287+
assert.equal(details?.scrollContainerRef, undefined);
288+
assert.equal(details?.hint, undefined);
289+
return true;
290+
},
291+
);
292+
});
293+
294+
test('runtime press offers no scroll for a measured target whose input bounds exclude it', async () => {
295+
const root = { x: 0, y: 0, width: 1080, height: 2400 };
296+
// A row with a real frame, wholly under the navigation bar the input bounds stop at.
297+
const row = attachSnapshotNodeInteractionBounds(
298+
{
299+
index: 2,
300+
depth: 2,
301+
parentIndex: 1,
302+
type: 'Button',
303+
identifier: 'row',
304+
label: 'Row',
305+
hittable: true,
306+
rect: { x: 0, y: 2305, width: 1080, height: 95 },
307+
},
308+
{ ...root, height: 2274 },
309+
);
310+
const snapshot = buildSnapshotState(
311+
{
312+
backend: 'android',
313+
producer: 'android-uiautomator',
314+
nodes: [
315+
{ index: 0, depth: 0, type: 'Application', rect: root },
316+
{ index: 1, depth: 1, parentIndex: 0, type: 'ScrollView', rect: root, hittable: true },
317+
row,
318+
],
319+
},
320+
{ snapshotRaw: true },
321+
);
322+
const device = createInteractionDevice(snapshot, {
323+
platform: 'android',
324+
tap: async () => ({ ok: true }),
325+
});
326+
327+
await assert.rejects(
328+
() => device.interactions.press(selector('id=row'), { session: 'default' }),
329+
(error: unknown) => {
330+
const details = (error as { details?: Record<string, unknown> }).details;
331+
assert.equal(details?.reason, 'target_bounds_invalid');
332+
assert.equal(details?.scrollContainerRef, undefined);
333+
assert.equal(details?.hint, undefined);
334+
return true;
335+
},
336+
);
337+
});
338+
339+
test('runtime press names the scroll view, not a scroll bar listed after it, as the scroll container', async () => {
340+
const state = scrolledOutButtonSnapshot();
341+
const [application, scrollView, block, button] = state.nodes;
342+
const device = createInteractionDevice(
343+
makeSnapshotState([
344+
application!,
345+
scrollView!,
346+
{
347+
...block!,
348+
type: 'Other',
349+
role: 'AXScrollBar',
350+
label: 'Vertical scroll bar, 2 pages',
351+
rect: { x: 396, y: 118, width: 6, height: 756 },
352+
},
353+
button!,
354+
]),
355+
{ tap: async () => ({ ok: true }) },
356+
);
357+
358+
await assert.rejects(
359+
() => device.interactions.press(selector('label="View all"'), { session: 'default' }),
360+
(error: unknown) => {
361+
const details = (error as { details?: Record<string, unknown> }).details;
362+
assert.equal(details?.reason, 'target_bounds_invalid');
363+
assert.equal(details?.scrollContainerRef, 'e2');
364+
return true;
365+
},
366+
);
367+
});

‎packages/command-runtime/src/commands/interaction/runtime/resolution.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { shellQuoteIfNeeded } from '@agent-device/kernel/device-shell';
12
import { AppError } from '@agent-device/kernel/errors';
23
import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract';
34
import {
@@ -162,6 +163,7 @@ async function resolveSelectorInteractionTarget(
162163
invalidMessage: `Selector ${resolved.selector} resolved to invalid bounds`,
163164
blockedTargetLabel: `Selector ${selectorExpression}`,
164165
blockedTargetDetails: { selector: selectorExpression },
166+
shellQuotedSelector: shellQuoteIfNeeded(selectorExpression),
165167
}),
166168
});
167169
return {

‎packages/command-runtime/src/commands/interaction/runtime/target-visibility-stages.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,41 @@ test('runtime press names a direction for a partial clip whose center is off-scr
105105
assert.equal(taps.length, 0);
106106
});
107107

108+
test('the scroll an off-screen selector target is pointed at survives the shell', async () => {
109+
// The hint is copied into a shell, and a bare single-quote wrap would end at the apostrophe.
110+
const device = createInteractionDevice(
111+
makeSnapshotState([
112+
{
113+
index: 0,
114+
depth: 0,
115+
type: 'Application',
116+
rect: { x: 0, y: 0, width: 400, height: 800 },
117+
hittable: true,
118+
},
119+
{
120+
index: 1,
121+
depth: 2,
122+
parentIndex: 0,
123+
type: 'Button',
124+
label: "Today's cash",
125+
rect: { x: 20, y: 900, width: 200, height: 44 },
126+
hittable: true,
127+
},
128+
]),
129+
{ tap: async () => {} },
130+
);
131+
132+
await assert.rejects(
133+
() => device.interactions.press(selector('label="Today\'s cash"'), { session: 'default' }),
134+
(error: unknown) => {
135+
const details = (error as { details?: Record<string, unknown> }).details;
136+
assert.equal(details?.reason, 'offscreen_selector');
137+
assert.match(String(details?.hint), /--until 'label="Today'\\''s cash"'/);
138+
return true;
139+
},
140+
);
141+
});
142+
108143
test('runtime click rejects refs covered by floating overlays', async () => {
109144
const calls: Point[] = [];
110145
const device = createInteractionDevice(coveredByTabBarSnapshot(), {

0 commit comments

Comments
 (0)