Skip to content

Commit 2addea4

Browse files
committed
fix(snapshot): mark unmeasured nodes in the shared snapshot assembly and quote scroll hints with shellQuote
The unmeasured mark moves from the iOS acquired-tree fold to buildSnapshotState, next to the covered annotation, so every producer gets it: the XCTest runner, the Simulator AX bridge, the provider trees, Android and desktop. Regular snapshots only, as before. The zero-area recovery takes its selector already quoted from the command-runtime caller, which uses the kernel's shellQuote; the selectors entry cannot import it without growing its eager closure. The off-screen hint uses the same helper, so a selector with an apostrophe no longer breaks its command. scrollContainerRef is the bare ref body, like details.ref.
1 parent 2cc05f3 commit 2addea4

11 files changed

Lines changed: 157 additions & 100 deletions

File tree

‎packages/capture-kit/src/ios-snapshot-engine/geometry-policy.test.ts‎

Lines changed: 0 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -257,58 +257,3 @@ test('fails closed when the producer did not read whether the dimming view takes
257257
]);
258258
expect(collectModalContainedIndexes(nodes).size).toBe(0);
259259
});
260-
261-
/**
262-
* A Flutter screen at the top of its content, as the Simulator AX bridge acquires it (#4451): one
263-
* container lists the scroll view and then its content, and the content below the fold keeps a
264-
* zero-area frame at the origin.
265-
*/
266-
function flutterScrollViewAtTop(): RawSnapshotNode[] {
267-
const unmeasured: Rect = { x: 0, y: 0, width: 0, height: 0 };
268-
return windowWith([
269-
snapshotNode(1, 1, 0, 'SemanticsObjectContainer', 'Other'),
270-
snapshotNode(2, 2, 1, 'FlutterSemanticsScrollView', 'ScrollView', undefined, {
271-
x: 0,
272-
y: 118,
273-
width: 402,
274-
height: 756,
275-
}),
276-
snapshotNode(3, 2, 1, 'FlutterSemanticsObject', 'StaticText', 'Tall block', {
277-
x: 157,
278-
y: 546,
279-
width: 86,
280-
height: 27,
281-
}),
282-
snapshotNode(4, 2, 1, 'FlutterSemanticsObject', 'Button', 'View all', unmeasured),
283-
]);
284-
}
285-
286-
test('marks a kept node whose producer reported a zero-area frame as unmeasured', () => {
287-
const folded = foldIosSnapshot(flutterScrollViewAtTop(), VIEWPORT, false, 'cursor-projected');
288-
expect(
289-
folded.nodes.map((node) => [node.raw.label ?? node.raw.type, node.raw.presentationHints]),
290-
).toEqual([
291-
['Window', undefined],
292-
['Other', undefined],
293-
['ScrollView', undefined],
294-
['Tall block', undefined],
295-
['View all', ['unmeasured']],
296-
]);
297-
});
298-
299-
test('leaves a zero-area root unmarked, since nothing measures a root against a parent', () => {
300-
const folded = foldIosSnapshot(
301-
[
302-
snapshotNode(0, 0, undefined, 'UIWindow', 'Window', undefined, {
303-
x: 0,
304-
y: 0,
305-
width: 0,
306-
height: 0,
307-
}),
308-
],
309-
VIEWPORT,
310-
false,
311-
'cursor-projected',
312-
);
313-
expect(folded.nodes.map((node) => node.raw.presentationHints)).toEqual([undefined]);
314-
});

‎packages/capture-kit/src/ios-snapshot-engine/geometry.ts‎

Lines changed: 0 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,6 @@ import type {
1515
IosSnapshotPresentationStats,
1616
} from './types.ts';
1717

18-
/**
19-
* The hint a presented node carries when its producer reported a frame that measures nothing. The
20-
* cursor-projected fold keeps such a node, because a missing measurement is not evidence that it
21-
* is off screen, but nothing confirms that it is on screen either: a scroll view keeps its
22-
* scrolled-out content in the tree this way (#4451), as SpringBoard keeps another page's icons
23-
* (#3841).
24-
*/
25-
const UNMEASURED_PRESENTATION_HINT = 'unmeasured';
26-
2718
export function foldIosSnapshot(
2819
nodes: readonly RawSnapshotNode[],
2920
viewport: Rect,
@@ -151,7 +142,6 @@ function appendFoldedNode(
151142
index,
152143
depth: keptDepth,
153144
parentIndex: keptIndex,
154-
...unmeasuredHint(node),
155145
...foldedHittability(
156146
sourceHittable,
157147
node.parentIndex !== undefined,
@@ -168,13 +158,6 @@ function appendFoldedNode(
168158
return { keptIndex, keptDepth };
169159
}
170160

171-
function unmeasuredHint(
172-
node: RawSnapshotNode,
173-
): Partial<Pick<RawSnapshotNode, 'presentationHints'>> {
174-
if (node.parentIndex === undefined || !node.rect || isPositiveFiniteRect(node.rect)) return {};
175-
return { presentationHints: [UNMEASURED_PRESENTATION_HINT] };
176-
}
177-
178161
function foldedHittability(
179162
sourceHittable: RawSnapshotNode['hittable'],
180163
hasParent: boolean,
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
import { expect, test } from 'vitest';
2+
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
3+
import type { SnapshotCaptureProvenance } from '@agent-device/contracts/snapshot-model';
4+
import { buildSnapshotState } from './snapshot-state.ts';
5+
6+
/**
7+
* A Flutter screen on iOS at the top of its content (#4451): the scroll view and its content are
8+
* siblings, and the content below the fold keeps a zero-area frame at the origin.
9+
*/
10+
function scrolledOutButtonTree(): RawSnapshotNode[] {
11+
return [
12+
{ index: 0, depth: 0, type: 'Application', rect: { x: 0, y: 0, width: 402, height: 874 } },
13+
{
14+
index: 1,
15+
depth: 1,
16+
parentIndex: 0,
17+
type: 'ScrollView',
18+
rect: { x: 0, y: 118, width: 402, height: 756 },
19+
},
20+
{
21+
index: 2,
22+
depth: 1,
23+
parentIndex: 0,
24+
type: 'StaticText',
25+
label: 'Tall block',
26+
rect: { x: 157, y: 546, width: 86, height: 27 },
27+
},
28+
{
29+
index: 3,
30+
depth: 1,
31+
parentIndex: 0,
32+
type: 'Button',
33+
label: 'View all',
34+
rect: { x: 0, y: 0, width: 0, height: 0 },
35+
},
36+
{ index: 4, depth: 1, parentIndex: 0, type: 'StaticText', label: 'No frame' },
37+
];
38+
}
39+
40+
function hintsByNode(nodes: readonly RawSnapshotNode[]) {
41+
return nodes.map((node) => [node.label ?? node.type, node.presentationHints]);
42+
}
43+
44+
const PROVENANCES: SnapshotCaptureProvenance[] = [
45+
{ backend: 'xctest', producer: 'apple-runner' },
46+
{ backend: 'android', producer: 'android-uiautomator' },
47+
];
48+
49+
test.each(PROVENANCES)(
50+
'marks a node published with a zero-area rect as unmeasured ($producer)',
51+
(provenance) => {
52+
const state = buildSnapshotState({ nodes: scrolledOutButtonTree(), ...provenance }, undefined);
53+
54+
expect(hintsByNode(state.nodes)).toEqual([
55+
['Application', undefined],
56+
['ScrollView', undefined],
57+
['Tall block', undefined],
58+
['View all', ['unmeasured']],
59+
// A node that reports no frame at all made no measurement to doubt.
60+
['No frame', undefined],
61+
]);
62+
},
63+
);
64+
65+
test('a raw snapshot keeps the reported nodes without the unmeasured mark', () => {
66+
const state = buildSnapshotState(
67+
{ nodes: scrolledOutButtonTree(), backend: 'xctest', producer: 'apple-runner' },
68+
{ snapshotRaw: true },
69+
);
70+
71+
expect(state.nodes.flatMap((node) => node.presentationHints ?? [])).toEqual([]);
72+
});

‎packages/capture-kit/src/snapshot-state.ts‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import {
2222
type SnapshotViewportSize,
2323
type SnapshotQualityNote,
2424
} from '@agent-device/kernel/snapshot';
25+
import { isPositiveFiniteRect } from '@agent-device/kernel/rect';
2526
import {
2627
snapshotRefMinting,
2728
type SnapshotBackend,
@@ -104,7 +105,9 @@ export function buildSnapshotState(
104105
const nodes = attachRefs(
105106
snapshotRaw
106107
? scopedNodes
107-
: annotateCoveredSnapshotNodes(scopedNodes, overlayOptionsForBackend(data?.backend)),
108+
: annotateUnmeasuredSnapshotNodes(
109+
annotateCoveredSnapshotNodes(scopedNodes, overlayOptionsForBackend(data?.backend)),
110+
),
108111
snapshotRefMinting(data),
109112
);
110113
const textInputGeometry = readSnapshotTextInputGeometry(data.nodes);
@@ -136,6 +139,25 @@ export function buildSnapshotState(
136139
};
137140
}
138141

142+
/**
143+
* The hint a node carries when the rect it is published with has no area. That is not evidence
144+
* that the node is off screen, but nothing confirms that it is on screen either: a scroll view
145+
* keeps its scrolled-out content in the tree with a zero-area frame (#4451), as SpringBoard keeps
146+
* another page's icons (#3841).
147+
*/
148+
const UNMEASURED_PRESENTATION_HINT = 'unmeasured';
149+
150+
function annotateUnmeasuredSnapshotNodes(nodes: RawSnapshotNode[]): RawSnapshotNode[] {
151+
return nodes.map((node) =>
152+
node.rect && !isPositiveFiniteRect(node.rect)
153+
? {
154+
...node,
155+
presentationHints: [...(node.presentationHints ?? []), UNMEASURED_PRESENTATION_HINT],
156+
}
157+
: node,
158+
);
159+
}
160+
139161
function annotateBackendReplacementSurfaces(
140162
owner: object & { backend?: SnapshotBackend },
141163
nodes: RawSnapshotNode[],

‎packages/command-runtime/src/commands/interaction/runtime/resolution-touch-point.test.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ test('runtime press on a zero-area target beside a scroll container names the sc
191191
const details = (error as { details?: Record<string, unknown> }).details;
192192
assert.equal(details?.reason, 'target_bounds_invalid');
193193
assert.equal(details?.dispatched, 'no');
194-
assert.equal(details?.scrollContainerRef, '@e2');
194+
assert.equal(details?.scrollContainerRef, 'e2');
195195
assert.match(String(details?.hint), /scroll <direction> --until 'label="View all"'/);
196196
return true;
197197
},
@@ -201,7 +201,7 @@ test('runtime press on a zero-area target beside a scroll container names the sc
201201
(error: unknown) => {
202202
const details = (error as { details?: Record<string, unknown> }).details;
203203
assert.equal(details?.reason, 'target_bounds_invalid');
204-
assert.equal(details?.scrollContainerRef, '@e2');
204+
assert.equal(details?.scrollContainerRef, 'e2');
205205
// A scroll expires the ref frame, so the ref recovery steers to a selector instead.
206206
assert.match(String(details?.hint), /retry with a selector/);
207207
assert.match(String(details?.hint), /re-run snapshot -i/);
@@ -360,7 +360,7 @@ test('runtime press names the scroll view, not a scroll bar listed after it, as
360360
(error: unknown) => {
361361
const details = (error as { details?: Record<string, unknown> }).details;
362362
assert.equal(details?.reason, 'target_bounds_invalid');
363-
assert.equal(details?.scrollContainerRef, '@e2');
363+
assert.equal(details?.scrollContainerRef, 'e2');
364364
return true;
365365
},
366366
);

‎packages/command-runtime/src/commands/interaction/runtime/resolution.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { shellQuote } from '@agent-device/kernel/device-shell';
12
import { AppError } from '@agent-device/kernel/errors';
23
import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract';
34
import {
@@ -162,6 +163,7 @@ async function resolveSelectorInteractionTarget(
162163
invalidMessage: `Selector ${resolved.selector} resolved to invalid bounds`,
163164
blockedTargetLabel: `Selector ${selectorExpression}`,
164165
blockedTargetDetails: { selector: selectorExpression },
166+
shellQuotedSelector: shellQuote(selectorExpression),
165167
}),
166168
});
167169
return {

‎packages/command-runtime/src/commands/interaction/runtime/target-visibility-stages.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,41 @@ test('runtime press names a direction for a partial clip whose center is off-scr
105105
assert.equal(taps.length, 0);
106106
});
107107

108+
test('the scroll an off-screen selector target is pointed at survives the shell', async () => {
109+
// The hint is copied into a shell, and a bare single-quote wrap would end at the apostrophe.
110+
const device = createInteractionDevice(
111+
makeSnapshotState([
112+
{
113+
index: 0,
114+
depth: 0,
115+
type: 'Application',
116+
rect: { x: 0, y: 0, width: 400, height: 800 },
117+
hittable: true,
118+
},
119+
{
120+
index: 1,
121+
depth: 2,
122+
parentIndex: 0,
123+
type: 'Button',
124+
label: "Today's cash",
125+
rect: { x: 20, y: 900, width: 200, height: 44 },
126+
hittable: true,
127+
},
128+
]),
129+
{ tap: async () => {} },
130+
);
131+
132+
await assert.rejects(
133+
() => device.interactions.press(selector('label="Today\'s cash"'), { session: 'default' }),
134+
(error: unknown) => {
135+
const details = (error as { details?: Record<string, unknown> }).details;
136+
assert.equal(details?.reason, 'offscreen_selector');
137+
assert.match(String(details?.hint), /--until 'label="Today'\\''s cash"'/);
138+
return true;
139+
},
140+
);
141+
});
142+
108143
test('runtime click rejects refs covered by floating overlays', async () => {
109144
const calls: Point[] = [];
110145
const device = createInteractionDevice(coveredByTabBarSnapshot(), {

‎packages/command-runtime/src/commands/interaction/runtime/target-visibility-stages.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { shellQuote } from '@agent-device/kernel/device-shell';
12
import { AppError, discloseDispatch } from '@agent-device/kernel/errors';
23
import { assertSnapshotScreenOrigin } from '@agent-device/contracts/capture';
34
import type {
@@ -175,7 +176,7 @@ function scrollRevealClause(
175176
): string {
176177
if (!direction) return 'Scroll toward it,';
177178
if (!selector) return `Scroll ${direction} toward it,`;
178-
return `Run scroll ${direction} --until '${selector}' to bring it on screen,`;
179+
return `Run scroll ${direction} --until ${shellQuote(selector)} to bring it on screen,`;
179180
}
180181

181182
// Full on-screen visibility (not only the effective-viewport form): items inside an

‎packages/command-surface/src/capture/snapshot.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ export const snapshotCommandFacet = defineCommandFacet({
7575
name: SNAPSHOT_COMMAND_NAME,
7676
text: {
7777
summary: 'Capture or diff the accessibility tree',
78-
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular Android ADB snapshots clip app rectangles outside measured status and navigation bars, including launcher-owned navigation; fully covered content is omitted and --raw retains reported bounds. Ref and selector taps honor measured visible bounds even for refs from raw snapshots. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. In a regular (non-raw) iOS snapshot presented from an acquired tree (the Simulator AX bridge and provider backends, not the XCTest runner), an element that reports a frame measuring nothing is kept and marked [unmeasured]: nothing confirms it is on screen. If it is inside scrollable content, reach it with scroll <direction> --until <selector> before acting on it. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS reads the on-screen keyboard's own system window and publishes the keyboard field of --json output: the field is visibility-only — visible or hidden per the keyboard window's z-order column; a failed probe, an unreadable or conflicting z-order, or a table without the panel publishes unmeasurable, never a guessed band. When an iOS snapshot reports a screen viewport, inspect rects with snapshot -i --json and press the intended control's center, then verify with diff snapshot -i or snapshot --diff. When a fallback cannot establish the screen origin, viewport is omitted and coordinate-derived element actions refuse with window-origin-unknown before input; the snapshot may warn about the unavailable origin. Use a plain screenshot to choose screen coordinates. An empty text field with no label or value prints its hint as [placeholder:"..."]; label=, text= and find do not match that hint. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
78+
cliDetail: `Scoped output (--scope) renders every returned ref with its ref generation pin (@eN~s<refsGeneration>) and one Refs generation line, so a pasted ref targets the ref frame that issued it; --json keeps plain refs plus the response-level refsGeneration, and --raw keeps byte-for-byte output. Regular Android ADB snapshots clip app rectangles outside measured status and navigation bars, including launcher-owned navigation; fully covered content is omitted and --raw retains reported bounds. Ref and selector taps honor measured visible bounds even for refs from raw snapshots. Regular iOS snapshots clip overflowing elements to their visible area and omit fully outside elements. In a regular (non-raw) snapshot, a listed element whose frame has no area is marked [unmeasured]: nothing confirms it is on screen. If it is inside scrollable content, reach it with scroll <direction> --until <selector> before acting on it. --json reports clipping counts in snapshotQualityNotes with code regular-geometry-clipped; --raw retains reported frames. --observe-only is local iOS snapshot-only: an already-ready compatible runner reads the session app without activation or foreground repair, refuses unavailable observation or a readable system alert over the app, and reports XCUIApplication state rather than verified foreground ownership (#2696). Regular targetActivation disclosure stays unchanged (#2682, #2694). Repeated equivalent unfiltered Android snapshots return a compact unchanged acknowledgement. Use --force-full to re-emit the tree; --json and --raw retain full output. Android Appium JSON snapshots publish a display viewport only when measurements before and after the tree agree, independent of tree extent or keyboard content resizing. On Android, JSON value carries field content rather than static labels; saved selectors and replay scripts using value= for static labels must use label= instead; a reported showing hint means value is empty, and text equal to the placeholder is preserved when hint state is unknown. HarmonyOS reads the on-screen keyboard's own system window and publishes the keyboard field of --json output: the field is visibility-only — visible or hidden per the keyboard window's z-order column; a failed probe, an unreadable or conflicting z-order, or a table without the panel publishes unmeasurable, never a guessed band. When an iOS snapshot reports a screen viewport, inspect rects with snapshot -i --json and press the intended control's center, then verify with diff snapshot -i or snapshot --diff. When a fallback cannot establish the screen origin, viewport is omitted and coordinate-derived element actions refuse with window-origin-unknown before input; the snapshot may warn about the unavailable origin. Use a plain screenshot to choose screen coordinates. An empty text field with no label or value prints its hint as [placeholder:"..."]; label=, text= and find do not match that hint. iOS backend capability contract: ${snapshotBackendCapabilityHelp}.`,
7979
},
8080
metadata: snapshotCommandMetadata,
8181
run: (client, input) => client.capture.snapshot(input),

0 commit comments

Comments
 (0)