Skip to content

Commit 428b8c9

Browse files
committed
feat(ios): publish effective snapshot geometry
1 parent e4b19df commit 428b8c9

10 files changed

Lines changed: 451 additions & 228 deletions

‎.github/workflows/ios.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ jobs:
151151
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainForSnapshotRaw \
152152
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds \
153153
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationPreservesCurrentWireShape \
154+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationPublishesEffectiveRectWhileRawKeepsReportedFrame \
154155
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsBackendNeutralEligibility \
155156
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \
156157
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \
@@ -159,6 +160,9 @@ jobs:
159160
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \
160161
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \
161162
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \
163+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyDoesNotClipToScrollAncestor \
164+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testEffectiveGeometryIntersectsViewportAndAncestorClip \
165+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testEffectiveGeometryKeepsReportedOriginWhenFullyClipped \
162166
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationRoutesThroughVisibilityFold \
163167
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \
164168
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
## Unreleased
44

5+
- iOS regular snapshot nodes now publish presentation-owned effective geometry through the existing
6+
`rect` field: backend-reported frames remain available to acquisition, while regular output uses
7+
the viewport and declared scroll-clip intersection. Raw snapshots and direct element reads retain
8+
reported geometry (#1797).
59
- Android recording-session commands now warn when blocking-dialog readiness inspection fails open. The requested command still runs, but the successful response discloses that readiness could not be inspected and preserves the inspection error's actionable hint, including `pnpm build:android` when the snapshot helper is unavailable (#1895).
610
- Breaking (device selection): when a command needs one concrete device, carries no `--device`/`--udid`/`--serial`, and more than one candidate is equally preferred, agent-device now refuses with `AMBIGUOUS_MATCH` and the candidate list instead of quietly picking one. Established preferences are unchanged — virtual over physical, booted over offline, and the Apple kind/target ranking — so a single booted emulator beside offline ones still resolves, an existing session binding still resolves (its identity is already fixed), and explicit selectors still resolve. What no longer happens is choosing between two equally booted devices by discovery order or alphabetically: that produced a **successful response describing a device the caller never selected**, and reads are no safer than writes there. `devices` and other genuinely multi-device commands never enter singular resolution and are unaffected. The error carries the bounded candidate list in the declared `devices` details domain, so CLI and MCP print it, with a hint naming the right selector for the platform (`--serial` for Android/HarmonyOS, `--udid` for Apple, or `--device "<name>"`).
711
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotPresentation.swift‎

Lines changed: 48 additions & 161 deletions
Original file line numberDiff line numberDiff line change
@@ -1,134 +1,5 @@
11
import Foundation
22

3-
/// Backend-owned snapshot output before it crosses the presentation seam.
4-
///
5-
/// The incremental #1797 migration still carries derived fields that later semantic layers move
6-
/// behind `SnapshotPresentation`. Eligibility is no longer one of those backend-owned decisions.
7-
struct RawAXNode {
8-
let index: Int
9-
let type: String
10-
let label: String?
11-
let identifier: String?
12-
let value: String?
13-
let rect: SnapshotRect
14-
let enabled: Bool
15-
let focused: Bool?
16-
let selected: Bool?
17-
let hittable: Bool
18-
let depth: Int
19-
let parentIndex: Int?
20-
let hiddenContentAbove: Bool?
21-
let hiddenContentBelow: Bool?
22-
var actions: [String]? = nil
23-
24-
var hasSemanticContent: Bool {
25-
[label, identifier, value].contains {
26-
!($0?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty ?? true)
27-
}
28-
}
29-
}
30-
31-
/// The acquisition-facing view of a snapshot request, derived once by
32-
/// `SnapshotPresentation.captureHint(for:)`.
33-
///
34-
/// Backends read a hint, never `PresentationOptions`: presentation owns interpretation, and a hint
35-
/// may narrow acquisition only where the backend can prove the narrowing complete for the requested
36-
/// projection (#1797 conservatism). Everything else a hint carries is budget and ordering.
37-
struct CaptureHint {
38-
/// Which projection this acquisition must serve. Backends that cannot serve one are not planned
39-
/// for it (`SnapshotBackendKind.supportsRawProjection`), and presentation refuses an acquisition
40-
/// captured for the other projection rather than relabeling it, so a `--raw` request can never be
41-
/// answered with regular-projection membership (#1797 D4).
42-
enum Projection: String {
43-
case regular
44-
case raw
45-
}
46-
47-
let projection: Projection
48-
/// Traversal-depth budget.
49-
///
50-
/// Declared residue (#1797): complete for the raw projection, whose presented depth *is*
51-
/// traversal depth. Regular presentation emits collapsed depth, so cutting the traversal at this
52-
/// limit can still drop a node that would have presented within it — the open "visible-depth
53-
/// frontier completeness" obligation, kept as-is here because the cut is also what keeps
54-
/// `--depth 1` probes cheap.
55-
let depth: Int?
56-
/// Regular-projection acquisition budget. The raw projection is the acquired tree, so it never
57-
/// carries this: `--raw -i` returns everything the backend serialized.
58-
let interactiveOnly: Bool
59-
let customActions: Bool
60-
61-
var isRaw: Bool { projection == .raw }
62-
}
63-
64-
/// One backend attempt after acquisition and its current backend-specific interpretation.
65-
///
66-
/// It is the only input snapshot presentation accepts, and it carries the hint it was captured
67-
/// under so the two sides of the seam cannot disagree about which projection this is. Later #1797
68-
/// steps move the interpretation that still precedes this value — the clip fold and hittability —
69-
/// into `SnapshotPresentation` without changing the capture-plan seam.
70-
struct SnapshotAcquisition {
71-
/// The hint this acquisition was captured under. Presentation compares it with the requested
72-
/// projection instead of trusting the backend's label.
73-
let hint: CaptureHint
74-
let nodes: [RawAXNode]
75-
let truncated: Bool
76-
let effectiveDepth: Int?
77-
var customActions: SnapshotCustomActionCoverage? = nil
78-
/// Viewport the regular projection's clip fold runs against. `.infinite` disables the fold --
79-
/// legitimate only for raw acquisitions and depth-0 probes, where no fold applies.
80-
let viewport: CGRect
81-
}
82-
83-
/// The only snapshot node shape accepted by response payload assembly.
84-
///
85-
/// Its initializer is private so a backend cannot bypass `SnapshotPresentation`. The encoded shape
86-
/// intentionally remains byte-for-byte compatible with the former `SnapshotNode` wire model.
87-
struct PresentedNode: Codable {
88-
let index: Int
89-
let type: String
90-
let label: String?
91-
let identifier: String?
92-
let value: String?
93-
let rect: SnapshotRect
94-
let enabled: Bool
95-
let focused: Bool?
96-
let selected: Bool?
97-
let hittable: Bool
98-
let depth: Int
99-
let parentIndex: Int?
100-
let hiddenContentAbove: Bool?
101-
let hiddenContentBelow: Bool?
102-
let actions: [String]?
103-
104-
fileprivate init(presenting raw: RawAXNode) {
105-
self.init(
106-
presenting: raw,
107-
index: raw.index,
108-
depth: raw.depth,
109-
parentIndex: raw.parentIndex
110-
)
111-
}
112-
113-
fileprivate init(presenting raw: RawAXNode, index: Int, depth: Int, parentIndex: Int?) {
114-
self.index = index
115-
type = raw.type
116-
label = raw.label
117-
identifier = raw.identifier
118-
value = raw.value
119-
rect = raw.rect
120-
enabled = raw.enabled
121-
focused = raw.focused
122-
selected = raw.selected
123-
hittable = raw.hittable
124-
self.depth = depth
125-
self.parentIndex = parentIndex
126-
hiddenContentAbove = raw.hiddenContentAbove
127-
hiddenContentBelow = raw.hiddenContentBelow
128-
actions = raw.actions
129-
}
130-
}
131-
1323
enum SnapshotPresentation {
1334
private static let eligibleInteractiveTypes: Set<String> = [
1345
"Button",
@@ -207,7 +78,12 @@ enum SnapshotPresentation {
20778
_ acquisition: SnapshotAcquisition,
20879
options: PresentationOptions
20980
) -> SnapshotBackendCapture {
210-
project(acquisition.nodes, acquisition: acquisition, options: options, projection: .raw)
81+
project(
82+
acquisition.nodes.map(SnapshotPresentationNode.reported),
83+
acquisition: acquisition,
84+
options: options,
85+
projection: .raw
86+
)
21187
}
21288

21389
/// Derives the one acquisition-facing view of a request, so no backend re-reads
@@ -227,7 +103,7 @@ enum SnapshotPresentation {
227103
}
228104

229105
private static func project(
230-
_ projectionNodes: [RawAXNode],
106+
_ projectionNodes: [SnapshotPresentationNode],
231107
acquisition: SnapshotAcquisition,
232108
options: PresentationOptions,
233109
projection: CaptureHint.Projection
@@ -258,16 +134,17 @@ enum SnapshotPresentation {
258134
}
259135

260136
private static func presentedNodes(
261-
from rawNodes: [RawAXNode],
137+
from rawNodes: [SnapshotPresentationNode],
262138
projection: CaptureHint.Projection
263139
) -> [PresentedNode] {
264140
if projection == .raw {
265-
return rawNodes.map(PresentedNode.init(presenting:))
141+
return rawNodes.map { PresentedNode(presenting: $0) }
266142
}
267143

268144
var nodes: [PresentedNode] = []
269145
var nearestPresentedNodeByRawIndex: [Int: (index: Int, depth: Int)] = [:]
270-
for raw in rawNodes {
146+
for node in rawNodes {
147+
let raw = node.raw
271148
let presentedParent = raw.parentIndex.flatMap {
272149
nearestPresentedNodeByRawIndex[$0]
273150
}
@@ -284,6 +161,7 @@ enum SnapshotPresentation {
284161
nodes.append(
285162
PresentedNode(
286163
presenting: raw,
164+
rect: node.effectiveRect,
287165
index: presentedIndex,
288166
depth: presentedDepth,
289167
parentIndex: presentedParent?.index
@@ -294,57 +172,66 @@ enum SnapshotPresentation {
294172
}
295173

296174
private static func applyScope(
297-
to rawNodes: [RawAXNode],
175+
to rawNodes: [SnapshotPresentationNode],
298176
options: PresentationOptions,
299177
projection: CaptureHint.Projection
300-
) -> [RawAXNode] {
178+
) -> [SnapshotPresentationNode] {
301179
switch SnapshotScopePolicy.select(
302180
fromPreorder: rawNodes,
303181
scope: options.scope,
304-
depth: \.depth,
305-
semanticValues: { [$0.label, $0.identifier, $0.value] },
182+
depth: { $0.raw.depth },
183+
semanticValues: { [$0.raw.label, $0.raw.identifier, $0.raw.value] },
306184
subtreeContributes: { range in
307-
projection == .raw || rawNodes[range].contains(where: isEligibleForRegularPresentation)
185+
projection == .raw
186+
|| rawNodes[range].contains { isEligibleForRegularPresentation($0.raw) }
308187
}
309188
) {
310189
case .unscoped:
311190
return rawNodes
312191
case .missing:
313192
return []
314193
case .matched(let startIndex):
315-
let startDepth = rawNodes[startIndex].depth
194+
let startDepth = rawNodes[startIndex].raw.depth
316195
let range = SnapshotScopePolicy.subtreeRange(
317196
from: startIndex,
318197
in: rawNodes,
319-
depth: \.depth
198+
depth: { $0.raw.depth }
320199
)
321200
let maxDepth = options.depth ?? Int.max
322201
return reindex(
323-
Array(rawNodes[range]).filter { $0.depth - startDepth <= maxDepth },
202+
Array(rawNodes[range]).filter { $0.raw.depth - startDepth <= maxDepth },
324203
depthOffset: startDepth
325204
)
326205
}
327206
}
328207

329-
private static func reindex(_ rawNodes: [RawAXNode], depthOffset: Int) -> [RawAXNode] {
330-
let indexMap = Dictionary(uniqueKeysWithValues: rawNodes.enumerated().map { ($0.element.index, $0.offset) })
331-
return rawNodes.enumerated().map { offset, raw in
332-
RawAXNode(
333-
index: offset,
334-
type: raw.type,
335-
label: raw.label,
336-
identifier: raw.identifier,
337-
value: raw.value,
338-
rect: raw.rect,
339-
enabled: raw.enabled,
340-
focused: raw.focused,
341-
selected: raw.selected,
342-
hittable: raw.hittable,
343-
depth: max(0, raw.depth - depthOffset),
344-
parentIndex: raw.parentIndex.flatMap { indexMap[$0] },
345-
hiddenContentAbove: raw.hiddenContentAbove,
346-
hiddenContentBelow: raw.hiddenContentBelow,
347-
actions: raw.actions
208+
private static func reindex(
209+
_ nodes: [SnapshotPresentationNode],
210+
depthOffset: Int
211+
) -> [SnapshotPresentationNode] {
212+
let indexMap = Dictionary(
213+
uniqueKeysWithValues: nodes.enumerated().map { ($0.element.raw.index, $0.offset) })
214+
return nodes.enumerated().map { offset, node in
215+
let raw = node.raw
216+
return SnapshotPresentationNode(
217+
raw: RawAXNode(
218+
index: offset,
219+
type: raw.type,
220+
label: raw.label,
221+
identifier: raw.identifier,
222+
value: raw.value,
223+
rect: raw.rect,
224+
enabled: raw.enabled,
225+
focused: raw.focused,
226+
selected: raw.selected,
227+
hittable: raw.hittable,
228+
depth: max(0, raw.depth - depthOffset),
229+
parentIndex: raw.parentIndex.flatMap { indexMap[$0] },
230+
hiddenContentAbove: raw.hiddenContentAbove,
231+
hiddenContentBelow: raw.hiddenContentBelow,
232+
actions: raw.actions
233+
),
234+
effectiveRect: node.effectiveRect
348235
)
349236
}
350237
}
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
import Foundation
2+
3+
/// The typed carrier between acquisition facts and snapshot wire projection.
4+
///
5+
/// `raw.rect` remains the backend-reported frame for runner-internal traversal and deduplication.
6+
/// `effectiveRect` is the geometry after the regular presentation policy. Raw projection sets both
7+
/// to the reported frame, so callers cannot accidentally publish an unclassified geometry source.
8+
struct SnapshotPresentationNode {
9+
let raw: RawAXNode
10+
let effectiveRect: SnapshotRect
11+
12+
init(raw: RawAXNode, effectiveRect: SnapshotRect) {
13+
self.raw = raw
14+
self.effectiveRect = effectiveRect
15+
}
16+
17+
static func reported(_ raw: RawAXNode) -> Self {
18+
Self(raw: raw, effectiveRect: raw.rect)
19+
}
20+
}
21+
22+
/// Geometry conversion owned by presentation. Acquisition reports frames; this type computes the
23+
/// only rectangle that can cross the regular snapshot wire boundary.
24+
enum SnapshotGeometry {
25+
static func effectiveFrame(
26+
reportedFrame: CGRect,
27+
viewport: CGRect,
28+
ancestorClip: CGRect?
29+
) -> CGRect {
30+
var frame = reportedFrame
31+
if !viewport.isInfinite {
32+
frame = clipped(frame, to: viewport)
33+
}
34+
if let ancestorClip {
35+
frame = clipped(frame, to: ancestorClip)
36+
}
37+
return frame
38+
}
39+
40+
static func snapshotRect(from frame: CGRect, reportedFrame: CGRect) -> SnapshotRect {
41+
guard !frame.isNull, !frame.isEmpty else {
42+
return SnapshotRect(
43+
x: Double(reportedFrame.minX), y: Double(reportedFrame.minY), width: 0, height: 0)
44+
}
45+
return SnapshotRect(
46+
x: Double(frame.origin.x),
47+
y: Double(frame.origin.y),
48+
width: Double(max(0, frame.size.width)),
49+
height: Double(max(0, frame.size.height))
50+
)
51+
}
52+
53+
private static func clipped(_ frame: CGRect, to clip: CGRect) -> CGRect {
54+
guard !frame.isNull, !frame.isEmpty else { return frame }
55+
let intersection = frame.intersection(clip)
56+
guard !intersection.isNull, !intersection.isEmpty else {
57+
return CGRect(x: frame.minX, y: frame.minY, width: 0, height: 0)
58+
}
59+
return intersection
60+
}
61+
}

0 commit comments

Comments
 (0)