Skip to content

Commit 4137e42

Browse files
authored
feat(ios): split raw and regular snapshot projections behind one capture hint (#1926)
The private-AX backend interpreted `--raw` as the regular projection: it folded the viewport and scroll clips and dropped sub-pixel decorations before returning, so a raw capture that recovered onto it answered with viewport-pruned nodes labeled raw (#1797 D4). Nothing related the two copies of that decision. Presentation now exposes the two projections it always implied. `presentRegular` folds visibility, eligibility, scope and scroll hints; `presentRaw` is the acquired tree, normalized, narrowed only by a scope or depth the request asked for. Acquisition reads one derived `CaptureHint` instead of the request itself, so what a capture may skip is stated once, beside the reason skipping it keeps the projection complete: scope and its relative depth never narrow, raw depth does (raw depth is traversal depth), and the raw projection never carries `interactiveOnly` — `--raw -i` is the acquired tree. Two structural rules replace the hand-synchronized ones. The raw plan is derived from `SnapshotBackendKind.supportsRawProjection`, so the query sweep — an interactive element query with no hierarchy to return — cannot be planned for a raw request. And presentation compares the requested projection with the hint the acquisition was captured under, dropping that tier with a structured `IOS_SNAPSHOT_PROJECTION_MISMATCH` failure rather than presenting it under the requested label. Declared residue: a regular `--depth` request still cuts the traversal at that depth while regular presentation emits collapsed depth, so a node that would present within the limit can be dropped. The cut is what keeps `--depth 1` probes cheap; making it complete is the open visible-depth frontier obligation. Refs #1797 (migration step 3, raw-projection delta).
1 parent aed00ae commit 4137e42

12 files changed

Lines changed: 564 additions & 145 deletions

‎.github/workflows/ios.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,12 +151,18 @@ jobs:
151151
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationPreservesCurrentWireShape \
152152
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsBackendNeutralEligibility \
153153
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \
154+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \
155+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \
154156
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \
155157
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \
156158
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \
157159
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \
158160
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \
161+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRawDiagnosticPlanCarriesOnlyBackendsThatCanServeRaw \
162+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testProjectionMismatchFailureIsStructuredAndNotAnAxFailure \
159163
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRegularPresentationProjectsToViewportAndKeepsScrollHint \
164+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRawProjectionKeepsEveryAcquiredNode \
165+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXRawProjectionAppliesRequestedTraversalDepth \
160166
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXPresentationKeepsOffscreenSubtreeExcludedWhenChildFramesAreClamped \
161167
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXGeometrylessSemanticsAreNeverActionableOrScrollContexts \
162168
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXAcquisitionDoesNotInterpretScope \

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
88
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
99
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
10+
- iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete.
1011
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
1112
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
1213
- Android `snapshot --scope` (and every selector command's `--scope`, e.g. `press "Save" --scope Panel`) now resolves scope exactly once, inside the Android projection, under the shared scope specification: the scope root is the first node **in document order** whose label, value, or identifier contains the scope text (case-insensitive) **and whose subtree still has content in the projection you asked for**, the result is that subtree re-rooted at depth 0, and no match returns an empty snapshot (#1832). That second clause is what makes `snapshot -i --scope panel` return the button inside a structural container `-i` drops, and stops a decorative heading that happens to match from emptying the snapshot. Before, Android ran two passes with contradictory rules — a breadth-first platform match that fell back to the full tree on a miss, then the daemon's document-order pass — so a shallower later container could win over an earlier match, and an interaction capture whose scope reached only the daemon layer was silently unscoped. `--depth` under `--scope` counts from the scope root, filtering the depths the response prints (a node shown at depth 0 is never hidden by `--depth 0`), and ancestor context above the scope root (a clickable row, a list) still shapes `-i` membership inside it. The rule is pinned by `contracts/fixtures/snapshot-scope-policy.json`, the same golden table the iOS runner will consume (#1797). Also new: `androidSnapshot.occlusionScanUnavailable: true` discloses an API 23 capture, where the helper cannot report `drawing-order` and covered same-window surfaces are therefore not pruned.

‎CONTEXT.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,11 @@ One backend attempt's raw accessibility nodes and attempt-level capture facts.
190190
**Presentation options**:
191191
The policy input controlling how one snapshot acquisition becomes a public projection.
192192

193+
**Capture hint**:
194+
The acquisition-facing view of a snapshot request, derived once from presentation options. It names
195+
the projection a backend must serve and may narrow acquisition only where that backend can prove the
196+
narrowing complete.
197+
193198
**Snapshot eligibility**:
194199
Membership in a presented snapshot projection, independent of whether a node is currently hittable.
195200

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -105,14 +105,14 @@ extension RunnerTests {
105105

106106
func privateAXSnapshotAcquisition(
107107
app: XCUIApplication,
108-
options: PresentationOptions,
108+
hint: CaptureHint,
109109
deadline: Date = .distantFuture
110110
) -> SnapshotAcquisition? {
111111
#if os(iOS) && targetEnvironment(simulator)
112-
let requestedDepth = options.depth ?? 64
112+
let requestedDepth = hint.depth ?? 64
113113
// An explicit --depth request is honored as asked: no accepted-depth
114114
// memory, no frontier extension past it.
115-
let exactDepthRequested = options.depth != nil
115+
let exactDepthRequested = hint.depth != nil
116116
let rememberedDepth =
117117
exactDepthRequested
118118
? nil
@@ -135,12 +135,16 @@ extension RunnerTests {
135135
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_BUDGET_EXHAUSTED depth=%ld", depth)
136136
break
137137
}
138+
// Declared residue (#1797): the bridge caps the tree at 5000 nodes while serializing,
139+
// BEFORE either projection exists, so a raw capture of a huge screen is bounded rather
140+
// than failing the way the tree backend's own raw cap does. The cap is disclosed as
141+
// `truncated`, and it applied to the acquired tree before this projection split too.
138142
response = RunnerAXSnapshotBridge.snapshotTree(
139143
for: app,
140144
maxDepth: depth,
141145
maxNodes: Self.privateAXSnapshotMaxNodes,
142146
deepExtensionCallLimit: exactDepthRequested ? 0 : Self.privateAXDeepExtensionCallLimit,
143-
customActionLimit: options.customActions ? Self.privateAXCustomActionLimit : 0,
147+
customActionLimit: hint.customActions ? Self.privateAXCustomActionLimit : 0,
144148
deadline: deadline
145149
)
146150
if response["ok"] as? Bool == true {
@@ -175,9 +179,9 @@ extension RunnerTests {
175179

176180
let rootFrame = privateAXRect(root["frame"])
177181
let viewport = privateAXSnapshotViewport(app: app, rootFrame: rootFrame)
178-
let nodes = privateAXPresentation(
182+
let nodes = privateAXAcquisition(
179183
rawRoot: root,
180-
options: options,
184+
hint: hint,
181185
viewport: viewport
182186
)
183187
if nodes.count <= 1 {
@@ -203,6 +207,7 @@ extension RunnerTests {
203207
deepExtension?[RunnerAXSnapshotDeepExtensionNodesAddedKey] as? Int ?? 0
204208
)
205209
return SnapshotAcquisition(
210+
hint: hint,
206211
nodes: nodes,
207212
truncated: (response["truncated"] as? Bool) == true,
208213
effectiveDepth: depthLimited ? effectiveDepth : nil,
@@ -635,9 +640,10 @@ extension RunnerTests {
635640
],
636641
],
637642
]
638-
let nodes = privateAXPresentation(
643+
let nodes = privateAXAcquisition(
639644
rawRoot: tree,
640-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
645+
hint: CaptureHint(
646+
projection: .regular, depth: nil, interactiveOnly: false, customActions: false),
641647
viewport: CGRect(x: 0, y: 0, width: 390, height: 844)
642648
)
643649

@@ -660,13 +666,17 @@ extension RunnerTests {
660666
["type": 9, "label": "unrelated sibling", "children": []],
661667
],
662668
]
663-
let nodes = privateAXPresentation(
669+
// Scope never reaches acquisition: the hint derived for a scoped request carries no scope,
670+
// and the backend has no way to interpret one.
671+
let nodes = privateAXAcquisition(
664672
rawRoot: tree,
665-
options: PresentationOptions(
666-
interactiveOnly: false,
667-
depth: nil,
668-
scope: "homeScreen",
669-
raw: false
673+
hint: SnapshotPresentation.captureHint(
674+
for: PresentationOptions(
675+
interactiveOnly: false,
676+
depth: nil,
677+
scope: "homeScreen",
678+
raw: false
679+
)
670680
),
671681
viewport: .infinite
672682
)
@@ -742,9 +752,10 @@ extension RunnerTests {
742752
]
743753
],
744754
]
745-
let nodes = privateAXPresentation(
755+
let nodes = privateAXAcquisition(
746756
rawRoot: tree,
747-
options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false),
757+
hint: CaptureHint(
758+
projection: .regular, depth: nil, interactiveOnly: true, customActions: false),
748759
viewport: CGRect(x: 0, y: 0, width: 390, height: 844)
749760
)
750761

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+FlatSnapshotFiltering.swift‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -108,16 +108,18 @@ extension RunnerTests {
108108
)
109109
}
110110

111+
/// Regular-projection acquisition gate. The raw projection never reaches it: raw acquisition is
112+
/// the acquired tree, so it has no visibility filter to consult (#1797 D4).
111113
func flatSnapshotFilterDecision(
112114
_ node: FlatSnapshotFilterNode,
113-
options: PresentationOptions,
115+
hint: CaptureHint,
114116
visibilityPolicy: FlatSnapshotVisibilityPolicy
115117
) -> FlatSnapshotFilterDecision {
116118
let include: Bool
117119
if node.isRoot {
118120
include = true
119121
} else if !node.visible
120-
&& (options.interactiveOnly || visibilityPolicy == .viewportProjected)
122+
&& (hint.interactiveOnly || visibilityPolicy == .viewportProjected)
121123
{
122124
include = false
123125
} else {
@@ -285,6 +287,10 @@ extension RunnerTests {
285287
}
286288

287289
func testFlatSnapshotFilterDecisionMatrixCoversOptions() {
290+
func hint(interactiveOnly: Bool) -> CaptureHint {
291+
CaptureHint(
292+
projection: .regular, depth: nil, interactiveOnly: interactiveOnly, customActions: false)
293+
}
288294
let visibleContent = FlatSnapshotFilterNode(
289295
isRoot: false,
290296
visible: true
@@ -305,42 +311,42 @@ extension RunnerTests {
305311
XCTAssertTrue(
306312
flatSnapshotFilterDecision(
307313
visibleContent,
308-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
314+
hint: hint(interactiveOnly: false),
309315
visibilityPolicy: .interactiveOnly
310316
).include
311317
)
312318
XCTAssertFalse(
313319
flatSnapshotFilterDecision(
314320
hiddenInteractive,
315-
options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false),
321+
hint: hint(interactiveOnly: true),
316322
visibilityPolicy: .interactiveOnly
317323
).include
318324
)
319325
XCTAssertFalse(
320326
flatSnapshotFilterDecision(
321327
hiddenInteractive,
322-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
328+
hint: hint(interactiveOnly: false),
323329
visibilityPolicy: .viewportProjected
324330
).include
325331
)
326332
XCTAssertTrue(
327333
flatSnapshotFilterDecision(
328334
hiddenInteractive,
329-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
335+
hint: hint(interactiveOnly: false),
330336
visibilityPolicy: .interactiveOnly
331337
).include
332338
)
333339
XCTAssertTrue(
334340
flatSnapshotFilterDecision(
335341
hiddenRoot,
336-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
342+
hint: hint(interactiveOnly: false),
337343
visibilityPolicy: .viewportProjected
338344
).include
339345
)
340346
XCTAssertTrue(
341347
flatSnapshotFilterDecision(
342348
decorative,
343-
options: PresentationOptions(interactiveOnly: false, depth: nil, scope: nil, raw: false),
349+
hint: hint(interactiveOnly: false),
344350
visibilityPolicy: .interactiveOnly
345351
).include
346352
)

0 commit comments

Comments
 (0)