Skip to content

Commit 73db7be

Browse files
authored
feat(ios): move the regular-projection clip fold into snapshot presentation (#1797) (#1929)
* feat(ios): move the regular-projection clip fold into snapshot presentation Both iOS snapshot backends carried their own copy of the visibility fold: the tree walker and the private-AX serializer each computed viewport-and-scroll-clip intersection, ancestor projection, hidden-content hints, and collapsed depth during acquisition. Hand-synchronized copies of that interpretation are what produced the scroll-overflow leak class (#1784), and C1 (fact-availability neutrality) could not hold while acquisition decided what a screen shows. Acquisition backends are now fact serializers: every traversed node is emitted at raw traversal depth with its reported frame, and SnapshotAcquisition carries the viewport. presentRegular runs the one clip fold for every backend -- viewport ∩ scroll clip, the ancestor cursor (an out-of-clip Cell or scroll container hides its clamped descendants), the sub-pixel decoration rule, scroll hints booked onto anchors, reparenting with collapsed depth -- and narrows the emitted hittable to the clip: nothing outside its clip, and nothing without geometry, is ever hittable, whatever the backend reported. Platform differences are a SnapshotFoldPolicy input to the shared algorithm (iOS cursor-projected; macOS/tvOS plain viewport), never a backend exception. The private-AX backend collapses to ONE serializer for both projections, and the flat filter-decision family dies with the acquisition gates it fed. Three intentional edge deltas, each toward one backend-neutral rule: sub-pixel content-free decorations now drop on every backend (was private-AX only); labeled offscreen Application/Window carriers survive on every backend (was tree only), never hittable; query-sweep regular without -i is viewport-folded. Declared acquisition residues: the traversal-depth budget cut, the sweep's frameless-element drop, the private-AX bridge's device-side cap. Refs #1797 (migration step 3, clip-fold delta). * refactor(ios): isolate snapshot visibility fold
1 parent 8e148e2 commit 73db7be

13 files changed

Lines changed: 673 additions & 792 deletions

‎.github/workflows/ios.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,8 +155,13 @@ jobs:
155155
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotPresentationOwnsScopeAndRelativeDepth \
156156
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPresentationRefusesAnAcquisitionCapturedForTheOtherProjection \
157157
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testCaptureHintIsTheOnlyAcquisitionViewOfARequest \
158+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldClipsScrollOverflowReparentsAndBooksHints \
159+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldKeepsWindowCarriersButNeverHittableOutsideClip \
160+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularFoldDropsSubPixelContentlessDecorationOnEveryBackend \
161+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPlainViewportPolicyFoldsWithoutAncestorCursor \
162+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRegularPresentationRoutesThroughVisibilityFold \
158163
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotScopePolicyMatchesGoldenParityTable \
159-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testFlatSnapshotProjectionMatchesElementReverseScrollCapture \
164+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testScrollContainerTypeNamesMatchElementTypeSet \
160165
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXDepthLimitedRequiresEveryFrontierResolved \
161166
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDeepExtensionCountsMissedFrontiers \
162167
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPreferredPrivateAXBackendPlansAsPenalized \

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
88
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
99
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
10+
- iOS regular snapshots now run one shared clip fold inside presentation for every capture backend (#1797). Backends serialize reported facts -- every traversed node, at raw traversal depth -- and presentation alone decides what the viewport and scroll clips hide, books the scroll hints, and collapses depth; no backend carries its own copy of that interpretation anymore (the copies are what produced the scroll-overflow leak class, #1784). Three intentional edge deltas ride along, all in the direction of one backend-neutral rule: sub-pixel content-free decorations are now dropped by every backend (previously private-AX only); labeled offscreen Application/Window carriers now survive on every backend (previously tree only), still never hittable; and a query-sweep recovery snapshot without `-i` no longer lists offscreen elements. Nothing outside its clip, and nothing without geometry, is ever `hittable` in a regular snapshot, whatever the backend reported.
1011
- iOS `snapshot --raw` is now the acquired accessibility tree on every backend that can serve it (#1797). A raw request that recovered onto the private-AX backend — the route an app whose XCTest tree capture fails takes — returned the *regular* projection's viewport-pruned nodes labeled raw: everything scrolled out of the viewport, and every sub-pixel decoration, was missing from the one view whose purpose is showing what the pruned view hid. Raw now keeps every node the backend serialized, at traversal depth, and `--depth` still narrows it (for raw, presented depth *is* traversal depth). Two structural rules replace the hand-synchronized ones: the raw capture plan is derived from each backend's declared ability to serve raw, so the interactive query sweep — which has no hierarchy to return — cannot be planned for a raw request; and presentation refuses an acquisition captured for the other projection instead of relabeling it, dropping that tier with a structured failure. Breaking in the same direction: `snapshot --raw -i` now returns the acquired tree instead of an interactive-filtered one — `-i` narrows the regular projection, and the pair used to produce a third membership rule that differed per backend. Regular and `-i` output is unchanged. Backends now read one derived capture hint rather than the request itself, so what a capture is allowed to skip is stated once, next to the proof that skipping it keeps the projection complete.
1112
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
1213
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.

‎CONTEXT.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,11 @@ narrowing complete.
198198
**Snapshot eligibility**:
199199
Membership in a presented snapshot projection, independent of whether a node is currently hittable.
200200

201+
**Clip fold**:
202+
The regular projection's single visibility interpreter, run inside presentation for every backend:
203+
viewport and scroll-container clipping, ancestor projection, scroll hints, and collapsed depth.
204+
Platform differences enter as a fold policy, never as a backend exception.
205+
201206
**Presented node**:
202207
A wire-facing snapshot value produced at the presentation boundary.
203208

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,8 @@ extension RunnerTests {
184184
hint: hint,
185185
viewport: viewport
186186
)
187+
// Serialization-level emptiness only: an acquired-but-fully-clipped tree is presentation's
188+
// verdict now, surfaced by the plan's sparse classifier on the presented payload (#1797).
187189
if nodes.count <= 1 {
188190
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_SNAPSHOT_SPARSE=%ld", nodes.count)
189191
return nil
@@ -213,7 +215,8 @@ extension RunnerTests {
213215
effectiveDepth: depthLimited ? effectiveDepth : nil,
214216
customActions: Self.privateAXCustomActionCoverage(
215217
response[RunnerAXSnapshotCustomActionsKey]
216-
)
218+
),
219+
viewport: viewport
217220
)
218221
#else
219222
return nil
@@ -752,14 +755,20 @@ extension RunnerTests {
752755
]
753756
],
754757
]
755-
let nodes = privateAXAcquisition(
756-
rawRoot: tree,
757-
hint: CaptureHint(
758-
projection: .regular, depth: nil, interactiveOnly: true, customActions: false),
759-
viewport: CGRect(x: 0, y: 0, width: 390, height: 844)
758+
let viewport = CGRect(x: 0, y: 0, width: 390, height: 844)
759+
let hint = CaptureHint(
760+
projection: .regular, depth: nil, interactiveOnly: true, customActions: false)
761+
let acquired = privateAXAcquisition(rawRoot: tree, hint: hint, viewport: viewport)
762+
// Acquisition serializes the drawer too; the shared fold is what hides it (#1797).
763+
XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings"))
764+
765+
let capture = SnapshotPresentation.presentRegular(
766+
SnapshotAcquisition(
767+
hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport),
768+
options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false),
769+
policy: .cursorProjected
760770
)
761-
762-
let labels = nodes.compactMap { $0.label }
771+
let labels = (capture.payload.nodes ?? []).compactMap { $0.label }
763772
XCTAssertEqual(
764773
labels,
765774
["Blue Sky", "Callstack", "Welcome back", "Email", "Password", "Sign in", "Forgot password?"]

0 commit comments

Comments
 (0)