Skip to content

Commit f65590a

Browse files
committed
refactor(android): raw is the acquired tree; one presentation for dialog recovery; residues declared (#1832 C3)
- C3: the three regular-projection pruners (invisible subtrees, stale application windows, covered same-window surfaces) move out of parseUiHierarchyTree into the projection as a non-mutating classification (collectAndroidHiddenNodes in ui-hierarchy-visibility.ts). --raw presents the acquired tree; interactive ⊆ regular ⊆ raw by construction. Hidden-content hints and the scope root are derived per projection over retained children, which is what the mutating pruners implied. Property-checked identical to main for regular/-i/depth/scope over 12,000 random tree × projection pairs; raw grew on 2,514/3,000 and never shrank. - Android blocking-dialog recovery routes through buildSnapshotState (the one presentation), which moves to src/daemon/snapshot-state.ts below the daemon-server type cycle; importers repointed, its tests mirror the module. - Freshness route signature drops role/selected (Android never carries them). - Residues declared at their sites and in CONTEXT.md; docs + CHANGELOG. - ui-hierarchy.ts split by question: node predicates (ui-hierarchy-node.ts), regular-projection visibility (ui-hierarchy-visibility.ts), scope (ui-hierarchy-scope.ts); 974 → 634 LOC.
1 parent d29dc22 commit f65590a

24 files changed

Lines changed: 1052 additions & 912 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
77
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
88
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
9+
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
910
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
1011
- Android `snapshot --scope` (and every selector command's `--scope`, e.g. `press "Save" --scope Panel`) now resolves scope exactly once, inside the Android projection, under the shared scope specification: the scope root is the first node **in document order** whose label, value, or identifier contains the scope text (case-insensitive) **and whose subtree still has content in the projection you asked for**, the result is that subtree re-rooted at depth 0, and no match returns an empty snapshot (#1832). That second clause is what makes `snapshot -i --scope panel` return the button inside a structural container `-i` drops, and stops a decorative heading that happens to match from emptying the snapshot. Before, Android ran two passes with contradictory rules — a breadth-first platform match that fell back to the full tree on a miss, then the daemon's document-order pass — so a shallower later container could win over an earlier match, and an interaction capture whose scope reached only the daemon layer was silently unscoped. `--depth` under `--scope` counts from the scope root, filtering the depths the response prints (a node shown at depth 0 is never hidden by `--depth 0`), and ancestor context above the scope root (a clickable row, a list) still shapes `-i` membership inside it. The rule is pinned by `contracts/fixtures/snapshot-scope-policy.json`, the same golden table the iOS runner will consume (#1797). Also new: `androidSnapshot.occlusionScanUnavailable: true` discloses an API 23 capture, where the helper cannot report `drawing-order` and covered same-window surfaces are therefore not pruned.
1112
- New `hover <x y|@ref|selector>` command for `--platform web` (#1783). It moves the pointer over the target without pressing, so hover-gated UI — a message row's `...` toolbar, a menu that opens on pointer enter — becomes reachable through agent-device the way it already was through the underlying `agent-browser` backend (`mouse move`). It is a member of the targeted-touch family: same `@ref`/selector/coordinate targeting, occlusion and off-screen guards, and `--settle` (the settled diff carries the revealed controls with fresh refs, e.g. `+ @e4 [button] "Delete"`), but no `--verify`, since hover reveals rather than activates. `hover @ref` publishes as a portable selector line in recorded scripts, and the Node client exposes `interactions.hover`. Hover is a pointer state that touch platforms do not have, so `capabilities` advertises it on web only and iOS/Android/Linux reject it during admission with `UNSUPPORTED_OPERATION` and a hint naming `--platform web`; `longpress` remains the mobile hold-gesture verb.

‎CONTEXT.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,21 @@ task touches:
241241
typed `WebView` root and WebKit's `Other -> StaticText` wrapper pairs. It keeps raw diagnostics
242242
unchanged, presents ordinary wrapper text as `StaticText`, and presents wrappers carrying WebKit's
243243
numeric HTML heading level as `Heading`.
244+
- Android snapshot projections (#1832 C3): the parsed helper tree is the acquired tree and is never
245+
mutated. `--raw` presents it whole (normalization only). The regular projection additionally hides
246+
what `collectAndroidHiddenNodes` classifies — nodes Android marks invisible to users, stale
247+
application windows, and same-window surfaces a higher drawing-order sibling covers — and the
248+
interactive projection applies membership on top. `interactive ⊆ regular ⊆ raw` holds by
249+
construction; hidden-content hints and the scope root are derived per projection.
250+
- Android declared residues (#1832): fidelity limits of the helper acquisition that no presentation
251+
can repair, disclosed rather than papered over. (1) API 23 helper trees carry no `drawing-order`,
252+
so the covered-surface rule cannot run — `androidSnapshot.occlusionScanUnavailable`. (2) The
253+
helper serializes no `checked`/`checkable`/`selected`/`long-clickable`, so toggle and selection
254+
state is invisible and nothing on Android may key on those fields (the freshness route signature
255+
does not). (3) The helper caps at 5000 nodes before any projection, so a scoped target past the
256+
cap vanishes with only the generic `truncated`. (4) Accessibility-cache reset differs by API level
257+
(`clearCache()` on 34+, `setServiceInfo` re-apply below), an undisclosed acquisition-freshness
258+
difference.
244259
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot
245260
failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect.
246261
The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeCapture.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ private static void clearAccessibilityCache(UiAutomation automation) {
6969
// destination inside one AndroidComposeView. A persistent helper session reuses a single
7070
// UiAutomation connection across captures, and its per-connection accessibility node cache is
7171
// not always invalidated by such a swap. Drop the cache before each traversal.
72+
//
73+
// Declared residue (agent-device #1832): the reset mechanism differs by API level — the public
74+
// clearCache() on 34+, a setServiceInfo() re-apply below — and the two are not known to be
75+
// equivalent. This is an acquisition-freshness difference the payload does not disclose.
7276
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
7377
try {
7478
automation.clearCache();

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeXml.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,9 @@ private static void appendTrueAttribute(StringBuilder xml, String name, boolean
121121
}
122122
}
123123

124+
// Declared residue (agent-device #1832): checked / checkable / selected / long-clickable are not
125+
// serialized, so toggle and selection state is invisible to agents. Adding them is a helper
126+
// protocol change (new attributes + host parser + fields on the wire node), tracked there.
124127
private static void appendDrawingOrderAttribute(StringBuilder xml, AccessibilityNodeInfo node) {
125128
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
126129
appendAttribute(xml, "drawing-order", Integer.toString(node.getDrawingOrder()));

‎src/daemon/__tests__/generic-settle.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import { activateCompleteRefFrame } from '../ref-frame.ts';
77
import { setSessionSnapshot } from '../session-snapshot.ts';
88
import type { SessionStore } from '../session-store.ts';
99
import type { DaemonRequest, DaemonResponse, SessionState } from '../types.ts';
10-
import { buildSnapshotState } from '../handlers/snapshot-capture.ts';
10+
import { buildSnapshotState } from '../snapshot-state.ts';
1111

1212
// #1638 `--settle` on the GENERIC daemon route (scroll/back): the settled diff,
1313
// its refs, and the ref-frame/generation dance are the same contract the touch

0 commit comments

Comments
 (0)