Skip to content

Commit 84e6f9b

Browse files
authored
refactor(android): raw is the acquired tree; one presentation for dialog recovery; residues declared (#1832 C3) (#1865)
* refactor(android): raw is the acquired tree; one presentation for dialog recovery; residues declared (#1832 C3) - C3: the three regular-projection pruners (invisible subtrees, stale application windows, covered same-window surfaces) move out of parseUiHierarchyTree into the projection as a non-mutating classification (collectAndroidHiddenNodes in ui-hierarchy-visibility.ts). --raw presents the acquired tree; interactive ⊆ regular ⊆ raw by construction. Hidden-content hints and the scope root are derived per projection over retained children, which is what the mutating pruners implied. Property-checked identical to main for regular/-i/depth/scope over 12,000 random tree × projection pairs; raw grew on 2,514/3,000 and never shrank. - Android blocking-dialog recovery routes through buildSnapshotState (the one presentation), which moves to src/daemon/snapshot-state.ts below the daemon-server type cycle; importers repointed, its tests mirror the module. - Freshness route signature drops role/selected (Android never carries them). - Residues declared at their sites and in CONTEXT.md; docs + CHANGELOG. - ui-hierarchy.ts split by question: node predicates (ui-hierarchy-node.ts), regular-projection visibility (ui-hierarchy-visibility.ts), scope (ui-hierarchy-scope.ts); 974 → 634 LOC. * test: lower the snapshot.test.ts size pin to its new length * fix(android): dialog recovery acts on the presentation's occlusion result Review P1 on #1865: routing blocking-dialog recovery through buildSnapshotState made the occlusion result available but nothing consumed it. containsBlockingDialog scanned every node and findCloseAppButton returned the first text match with a rect, so a stale ANR surface left under the foreground one could still trigger recovery, and a covered "Close app" could be tapped ahead of the visible top button — the disagreement the routing was supposed to remove. Both decisions now filter through isSnapshotNodeInteractionBlocked, the shared predicate over the annotator's structured result. Two regressions cover it, both proven red against an unfiltered selection: a covered Close app preceding a visible one (asserts the visible center is tapped) and a fully covered dialog signal (asserts recovery does not trigger, no tap dispatched). Rebase reconciliation: screenshot-runtime.ts arrived on main (#1878) importing buildSnapshotState from its old home; repointed to src/daemon/snapshot-state.ts with the other importers.
1 parent 139e3aa commit 84e6f9b

28 files changed

Lines changed: 1184 additions & 917 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
77
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
88
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
9+
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
910
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
1011
- Android `snapshot --scope` (and every selector command's `--scope`, e.g. `press "Save" --scope Panel`) now resolves scope exactly once, inside the Android projection, under the shared scope specification: the scope root is the first node **in document order** whose label, value, or identifier contains the scope text (case-insensitive) **and whose subtree still has content in the projection you asked for**, the result is that subtree re-rooted at depth 0, and no match returns an empty snapshot (#1832). That second clause is what makes `snapshot -i --scope panel` return the button inside a structural container `-i` drops, and stops a decorative heading that happens to match from emptying the snapshot. Before, Android ran two passes with contradictory rules — a breadth-first platform match that fell back to the full tree on a miss, then the daemon's document-order pass — so a shallower later container could win over an earlier match, and an interaction capture whose scope reached only the daemon layer was silently unscoped. `--depth` under `--scope` counts from the scope root, filtering the depths the response prints (a node shown at depth 0 is never hidden by `--depth 0`), and ancestor context above the scope root (a clickable row, a list) still shapes `-i` membership inside it. The rule is pinned by `contracts/fixtures/snapshot-scope-policy.json`, the same golden table the iOS runner will consume (#1797). Also new: `androidSnapshot.occlusionScanUnavailable: true` discloses an API 23 capture, where the helper cannot report `drawing-order` and covered same-window surfaces are therefore not pruned.
1112
- New `hover <x y|@ref|selector>` command for `--platform web` (#1783). It moves the pointer over the target without pressing, so hover-gated UI — a message row's `...` toolbar, a menu that opens on pointer enter — becomes reachable through agent-device the way it already was through the underlying `agent-browser` backend (`mouse move`). It is a member of the targeted-touch family: same `@ref`/selector/coordinate targeting, occlusion and off-screen guards, and `--settle` (the settled diff carries the revealed controls with fresh refs, e.g. `+ @e4 [button] "Delete"`), but no `--verify`, since hover reveals rather than activates. `hover @ref` publishes as a portable selector line in recorded scripts, and the Node client exposes `interactions.hover`. Hover is a pointer state that touch platforms do not have, so `capabilities` advertises it on web only and iOS/Android/Linux reject it during admission with `UNSUPPORTED_OPERATION` and a hint naming `--platform web`; `longpress` remains the mobile hold-gesture verb.

‎CONTEXT.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,21 @@ task touches:
241241
typed `WebView` root and WebKit's `Other -> StaticText` wrapper pairs. It keeps raw diagnostics
242242
unchanged, presents ordinary wrapper text as `StaticText`, and presents wrappers carrying WebKit's
243243
numeric HTML heading level as `Heading`.
244+
- Android snapshot projections (#1832 C3): the parsed helper tree is the acquired tree and is never
245+
mutated. `--raw` presents it whole (normalization only). The regular projection additionally hides
246+
what `collectAndroidHiddenNodes` classifies — nodes Android marks invisible to users, stale
247+
application windows, and same-window surfaces a higher drawing-order sibling covers — and the
248+
interactive projection applies membership on top. `interactive ⊆ regular ⊆ raw` holds by
249+
construction; hidden-content hints and the scope root are derived per projection.
250+
- Android declared residues (#1832): fidelity limits of the helper acquisition that no presentation
251+
can repair, disclosed rather than papered over. (1) API 23 helper trees carry no `drawing-order`,
252+
so the covered-surface rule cannot run — `androidSnapshot.occlusionScanUnavailable`. (2) The
253+
helper serializes no `checked`/`checkable`/`selected`/`long-clickable`, so toggle and selection
254+
state is invisible and nothing on Android may key on those fields (the freshness route signature
255+
does not). (3) The helper caps at 5000 nodes before any projection, so a scoped target past the
256+
cap vanishes with only the generic `truncated`. (4) Accessibility-cache reset differs by API level
257+
(`clearCache()` on 34+, `setServiceInfo` re-apply below), an undisclosed acquisition-freshness
258+
difference.
244259
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot
245260
failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect.
246261
The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeCapture.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ private static void clearAccessibilityCache(UiAutomation automation) {
6969
// destination inside one AndroidComposeView. A persistent helper session reuses a single
7070
// UiAutomation connection across captures, and its per-connection accessibility node cache is
7171
// not always invalidated by such a swap. Drop the cache before each traversal.
72+
//
73+
// Declared residue (agent-device #1832): the reset mechanism differs by API level — the public
74+
// clearCache() on 34+, a setServiceInfo() re-apply below — and the two are not known to be
75+
// equivalent. This is an acquisition-freshness difference the payload does not disclose.
7276
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
7377
try {
7478
automation.clearCache();

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeXml.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,9 @@ private static void appendTrueAttribute(StringBuilder xml, String name, boolean
121121
}
122122
}
123123

124+
// Declared residue (agent-device #1832): checked / checkable / selected / long-clickable are not
125+
// serialized, so toggle and selection state is invisible to agents. Adding them is a helper
126+
// protocol change (new attributes + host parser + fields on the wire node), tracked there.
124127
private static void appendDrawingOrderAttribute(StringBuilder xml, AccessibilityNodeInfo node) {
125128
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
126129
appendAttribute(xml, "drawing-order", Integer.toString(node.getDrawingOrder()));

‎src/__tests__/test-file-size-ratchet.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ const PINNED_TEST_FILE_LINES: Readonly<Record<string, number>> = Object.freeze({
4040
'src/platforms/apple/core/__tests__/runner-session.test.ts': 2001,
4141
'src/utils/__tests__/daemon-client.test.ts': 1910,
4242
'src/utils/__tests__/output.test.ts': 1861,
43-
'src/platforms/android/__tests__/snapshot.test.ts': 1660,
43+
'src/platforms/android/__tests__/snapshot.test.ts': 1658,
4444
'src/platforms/apple/core/__tests__/runner-client.test.ts': 1615,
4545
'src/__tests__/client.test.ts': 1598,
4646
'test/integration/provider-scenarios/android-lifecycle.test.ts': 1559,
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
import { test, expect, vi } from 'vitest';
2+
3+
// #1832: blocking-dialog recovery reads the daemon presentation, so it must also consume that
4+
// presentation's OCCLUSION result. A stale "App isn't responding" surface left under the foreground
5+
// one must not trigger recovery, and a covered "Close app" must not be tapped ahead of the visible
6+
// one — the exact disagreement routing through buildSnapshotState exists to remove.
7+
vi.mock('../../platforms/android/snapshot.ts', () => ({ snapshotAndroid: vi.fn() }));
8+
vi.mock('../../platforms/android/adb.ts', () => ({ runAndroidAdb: vi.fn() }));
9+
10+
import { snapshotAndroid } from '../../platforms/android/snapshot.ts';
11+
import { runAndroidAdb } from '../../platforms/android/adb.ts';
12+
import { recoverAndroidBlockingSystemDialog } from '../android-system-dialog.ts';
13+
import { makeAndroidSession } from '../../__tests__/test-utils/session-factories.ts';
14+
import { makeTestScreenRecordingResource } from '../../__tests__/test-utils/screen-recording-live-handle.ts';
15+
16+
const SCREEN = { x: 0, y: 0, width: 1080, height: 2400 };
17+
18+
/** A foreground surface drawn over everything below it — what makes the stale dialog unreachable. */
19+
const foregroundSheet = (index: number) => ({
20+
index,
21+
type: 'com.google.android.material.bottomsheet.BottomSheetDialog',
22+
identifier: 'com.example:id/bottom_sheet',
23+
hittable: true,
24+
rect: SCREEN,
25+
});
26+
27+
function recordingSession(name: string) {
28+
const session = makeAndroidSession(name);
29+
session.screenRecording = makeTestScreenRecordingResource(session, {
30+
backend: 'adb screenrecord',
31+
outPath: '/tmp/anr.mp4',
32+
startedAt: 0,
33+
});
34+
return session;
35+
}
36+
37+
function tappedPoints(): Array<{ x: number; y: number }> {
38+
return vi
39+
.mocked(runAndroidAdb)
40+
.mock.calls.filter((call) => (call[1] as string[]).includes('tap'))
41+
.map((call) => {
42+
const args = call[1] as string[];
43+
const at = args.indexOf('tap');
44+
return { x: Number(args[at + 1]), y: Number(args[at + 2]) };
45+
});
46+
}
47+
48+
test('recovery taps the reachable Close app, not a covered one that comes first', async () => {
49+
vi.mocked(runAndroidAdb).mockReset();
50+
// The live ANR dialog is on top; a stale "Close app" from an earlier one is still in the tree
51+
// underneath the sheet. Document order puts the stale button FIRST, which is what the previous
52+
// hand-rolled presentation selected: the first text match carrying a rect.
53+
const staleCloseApp = {
54+
index: 0,
55+
type: 'Button',
56+
label: 'Close app',
57+
hittable: true,
58+
rect: { x: 40, y: 300, width: 200, height: 80 },
59+
};
60+
const sheet = { ...foregroundSheet(1), rect: { x: 0, y: 0, width: 1080, height: 1000 } };
61+
const dialog = {
62+
index: 2,
63+
type: 'TextView',
64+
label: "App isn't responding",
65+
rect: { x: 0, y: 1700, width: 1080, height: 60 },
66+
};
67+
const visibleCloseApp = {
68+
index: 3,
69+
type: 'Button',
70+
label: 'Close app',
71+
hittable: true,
72+
rect: { x: 40, y: 1800, width: 200, height: 80 },
73+
};
74+
vi.mocked(snapshotAndroid)
75+
.mockResolvedValueOnce({ nodes: [staleCloseApp, sheet, dialog, visibleCloseApp] } as never)
76+
.mockResolvedValue({ nodes: [] } as never);
77+
vi.mocked(runAndroidAdb).mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' } as never);
78+
79+
const result = await recoverAndroidBlockingSystemDialog({
80+
session: recordingSession('anr-covered-button'),
81+
});
82+
83+
expect(result.status).toBe('recovered');
84+
expect(tappedPoints()).toEqual([{ x: 140, y: 1840 }]);
85+
});
86+
87+
test('a fully covered dialog does not trigger recovery', async () => {
88+
vi.mocked(runAndroidAdb).mockReset();
89+
const staleDialog = {
90+
index: 0,
91+
type: 'TextView',
92+
label: "App isn't responding",
93+
hittable: true,
94+
rect: { x: 0, y: 100, width: 1080, height: 60 },
95+
};
96+
const staleCloseApp = {
97+
index: 1,
98+
type: 'Button',
99+
label: 'Close app',
100+
hittable: true,
101+
rect: { x: 40, y: 300, width: 200, height: 80 },
102+
};
103+
vi.mocked(snapshotAndroid).mockResolvedValue({
104+
nodes: [staleDialog, staleCloseApp, foregroundSheet(2)],
105+
} as never);
106+
vi.mocked(runAndroidAdb).mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' } as never);
107+
108+
const result = await recoverAndroidBlockingSystemDialog({
109+
session: recordingSession('anr-covered-dialog'),
110+
});
111+
112+
expect(result.status).toBe('absent');
113+
expect(tappedPoints()).toEqual([]);
114+
});

‎src/daemon/__tests__/generic-settle.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import { activateCompleteRefFrame } from '../ref-frame.ts';
77
import { setSessionSnapshot } from '../session-snapshot.ts';
88
import type { SessionStore } from '../session-store.ts';
99
import type { DaemonRequest, DaemonResponse, SessionState } from '../types.ts';
10-
import { buildSnapshotState } from '../handlers/snapshot-capture.ts';
10+
import { buildSnapshotState } from '../snapshot-state.ts';
1111

1212
// #1638 `--settle` on the GENERIC daemon route (scroll/back): the settled diff,
1313
// its refs, and the ref-frame/generation dance are the same contract the touch

0 commit comments

Comments
 (0)