Skip to content

Commit e1b7be6

Browse files
committed
fix(android): carry field facts through the attrs digest and selection offsets past editability
- `get attrs --level digest` kept only the pre-#2288 semantic fields, so `editable`/`password`/`hintShowing`/`selectionStart`/`selectionEnd` vanished on the token-cheap route. The digest now keeps them, with a regression covering explicit false/zero/empty and omission when unavailable. - The helper emitted selection offsets only inside `isEditable()`, but read-only selectable text exposes a selection too. Each nonnegative offset is now emitted independently; -1 stays absent. Parser-to-snapshot regression for a non-editable selectable node. - Docs: the field-metadata notes get their own section instead of leading the efficiency tips. - Dropped the test-isolation commit: main already mocks notifyIosRunnerAppRelaunched, and the lifecycle test passes without it.
1 parent b4aa7a3 commit e1b7be6

5 files changed

Lines changed: 68 additions & 21 deletions

File tree

‎android/snapshot-helper/src/main/java/com/callstack/agentdevice/snapshothelper/AccessibilityTreeXml.java‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -45,15 +45,10 @@ static void appendNode(
4545
appendAttribute(xml, "hint-showing", Boolean.toString(node.isShowingHintText()));
4646
}
4747
appendAttribute(xml, "editable", Boolean.toString(node.isEditable()));
48-
if (node.isEditable()) {
49-
// These are accessibility offsets, not a measurement of the entered value's length.
50-
int selectionStart = node.getTextSelectionStart();
51-
int selectionEnd = node.getTextSelectionEnd();
52-
if (selectionStart >= 0 && selectionEnd >= 0) {
53-
appendAttribute(xml, "selection-start", Integer.toString(selectionStart));
54-
appendAttribute(xml, "selection-end", Integer.toString(selectionEnd));
55-
}
56-
}
48+
// Accessibility selection offsets, not a measurement of the value's length. Read-only
49+
// selectable text exposes a selection too, so they do not depend on `editable`; -1 = unavailable.
50+
appendNonNegativeAttribute(xml, "selection-start", node.getTextSelectionStart());
51+
appendNonNegativeAttribute(xml, "selection-end", node.getTextSelectionEnd());
5752
appendNonEmptyAttribute(xml, "resource-id", node.getViewIdResourceName());
5853
appendAttribute(xml, "class", node.getClassName());
5954
appendNonEmptyAttribute(xml, "package", node.getPackageName());
@@ -134,6 +129,12 @@ private static void appendNonEmptyAttribute(
134129
appendAttribute(xml, name, value);
135130
}
136131

132+
private static void appendNonNegativeAttribute(StringBuilder xml, String name, int value) {
133+
if (value >= 0) {
134+
appendAttribute(xml, name, Integer.toString(value));
135+
}
136+
}
137+
137138
private static void appendTrueAttribute(StringBuilder xml, String name, boolean value) {
138139
if (value) {
139140
appendAttribute(xml, name, "true");

‎packages/platform-android/src/__tests__/ui-hierarchy-field-metadata.test.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,18 @@ test.each([false, true])(
2020
},
2121
);
2222

23+
test('selection offsets survive on a read-only selectable node (independent of editable)', () => {
24+
const tree = parseUiHierarchyTree(`<hierarchy><node class="android.widget.TextView"
25+
resource-id="field" text="Read-only text" bounds="[0,0][200,100]" visible-to-user="true"
26+
editable="false" selection-start="2" selection-end="5" /></hierarchy>`);
27+
const { nodes } = buildUiHierarchySnapshot(tree, undefined, { raw: true });
28+
expect(nodes.find((node) => node.identifier === 'field')).toMatchObject({
29+
editable: false,
30+
selectionStart: 2,
31+
selectionEnd: 5,
32+
});
33+
});
34+
2335
test('missing native field metadata remains unknown instead of becoming false or zero', () => {
2436
const tree = parseUiHierarchyTree(`<hierarchy><node class="android.widget.EditText"
2537
resource-id="field" bounds="[0,0][200,100]" focusable="true" /></hierarchy>`);

‎src/daemon/__tests__/response-views.test.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,27 @@ test('get attrs digest compacts the node under a ref target', () => {
300300
expect(digest).toEqual({ ref: 'e7', node: COMPACT_NODE });
301301
});
302302

303+
test('attrs digest keeps explicit false/zero/empty field facts; unavailable ones stay absent (#2288)', () => {
304+
const fieldFacts = {
305+
value: '',
306+
editable: false,
307+
password: false,
308+
hintShowing: false,
309+
selectionStart: 0,
310+
selectionEnd: 0,
311+
};
312+
const digest = getView!({ ref: 'e7', node: { ...MATCHED_NODE, ...fieldFacts } }, 'digest');
313+
expect(digest.node).toEqual({ ...COMPACT_NODE, ...fieldFacts });
314+
// MATCHED_NODE carries none of the field facts: the digest must not invent them.
315+
const unavailable = getView!({ ref: 'e7', node: MATCHED_NODE }, 'digest').node as Record<
316+
string,
317+
unknown
318+
>;
319+
for (const field of Object.keys(fieldFacts)) {
320+
if (field !== 'value') expect(field in unavailable).toBe(false);
321+
}
322+
});
323+
303324
test('find/get default and full return today’s shape unchanged (same reference)', () => {
304325
const data: DaemonResponseData = { ref: '@e7', text: 'Sign in', node: MATCHED_NODE };
305326
expect(findView!(data, 'default')).toBe(data);

‎src/daemon/response-views.ts‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -90,10 +90,12 @@ function pickScreenshotDigestMetadata(data: DaemonResponseData): DaemonResponseD
9090
return metadata;
9191
}
9292

93-
// The semantic attributes of a single matched node an agent reasons about. The
94-
// verbose framing a digest drops — geometry (`rect`), tree indices
95-
// (`index`/`parentIndex`/`depth`), and process/app plumbing
96-
// (`pid`/`bundleId`/`appName`/`windowTitle`/`surface`/…) — is intentionally absent.
93+
// The semantic attributes of a single matched node an agent reasons about,
94+
// including the field facts whose explicit false/zero/empty is the signal (#2288:
95+
// absent means unavailable). The verbose framing a digest drops — geometry
96+
// (`rect`), tree indices (`index`/`parentIndex`/`depth`), and process/app
97+
// plumbing (`pid`/`bundleId`/`appName`/`windowTitle`/`surface`/…) — is
98+
// intentionally absent.
9799
const SELECTOR_DIGEST_NODE_FIELDS = [
98100
'role',
99101
'type',
@@ -104,6 +106,11 @@ const SELECTOR_DIGEST_NODE_FIELDS = [
104106
'enabled',
105107
'selected',
106108
'focused',
109+
'editable',
110+
'password',
111+
'hintShowing',
112+
'selectionStart',
113+
'selectionEnd',
107114
'hittable',
108115
] as const;
109116

‎website/docs/docs/snapshots.md‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,14 +46,6 @@ agent-device snapshot --diff # Alias for the same diff operation
4646

4747
## Efficient snapshot usage
4848

49-
Android structured snapshot nodes and `get attrs` expose native `editable`, `password`,
50-
`hintShowing`, `selectionStart`, and `selectionEnd` facts when available. Missing fields mean
51-
unknown; `hintShowing` requires Android API 26 or later. Selection values are accessibility
52-
offsets, not character counts, and cannot verify a secure value or its equality to expected text.
53-
An explicitly empty accessibility text remains `value: ""`; missing text remains unavailable.
54-
These facts describe the accessibility observation, which may contain a hint or masked text,
55-
rather than privileged access to an application's backing value.
56-
5749
- iOS and Android share the same mobile snapshot contract: visible-first output, actionable-now refs, and hidden list content communicated via discovery hints.
5850
- Default to `snapshot -i` for agent loops.
5951
- Default snapshot text is an agent-facing, token-efficient view for planning and targeting actions. It is visible-first and may collapse helper/accessibility noise; use `--raw` or `--json` when you need the full provider tree.
@@ -119,3 +111,17 @@ the strategy owns which tiers it may use.
119111
an empty tree.
120112
- Private-accessibility recovery and `--actions` reads are simulator-specific. Physical iOS devices
121113
have no equivalent independent semantic backend; they bound the XCTest work with a probe instead.
114+
115+
## Android field metadata
116+
117+
Android snapshot nodes and `get attrs` (including the digest response) carry the native
118+
`editable`, `password`, `hintShowing`, `selectionStart`, and `selectionEnd` facts whenever the
119+
accessibility tree reports them. Explicit `false` and `0` are kept; an absent field means the fact
120+
was unavailable, not false. `hintShowing` needs Android API 26 or later.
121+
122+
- `value: ""` is an explicitly empty accessibility text; a missing `value` means no text was
123+
reported. The text of an empty field is its hint on modern Android, so check `hintShowing`
124+
before reading `value` as the entered contents.
125+
- `selectionStart`/`selectionEnd` are accessibility selection offsets. They are independent of
126+
`editable` (read-only selectable text exposes them too), they are not a character count, and
127+
they do not prove that a masked or secure value equals expected text.

0 commit comments

Comments
 (0)