Skip to content

Commit 5f179ae

Browse files
committed
fix(selectors): count only measurable fields and read password facts
A field with no usable bounds counted as on-screen, so it could take the focus or turn a single field into an ambiguity before the touch point could reject it. A password fact on an unrecognized type was ignored, so focus went to the container instead. Say in the help and docs that a named field wins over its neighbours and that target_disabled is a reason, and record why the nested-field refusal follows find's refusal contract for printed refs. Refs callstack#3734
1 parent 5daff16 commit 5f179ae

8 files changed

Lines changed: 82 additions & 49 deletions

File tree

‎packages/command-surface/src/interaction/metadata.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ const interactionCommandDescriptions = {
7777
'Scroll in a direction, or toward the top/bottom edge of scrollable content. Set until to a selector to reach an off-screen target in one command rather than a scroll-and-check loop. The optional amount is the finger-path fraction of the viewport axis, honored up to 0.8 of it; directional scrolls reduce release momentum, while app scroll physics determine the final content offset. A visible keyboard shortens the swiped band instead of being dismissed; when too little is left, the command refuses with scroll_keyboard_occludes_surface. A directional scroll also reports the movement it observed as movement: moved, at-edge, unchanged, or unobserved when the two reads could not back a claim either way; an unchanged surface inside a container that still hides content in that direction refuses with scroll_no_progress rather than repeating the requested distance. The movement field is absent where a tier verifies per pass (top/bottom, until), where the runtime cannot read a screen, or where a settle observation or a replay already owns that observation.',
7878
get: 'Read text or accessibility attributes from a snapshot ref or selector without changing the app. Use format text for visible content or attrs for the element attribute map.',
7979
is: 'Check whether a selector satisfies a UI predicate such as visible, hidden, exists, absent, editable, selected, focused, or text. `absent` passes only when one readable, complete, settled, unscoped, full-depth accessibility capture has zero matches. Use wait when the condition may appear asynchronously.',
80-
find: 'Find by text/label/value/role/id and run action; the action defaults to click. The role locator matches snapshot kind vocabulary (button, text, text-field, switch, ...); pre-reconciliation leaf spellings (statictext, edittext, textview) stay accepted on the nodes that carried them during a deprecation window. The focus and type actions aim at a text field: the match when it is one or sits inside one, or else the one enabled on-screen field inside it. Several such fields refuse with AMBIGUOUS_MATCH and the field candidates, and a disabled field refuses with target_disabled. Web text boxes are not recognized as text fields.',
80+
find: 'Find by text/label/value/role/id and run action; the action defaults to click. The role locator matches snapshot kind vocabulary (button, text, text-field, switch, ...); pre-reconciliation leaf spellings (statictext, edittext, textview) stay accepted on the nodes that carried them during a deprecation window. The focus and type actions aim at a text field: the match or the field holding it, even beside other fields, or else the one enabled on-screen field inside the match. Several fields inside the match refuse with AMBIGUOUS_MATCH and the field candidates, and a disabled field refuses with reason target_disabled. Web text boxes are not recognized as text fields.',
8181
gesture:
8282
'Perform a structured pan, fling, swipe, pinch, rotate, transform, tap, or drag gesture. Select the gesture kind, then provide only the inputs that apply to that kind. A tap touches 2 to 5 points at once; repeat it with count and pauseMs.',
8383
} as const;

‎packages/contracts/src/interaction-guarantees.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -589,7 +589,7 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
589589
nonHittable: {
590590
kind: 'waived',
591591
reason:
592-
"gap: the promotion half does run — the `findAct` row declares `hittable-ancestor-below-root`, so the seam dispatches the promoted ancestor's touch point unless `resolveFocusTarget` keeps a text field the locator named (regression-proven in packages/daemon/src/execution/interaction/internal/__tests__/find.test.ts) — but the annotation half never does: the response is the focus runtime's point payload, so a match still reporting `hittable: false` after promotion is accepted silently, where press/click/fill answer with `targetHittable: false` plus a hint. Closing it needs the annotation composed at this seam, which is response-shape work the `responseIdentity` waiver below also records.",
592+
"gap: the promotion half does run — the `findAct` row declares `hittable-ancestor-below-root`, so the seam dispatches the promoted ancestor's touch point unless `resolveFocusTarget` chooses a text field instead (regression-proven in packages/daemon/src/execution/interaction/internal/__tests__/find.test.ts) — but the annotation half never does: the response is the focus runtime's point payload, so a match still reporting `hittable: false` after promotion is accepted silently, where press/click/fill answer with `targetHittable: false` plus a hint. Closing it needs the annotation composed at this seam, which is response-shape work the `responseIdentity` waiver below also records.",
593593
trackingIssue: FIND_FOCUS_DISPATCH_GAPS_ISSUE,
594594
},
595595
// One construction site per dispatch kind, both shared with the plain commands: the focus

‎packages/daemon/src/execution/selector-match-errors.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,11 @@ export function buildAmbiguousMatchError(
2727
);
2828
}
2929

30-
/** The match is one element, but which of its text fields `find focus` or `find type` means is not. */
30+
/**
31+
* The match is one element, but which of its text fields `find focus` or `find type` means is not.
32+
* It follows find's refusal contract: its message routes the caller to narrow the locator, and it
33+
* does not pin the printed refs to a generation (see `publishAmbiguousMatchCandidateRefs`).
34+
*/
3135
export function buildNestedTextFieldsError(
3236
ref: string,
3337
fields: SnapshotState['nodes'],

‎packages/daemon/src/execution/session-snapshot.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -144,12 +144,12 @@ export function issueSettleRefs(
144144
* owe that contract run their error through here — the acting touch runtime
145145
* (`press`/`click`/`fill`) and the strict-read dispatches (`is`, `get attrs`)
146146
* — so the rule has one implementation beside the partial-frame primitive it
147-
* wraps, exactly like {@link issueSettleRefs}. The `find` refusal
148-
* (`buildAmbiguousMatchError`) deliberately does NOT consume this rule: it
149-
* predates it (#1597), prints no `refsGeneration`, and its hint routes the
150-
* caller to narrow the locator; unlike this rule, it does not pin the printed
151-
* candidate refs to a generation. Making those refs issuable is a separate
152-
* contract decision.
147+
* wraps, exactly like {@link issueSettleRefs}. The `find` refusals
148+
* (`buildAmbiguousMatchError`, `buildNestedTextFieldsError`) deliberately do
149+
* NOT consume this rule: the first predates it (#1597), neither prints a
150+
* `refsGeneration`, and both route the caller to narrow the locator; unlike
151+
* this rule, they do not pin the printed candidate refs to a generation.
152+
* Making those refs issuable is a separate contract decision.
153153
*
154154
* The two branches are one decision, not two features:
155155
* - The request's consumed capture IS the session's stored tree (node

‎packages/selectors/src/node-touch-point.test.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,23 @@ test('a label inside an enabled field aims at that field when promotion passes i
110110
assert.deepEqual(resolve(label, form, nodes), { kind: 'target', node: field });
111111
});
112112

113+
test('a published password fact makes an unrecognized type a text field', () => {
114+
const card = child(1, { type: 'Other', rect: { x: 0, y: 0, width: 400, height: 400 } });
115+
const secret = child(2, { parentIndex: 1, type: 'Other', password: true });
116+
assert.deepEqual(resolve(card, card, [card, secret]), { kind: 'target', node: secret });
117+
});
118+
119+
test.each([
120+
['no rect', undefined],
121+
['an empty rect', { x: 20, y: 200, width: 0, height: 0 }],
122+
])('a field with %s neither takes the focus nor counts', (_name, rect) => {
123+
const card = child(1, { type: 'Other', rect: { x: 0, y: 0, width: 400, height: 400 } });
124+
const unmeasured = { ...child(2, { parentIndex: 1, type: 'TextField' }), rect };
125+
const field = child(3, { parentIndex: 1, type: 'TextField' });
126+
assert.deepEqual(resolve(card, card, [card, unmeasured, field]), { kind: 'target', node: field });
127+
assert.deepEqual(resolve(card, card, [card, unmeasured]), { kind: 'target', node: card });
128+
});
129+
113130
test('android reads its own field types', () => {
114131
const row = child(1, {
115132
type: 'android.widget.LinearLayout',

‎packages/selectors/src/node-touch-point.ts‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import type {
55
SnapshotNode,
66
SnapshotState,
77
} from '@agent-device/kernel/snapshot';
8+
import { normalizeRect } from '@agent-device/kernel/rect-center';
89
import { normalizeRef } from '@agent-device/kernel/snapshot';
910
import { createSnapshotVisibility, isFillableType } from '@agent-device/contracts/snapshot';
1011
import type { Platform } from '@agent-device/kernel/device';
@@ -82,18 +83,29 @@ function collectOutermostTextFields(
8283
isAddressable(candidate) &&
8384
isTextField(candidate, platform) &&
8485
isDescendantOf(candidate, node, byIndex) &&
86+
hasUsableRect(candidate) &&
8587
visibility.isVisibleOnScreen(candidate),
8688
);
8789
return fields.filter((field) => !fields.some((other) => isDescendantOf(field, other, byIndex)));
8890
}
8991

92+
/** Visibility reads a node without bounds as visible, but a touch point could not aim at it. */
93+
function hasUsableRect(node: SnapshotNode): boolean {
94+
const rect = normalizeRect(node.rect);
95+
return rect !== null && rect.width > 0 && rect.height > 0;
96+
}
97+
9098
function isAddressable(node: SnapshotNode): node is SnapshotAddressableNode {
9199
return node.ref !== undefined;
92100
}
93101

94102
/** An untyped node is lenient for a fill guard, but here being a field would make it the target. */
95103
function isTextField(node: SnapshotNode, platform: Platform): boolean {
96-
return node.editable ?? (Boolean(node.type?.trim()) && isFillableType(node.type ?? '', platform));
104+
return (
105+
node.editable ??
106+
(node.password === true ||
107+
(Boolean(node.type?.trim()) && isFillableType(node.type ?? '', platform)))
108+
);
97109
}
98110

99111
export function resolveNodeTouchPoint(

‎test/integration/interaction-contract/find-focus-dispatch.contract.test.ts‎

Lines changed: 37 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -419,43 +419,6 @@ test(
419419
PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS,
420420
);
421421

422-
test(
423-
nestedFieldScenario(14),
424-
async () => {
425-
// Promotion climbs past the disabled field to the form, whose only enabled field is another one.
426-
const [application, form, field, sibling] = namedFieldBesideSiblingTree({ enabled: false }) as [
427-
object,
428-
object,
429-
object,
430-
object,
431-
];
432-
const tree = [
433-
application,
434-
form,
435-
{ ...field, label: undefined },
436-
{
437-
index: 3,
438-
parentIndex: 2,
439-
type: 'StaticText',
440-
label: 'Notes',
441-
rect: { x: 40, y: 140, width: 100, height: 20 },
442-
hittable: false,
443-
},
444-
{ ...sibling, index: 4 },
445-
];
446-
await withIosContractDaemon([runnerSnapshotEntry(tree)], async (daemon) => {
447-
const error = assertRpcError(
448-
await daemon.callCommand('find', ['label', 'Notes', 'type', 'hello']),
449-
500,
450-
'COMMAND_FAILED',
451-
/Matched element @e3 is a disabled text field and cannot be focused/,
452-
);
453-
assert.equal(readErrorDetails(error).reason, 'target_disabled');
454-
});
455-
},
456-
PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS,
457-
);
458-
459422
test(
460423
nestedFieldScenario(10),
461424
async () => {
@@ -569,6 +532,43 @@ test(
569532
PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS,
570533
);
571534

535+
test(
536+
nestedFieldScenario(14),
537+
async () => {
538+
// Promotion climbs past the disabled field to the form, whose only enabled field is another one.
539+
const [application, form, field, sibling] = namedFieldBesideSiblingTree({ enabled: false }) as [
540+
object,
541+
object,
542+
object,
543+
object,
544+
];
545+
const tree = [
546+
application,
547+
form,
548+
{ ...field, label: undefined },
549+
{
550+
index: 3,
551+
parentIndex: 2,
552+
type: 'StaticText',
553+
label: 'Notes',
554+
rect: { x: 40, y: 140, width: 100, height: 20 },
555+
hittable: false,
556+
},
557+
{ ...sibling, index: 4 },
558+
];
559+
await withIosContractDaemon([runnerSnapshotEntry(tree)], async (daemon) => {
560+
const error = assertRpcError(
561+
await daemon.callCommand('find', ['label', 'Notes', 'type', 'hello']),
562+
500,
563+
'COMMAND_FAILED',
564+
/Matched element @e3 is a disabled text field and cannot be focused/,
565+
);
566+
assert.equal(readErrorDetails(error).reason, 'target_disabled');
567+
});
568+
},
569+
PARALLEL_PROVIDER_SCENARIO_TIMEOUT_MS,
570+
);
571+
572572
test(
573573
scenario('responseConstruction'),
574574
async () => {

‎website/docs/docs/selectors.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,8 @@ Tips:
5454
[Structured node fields](/docs/snapshots#structured-node-fields---json).
5555
- Use `find ... wait <timeoutMs>` to wait for an element to appear.
5656
- Pair `find` with a scoped snapshot (`snapshot -s "<label>"`) for speed.
57-
- [Android] If the matched node is not hittable, agent-device clicks or focuses the nearest hittable ancestor. `find ... focus` and `find ... type` keep a matched text field as their target instead.
58-
- `find ... focus` and `find ... type` aim at a text field: the match when it is one or sits inside one, or else the one enabled on-screen field inside it. Several such fields refuse with `AMBIGUOUS_MATCH` and the field candidates, and a disabled field refuses with `target_disabled`. Web text boxes are not recognized as text fields, so a web match keeps its own aim.
57+
- [Android] If the matched node is not hittable, agent-device clicks or focuses the nearest hittable ancestor.
58+
- `find ... focus` and `find ... type` aim at a text field: the match or the field holding it, even when it is not hittable or sits beside other fields, or else the one enabled on-screen field inside the match. Several fields inside the match refuse with `AMBIGUOUS_MATCH` and the field candidates, and a disabled field refuses with `COMMAND_FAILED` and reason `target_disabled`. Web text boxes are not recognized as text fields, so a web match keeps its own aim.
5959
- Use `fill` to find a field and replace its text in one step. To append to the text in the focused field, use `click` or `press`, then `type`.
6060

6161
## Selector syntax

0 commit comments

Comments
 (0)