Skip to content

Commit 8539d87

Browse files
authored
fix(daemon): run find focus through the shared touch-point decisions (#4223)
* fix(daemon): aim find focus at the parent-owned touch point The find focus seam dispatches its own point instead of re-entering the interaction leaf, so it skipped the touch-point decision the acting paths run. A match fully tiled by interactive children now refuses before dispatch, and a child owning the center moves the aim to free parent space. Missing or zero-area rects get the typed invalid-bounds refusal. Move resolveNodeTouchPoint to @agent-device/selectors/node-touch-point so the daemon and command-runtime share one owner, and close the parentOwnedTouchPoint cell of the find-focus-dispatch guarantee row. Refs #3734 * chore(gates): declare the selectors node-touch-point subpath The package-boundaries enumeration lists every selectors subpath, so the new export has to be declared for the R11 layering rule to accept it. * fix(contracts): describe the find focus touch point in its waivers The offscreen and nonHittable waiver reasons still said the seam dispatches a rect center. * fix(daemon): aim find focus and type at a nested text field The touch-point rule treats a nested text field as a competitor to avoid, so a match wrapping a field reported focus success beside the field and the typed text then went wherever the focus already was. A sole enabled, visible field now becomes the target, with the same touch-point, covered and keyboard checks. Several fields refuse with AMBIGUOUS_MATCH and the field candidates. Refs #3734 * style(test): format the find focus contract scenarios * fix(daemon): keep find focus on the field the locator named A disabled field promotes to its parent on iOS, and the parent's other field then counted as the sole one, so find type wrote into a field the locator never named. The located field now decides first: an enabled one stays the target and a disabled one refuses with target_disabled, as does a match holding only disabled fields. Count only outermost on-screen typed fields, so a hosted text field, a scrolled-away field and an untyped child no longer change the target. The rule moves to the find description so MCP clients see it, and it names web text boxes as unrecognized. Refs #3734 * fix(daemon): refuse a disabled field reached through promotion The disabled check only read the located node, so a same-rect wrapper over a disabled field was still focused, and a label inside one was promoted past it to an enabled sibling. find type then wrote into another field and reported success. The field decision now walks from the located node to the promoted one, and the nearest field on that path is the named field. The find help, selector docs and nonHittable waiver now state that a named text field stays the target. Refs #3734 * fix(selectors): count only measurable fields and read password facts A field with no usable bounds counted as on-screen, so it could take the focus or turn a single field into an ambiguity before the touch point could reject it. A password fact on an unrecognized type was ignored, so focus went to the container instead. Say in the help and docs that a named field wins over its neighbours and that target_disabled is a reason, and record why the nested-field refusal follows find's refusal contract for printed refs. Refs #3734 * docs(selectors): split the find focus rule and name the covered refusal Refs #3734 * docs(selectors): qualify which fields refuse find focus Refs #3734 * docs(selectors): name the covered refusal's reason Refs #3734
1 parent 4abff15 commit 8539d87

18 files changed

Lines changed: 988 additions & 95 deletions

File tree

‎packages/command-runtime/src/commands/interaction/runtime/ref-target-resolution.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ import type {
2525
SurfaceScopedNodes,
2626
} from '@agent-device/contracts/interaction';
2727
import { INTERACTION_ERROR_REASONS } from '@agent-device/selectors/interaction-error';
28+
import { resolveNodeTouchPoint } from '@agent-device/selectors/node-touch-point';
2829
import { localIdentitiesEqual, readNodeLocalIdentity } from '@agent-device/ad-script';
2930
import type { ResolveInteractionTargetParams } from './interaction-resolution-request.ts';
3031
import { assertReplayTargetResolution } from './replay-target-guard.ts';
@@ -37,7 +38,6 @@ import {
3738
assertVisibleRefTarget,
3839
runInteractionPipelineStages,
3940
} from './target-visibility-stages.ts';
40-
import { resolveNodeTouchPoint } from './resolution-touch-point.ts';
4141

4242
/** The node a ref target acts on, plus the tree the shared guards read it against. */
4343
type RefResolution = {

‎packages/command-runtime/src/commands/interaction/runtime/resolution-touch-point.ts‎

Lines changed: 0 additions & 54 deletions
This file was deleted.

‎packages/command-runtime/src/commands/interaction/runtime/resolution.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import type {
1515
SurfaceScopedNodes,
1616
} from '@agent-device/contracts/interaction';
1717
import { describeKeyboardOccludedPointWarning } from '@agent-device/selectors/keyboard-occlusion';
18+
import { resolveNodeTouchPoint } from '@agent-device/selectors/node-touch-point';
1819
import type { InteractionAction } from '@agent-device/selectors/interaction-action';
1920
import { assertReplayTargetResolution } from './replay-target-guard.ts';
2021
import type { ResolveInteractionTargetParams } from './interaction-resolution-request.ts';
@@ -27,7 +28,6 @@ import {
2728
assertVisibleSelectorTarget,
2829
runInteractionPipelineStages,
2930
} from './target-visibility-stages.ts';
30-
import { resolveNodeTouchPoint } from './resolution-touch-point.ts';
3131

3232
export type { InteractionTarget, ResolvedInteractionTarget };
3333

‎packages/command-surface/src/interaction/metadata.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ const interactionCommandDescriptions = {
7777
'Scroll in a direction, or toward the top/bottom edge of scrollable content. Set until to a selector to reach an off-screen target in one command rather than a scroll-and-check loop. The optional amount is the finger-path fraction of the viewport axis, honored up to 0.8 of it; directional scrolls reduce release momentum, while app scroll physics determine the final content offset. A visible keyboard shortens the swiped band instead of being dismissed; when too little is left, the command refuses with scroll_keyboard_occludes_surface. A directional scroll also reports the movement it observed as movement: moved, at-edge, unchanged, or unobserved when the two reads could not back a claim either way; an unchanged surface inside a container that still hides content in that direction refuses with scroll_no_progress rather than repeating the requested distance. The movement field is absent where a tier verifies per pass (top/bottom, until), where the runtime cannot read a screen, or where a settle observation or a replay already owns that observation.',
7878
get: 'Read text or accessibility attributes from a snapshot ref or selector without changing the app. Use format text for visible content or attrs for the element attribute map.',
7979
is: 'Check whether a selector satisfies a UI predicate such as visible, hidden, exists, absent, editable, selected, focused, or text. `absent` passes only when one readable, complete, settled, unscoped, full-depth accessibility capture has zero matches. Use wait when the condition may appear asynchronously.',
80-
find: 'Find by text/label/value/role/id and run action; the action defaults to click. The role locator matches snapshot kind vocabulary (button, text, text-field, switch, ...); pre-reconciliation leaf spellings (statictext, edittext, textview) stay accepted on the nodes that carried them during a deprecation window.',
80+
find: 'Find by text/label/value/role/id and run action; the action defaults to click. The role locator matches snapshot kind vocabulary (button, text, text-field, switch, ...); pre-reconciliation leaf spellings (statictext, edittext, textview) stay accepted on the nodes that carried them during a deprecation window. The focus and type actions aim at a text field: the match or the field holding it, even beside other fields, or else the one enabled on-screen field inside the match. Several enabled on-screen fields inside the match refuse with AMBIGUOUS_MATCH and the field candidates, and a disabled target refuses with reason target_disabled. Web text boxes are not recognized as text fields.',
8181
gesture:
8282
'Perform a structured pan, fling, swipe, pinch, rotate, transform, tap, or drag gesture. Select the gesture kind, then provide only the inputs that apply to that kind. A tap touches 2 to 5 points at once; repeat it with count and pauseMs.',
8383
} as const;

‎packages/contracts/src/interaction-guarantees.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -566,30 +566,30 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
566566
},
567567
// #2622: the seam dispatches the point itself, so the shared #2589 guard runs here rather
568568
// than at the interaction leaf's door. Same refusal, same typed details, measured at the
569-
// center the seam dispatches.
569+
// point the seam dispatches.
570570
keyboardOcclusion: {
571571
kind: 'runtime',
572572
via: 'packages/selectors/src/keyboard-occlusion.ts#assertTapTargetClearOfVisibleKeyboard',
573573
},
574574
offscreen: {
575575
kind: 'waived',
576576
reason:
577-
'gap: the `findAct` row declares `offscreen: "ignore"` because find ranks across the whole tree and prefers on-screen matches rather than refusing off-screen ones (#1625), and focus/type never re-enter the acting leaf where the runtime rows refuse. So the seam dispatches at a captured rect center the acting paths would reject as outside the root viewport — a closed drawer or an off-viewport carousel is a silent no-op focus here.',
577+
'gap: the `findAct` row declares `offscreen: "ignore"` because find ranks across the whole tree and prefers on-screen matches rather than refusing off-screen ones (#1625), and focus/type never re-enter the acting leaf where the runtime rows refuse. So the seam dispatches at a captured touch point the acting paths would reject as outside the root viewport — a closed drawer or an off-viewport carousel is a silent no-op focus here.',
578578
trackingIssue: FIND_FOCUS_DISPATCH_GAPS_ISSUE,
579579
},
580+
// The rule treats a nested text field as a competitor to avoid, but this seam means to focus
581+
// one, so `resolveFocusTarget` chooses the node the rule then aims inside (#3734).
580582
parentOwnedTouchPoint: {
581-
kind: 'waived',
582-
reason:
583-
'gap: the seam aims `centerOfRect` of the resolved node rather than the parent-owned point `resolveInteractionTouchPoint` chooses, so a match whose center is tiled by an independently interactive descendant activates that descendant instead of failing closed — and the keyboard guard above measures the same center the tap uses, which is the point actually dispatched but not the point the acting paths would aim.',
584-
trackingIssue: FIND_FOCUS_DISPATCH_GAPS_ISSUE,
583+
kind: 'runtime',
584+
via: 'packages/selectors/src/interaction-touch-point.ts#resolveInteractionTouchPoint',
585585
},
586586
// The guarantee is promotion AND annotation. This seam runs only the first half, so the
587587
// cell cannot claim `runtime` for the whole guarantee (the same shape `native-ref` classifies
588588
// the other way round, annotation without promotion).
589589
nonHittable: {
590590
kind: 'waived',
591591
reason:
592-
"gap: the promotion half does run — the `findAct` row declares `hittable-ancestor-below-root`, so the seam dispatches the promoted ancestor's center (regression-proven in packages/daemon/src/execution/interaction/internal/__tests__/find.test.ts) — but the annotation half never does: the response is the focus runtime's point payload, so a match still reporting `hittable: false` after promotion is accepted silently, where press/click/fill answer with `targetHittable: false` plus a hint. Closing it needs the annotation composed at this seam, which is response-shape work the `responseIdentity` waiver below also records.",
592+
"gap: the promotion half does run — the `findAct` row declares `hittable-ancestor-below-root`, so the seam dispatches the promoted ancestor's touch point unless `resolveFocusTarget` chooses a text field instead (regression-proven in packages/daemon/src/execution/interaction/internal/__tests__/find.test.ts) — but the annotation half never does: the response is the focus runtime's point payload, so a match still reporting `hittable: false` after promotion is accepted silently, where press/click/fill answer with `targetHittable: false` plus a hint. Closing it needs the annotation composed at this seam, which is response-shape work the `responseIdentity` waiver below also records.",
593593
trackingIssue: FIND_FOCUS_DISPATCH_GAPS_ISSUE,
594594
},
595595
// One construction site per dispatch kind, both shared with the plain commands: the focus
@@ -636,7 +636,7 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
636636
targetStability: {
637637
kind: 'waived',
638638
reason:
639-
"gap: the seam dispatches a rect center captured one runner round trip earlier without the #3471 re-observation, so a focus issued right after an action that animated the surface can aim a position the target has already vacated and report success on the vacancy. The gate itself is the acting leaf's (`guardResolvedTargetAgainstMotion`), and this route does not pass through it.",
639+
"gap: the seam dispatches a touch point captured one runner round trip earlier without the #3471 re-observation, so a focus issued right after an action that animated the surface can aim a position the target has already vacated and report success on the vacancy. The gate itself is the acting leaf's (`guardResolvedTargetAgainstMotion`), and this route does not pass through it.",
640640
trackingIssue: FIND_FOCUS_DISPATCH_GAPS_ISSUE,
641641
},
642642
},

‎packages/daemon/src/execution/interaction/internal/find.ts‎

Lines changed: 51 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,16 @@ import {
1212
} from '@agent-device/selectors';
1313
import { runNodePipelineStages } from '@agent-device/selectors/selector-pipeline';
1414
import { SELECTOR_PIPELINE_POLICIES } from '@agent-device/selectors/selector-pipeline-policy';
15-
import { coveredInteractionDetails } from '@agent-device/selectors/interaction-error';
15+
import {
16+
coveredInteractionDetails,
17+
INTERACTION_ERROR_REASONS,
18+
} from '@agent-device/selectors/interaction-error';
19+
import {
20+
resolveFocusTarget,
21+
resolveNodeTouchPoint,
22+
} from '@agent-device/selectors/node-touch-point';
23+
import { isSnapshotNodeInteractionBlocked } from '@agent-device/capture-kit/snapshot-occlusion';
24+
import { buildNestedTextFieldsError } from '../../selector-match-errors.ts';
1625
import { assertTapTargetClearOfVisibleKeyboard } from '@agent-device/selectors/keyboard-occlusion';
1726
import {
1827
interactionVerb,
@@ -21,6 +30,7 @@ import {
2130
import { normalizeError } from '@agent-device/kernel/errors';
2231
import {
2332
centerOfRect,
33+
type Point,
2434
type SnapshotAddressableNode,
2535
type SnapshotKeyboardBandFact,
2636
type SnapshotState,
@@ -397,18 +407,43 @@ async function dispatchFocusForFindMatch(
397407
const { req, logPath } = ctx;
398408
const coveredResponse = rejectCoveredFindMatch(match, 'focus');
399409
if (coveredResponse) return coveredResponse;
400-
const coords = match.resolvedNode.rect ? centerOfRect(match.resolvedNode.rect) : null;
401-
if (!coords) {
402-
return errorResponse('COMMAND_FAILED', 'matched element has no bounds');
410+
const focusTarget = resolveFocusTarget({
411+
located: match.node,
412+
resolved: match.resolvedNode,
413+
nodes: match.tree.nodes,
414+
platform: ctx.device.platform,
415+
});
416+
if (focusTarget.kind === 'several-fields') {
417+
return buildNestedTextFieldsError(match.ref, focusTarget.fields);
418+
}
419+
if (focusTarget.kind === 'disabled-field') {
420+
return errorResponse(
421+
'COMMAND_FAILED',
422+
`Matched element @${focusTarget.node.ref} is a disabled text field and cannot be focused`,
423+
{
424+
reason: INTERACTION_ERROR_REASONS.targetDisabled,
425+
ref: `@${focusTarget.node.ref}`,
426+
hint: 'Enable the field first, or target a different one.',
427+
},
428+
);
403429
}
404-
// The same keyboard decision the acting paths run (#2589/#2622): this seam dispatches the point
405-
// itself instead of re-entering the interaction leaf, so the guard has to run here.
430+
const target = focusTarget.node;
431+
if (isSnapshotNodeInteractionBlocked(target)) return coveredFindResponse(target, 'focus');
432+
const targetRef = `@${target.ref}`;
433+
let coords: Point;
434+
// The same touch-point and keyboard decisions the acting paths run (#2589/#2622/#3734): this seam
435+
// dispatches the point itself instead of re-entering the interaction leaf, so they have to run here.
406436
try {
437+
coords = resolveNodeTouchPoint(target, match.tree.nodes, {
438+
invalidMessage: `Ref ${targetRef} has no usable bounds`,
439+
blockedTargetLabel: `Ref ${targetRef}`,
440+
blockedTargetDetails: { ref: targetRef },
441+
});
407442
assertTapTargetClearOfVisibleKeyboard({
408443
nodes: match.tree.nodes,
409-
node: match.resolvedNode,
444+
node: target,
410445
action: 'focus',
411-
label: `Ref ${match.ref}`,
446+
label: `Ref ${targetRef}`,
412447
...(match.keyboard ? { keyboard: match.keyboard } : {}),
413448
tapPoint: coords,
414449
});
@@ -438,11 +473,16 @@ function rejectCoveredFindMatch(
438473
match: ResolvedMatch,
439474
action: InteractionAction,
440475
): DaemonResponse | null {
441-
const blockedNode = match.occludedNode;
442-
if (!blockedNode) return null;
476+
return match.occludedNode ? coveredFindResponse(match.occludedNode, action) : null;
477+
}
478+
479+
function coveredFindResponse(
480+
blockedNode: SnapshotAddressableNode,
481+
action: InteractionAction,
482+
): DaemonResponse {
443483
return errorResponse(
444484
'COMMAND_FAILED',
445-
`Matched element ${match.ref} is covered by another visible element and cannot ${interactionVerb(action)} safely`,
485+
`Matched element @${blockedNode.ref} is covered by another visible element and cannot ${interactionVerb(action)} safely`,
446486
coveredInteractionDetails(blockedNode),
447487
);
448488
}

‎packages/daemon/src/execution/selector-match-errors.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import { errorResponse } from '@agent-device/kernel/contracts';
1212
// (`elementMatchCandidateDetails`), beside the renderer — `matches` (the true
1313
// total) is what a
1414
// "+N more" marker is computed from at render time by the surface owners.
15-
// Exported as the single AMBIGUOUS_MATCH producer so the help-benchmark
15+
// Exported as find's locator AMBIGUOUS_MATCH producer so the help-benchmark
1616
// sample parity test renders the exact error this handler returns; a message
1717
// change here fails that gate instead of drifting past it.
1818
export function buildAmbiguousMatchError(
@@ -26,3 +26,19 @@ export function buildAmbiguousMatchError(
2626
{ locator, query, ...elementMatchCandidateDetails(matches) },
2727
);
2828
}
29+
30+
/**
31+
* The match is one element, but which of its text fields `find focus` or `find type` means is not.
32+
* It follows find's refusal contract: its message routes the caller to narrow the locator, and it
33+
* does not pin the printed refs to a generation (see `publishAmbiguousMatchCandidateRefs`).
34+
*/
35+
export function buildNestedTextFieldsError(
36+
ref: string,
37+
fields: SnapshotState['nodes'],
38+
): DaemonResponse {
39+
return errorResponse(
40+
'AMBIGUOUS_MATCH',
41+
`find matched ${ref}, which contains ${fields.length} text fields. Use a more specific locator or selector.`,
42+
{ ref, ...elementMatchCandidateDetails(fields) },
43+
);
44+
}

‎packages/daemon/src/execution/session-snapshot.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -144,12 +144,12 @@ export function issueSettleRefs(
144144
* owe that contract run their error through here — the acting touch runtime
145145
* (`press`/`click`/`fill`) and the strict-read dispatches (`is`, `get attrs`)
146146
* — so the rule has one implementation beside the partial-frame primitive it
147-
* wraps, exactly like {@link issueSettleRefs}. The `find` refusal
148-
* (`buildAmbiguousMatchError`) deliberately does NOT consume this rule: it
149-
* predates it (#1597), prints no `refsGeneration`, and its hint routes the
150-
* caller to narrow the locator; unlike this rule, it does not pin the printed
151-
* candidate refs to a generation. Making those refs issuable is a separate
152-
* contract decision.
147+
* wraps, exactly like {@link issueSettleRefs}. The `find` refusals
148+
* (`buildAmbiguousMatchError`, `buildNestedTextFieldsError`) deliberately do
149+
* NOT consume this rule: the first predates it (#1597), neither prints a
150+
* `refsGeneration`, and both route the caller to narrow the locator; unlike
151+
* this rule, they do not pin the printed candidate refs to a generation.
152+
* Making those refs issuable is a separate contract decision.
153153
*
154154
* The two branches are one decision, not two features:
155155
* - The request's consumed capture IS the session's stored tree (node

‎packages/selectors/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,10 @@
6060
"types": "./src/interaction-touch-point.ts",
6161
"default": "./src/interaction-touch-point.ts"
6262
},
63+
"./node-touch-point": {
64+
"types": "./src/node-touch-point.ts",
65+
"default": "./src/node-touch-point.ts"
66+
},
6367
"./keyboard-occlusion": {
6468
"types": "./src/keyboard-occlusion.ts",
6569
"default": "./src/keyboard-occlusion.ts"

‎packages/selectors/src/interaction-error.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ export const INTERACTION_ERROR_REASONS = {
2727
targetBoundsInvalid: 'target_bounds_invalid',
2828
/** The target resolved, but another visible element covers it. */
2929
targetCovered: 'target_covered',
30+
/** `find focus` or `find type` resolved to a disabled text field, which `details.ref` names. */
31+
targetDisabled: 'target_disabled',
3032
/**
3133
* A press or click refused because it could not confirm a resting position for its target: the
3234
* target's rect was still changing when the transition window after the session's last mutating

0 commit comments

Comments
 (0)