Skip to content

Commit 694e52b

Browse files
authored
feat(selectors): match role= against snapshot kind with a node-scoped alias window (#3232)
* feat(selectors): match role= against the snapshot kind vocabulary with a node-scoped alias window #3021: the role= selector term and the find role= locator each normalized node.type into a raw leaf class name (statictext, edittext), diverging from the coarse kind vocabulary formatRole publishes on every snapshot node. Both surfaces now read one kernel-owned reader, roleSpellingsOfNode: the canonical kind (the published kind, formatRole-derived only for nodes that bypass attachRefs) plus a deprecation window carrying exactly the node's retired spelling, so role=text and role=text-field start matching while released role=statictext/role=edittext/role=textview scripts keep resolving — on the nodes that actually carried the leaf, never a kind sibling's (role=linearlayout still refuses a FrameLayout, pinned by the #1280 daemon fixtures). No new runtime warning or response shape: the window is disclosed in the versioned find help, user docs, and pinned by the alias tests. contracts normalizeType (fillable/occlusion) and provider-webdriver roleFromWebDriverType (the iOS role field) serve different fields and stay untouched. * fix(selectors): keep find role= substring scoring on the leaf segment Adversarial review of the #3021 window: substring scoring against the whole spellings let an unrecognized dotted class (whose kind keeps its package path) match queries the retired locator refused — find role=fenix scored org.mozilla.fenix.ReaderView. Substring scoring now reads only the last '.'-separated segment, the retired locator's leaf; exact-kind scoring is unchanged. Pins the AX-stripped locator promotion as the intended term/ locator parity it is (role=link was always exact on an AXLink node). * fix(selectors): exact-score role locators on term-normalized spellings The selector term compared normalizeText on both sides, so role="some widget" always matched the class 'Some Widget' while the find locator scored that query 0. #3021 promises one shared authority for both role= surfaces, so the locator's exact compare now normalizes spellings the same way; substring scoring stays on the raw retired leaf. Round-2 adversarial review nit folded into behavior instead of a comment softening. * docs(selectors): disclose the alias window's kind-adjacent matches precisely PR review: is guaranteed BY a kind, not exactly equivalent to one — a windowed legacy node can match its old spelling beside a kind the vocabulary never renamed (AXWebArea), so the docs now state the guarantee and the alias separately, and the client-api anchor now carries the real github-slugger slug. Adds a kernel pin that the published kind leads the spellings even when it diverges from what formatRole(type) would derive — the mutation the prior test could not catch. Clarifies that the locator's substring leaf deliberately keeps the retired LOCATOR's '.'-only split (each surface's historical substring meaning survives); the retired TERM's '/'-reduced spelling rides as the exact alias, which is where term/locator agreement comes from. * docs(kernel): name the chain-builder precondition for closing the role window Final adversarial nit: the sunset clause on the retired role spelling named an exit criterion that is not self-sufficient — buildSelectorChainForNode still emits the leaf spelling into recorded chains, so the window must outlive the chain builder's emission, not the matcher's need. * refactor(selectors): window role= through contracts' normalizeType owner, leave kernel untouched PR review: the kernel's legacyRoleSpellingOfType was a verbatim copy of contracts' normalizeType, whose live callers include the chain recorder — a drift there would silently split the window from what recorded scripts carry. One owner for the legacy spelling: the window now lives in @agent-device/selectors internals (node.ts roleSpellingsOfNode) as [kind, normalizeType(type)] when they differ, read by both the selector term and the find locator. contracts/snapshot is already a selectors dependency and is the function buildSelectorChainForNode records chains through, so window and recorded spelling move together by construction. The kernel keeps only what it already published (kind, formatRole): the formatRole leaf refactor, the new exports, and the kernel test all leave with the copy; the vocabulary suite moves to node.test.ts beside the helper, including a pin that the window entry equals normalizeType for every probed backend spelling. * docs(selectors): point the locator leaf comment at normalizeType, not kernel The retired-term spelling moved out of kernel with the window; the comment named the stale owner.
1 parent 3df0979 commit 694e52b

11 files changed

Lines changed: 558 additions & 26 deletions

File tree

‎packages/selectors/src/internal/find.test.ts‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@ import { test } from 'vitest';
22
import assert from 'node:assert/strict';
33
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
44
import { findBestMatchesByLocator } from './find.ts';
5+
import { parseSelectorChain } from './parse.ts';
6+
import { resolveSelectorChain } from './resolve.ts';
57
import { formatRole } from '@agent-device/kernel/snapshot';
68

79
function makeNode(ref: string, label?: string, identifier?: string): SnapshotNode {
@@ -28,3 +30,139 @@ test('findBestMatchesByLocator returns all best-scored matches', () => {
2830
assert.equal(result.matches[0]?.ref, 'e1');
2931
assert.equal(result.matches[1]?.ref, 'e2');
3032
});
33+
34+
// ── #3021: the `find role=` locator shares the matcher's kind vocabulary ──
35+
36+
function makeTypedNode(ref: string, type: string): SnapshotNode {
37+
return {
38+
index: Number(ref.replace('e', '')) || 0,
39+
ref,
40+
type,
41+
kind: formatRole(type),
42+
rect: { x: 0, y: 0, width: 100, height: 20 },
43+
};
44+
}
45+
46+
function roleMatches(query: string, types: string[]) {
47+
const nodes = types.map((type, idx) => makeTypedNode(`e${idx + 1}`, type));
48+
return findBestMatchesByLocator(nodes, 'role', query);
49+
}
50+
51+
test('find role= scores the coarse kind vocabulary exactly', () => {
52+
// Forward direction: the kind a snapshot publishes for the node.
53+
const text = roleMatches('text', ['android.widget.TextView', 'XCUIElementTypeStaticText']);
54+
assert.equal(text.score, 2);
55+
assert.deepEqual(
56+
text.matches.map((node) => node.ref),
57+
['e1', 'e2'],
58+
);
59+
});
60+
61+
test('find role= still scores legacy leaf spellings exactly during the window', () => {
62+
// Compatibility direction: `textview` was the old exact leaf for both class
63+
// spellings; the kind is context-coarse (`text` vs `text-view`), and the
64+
// window covers both.
65+
const textView = roleMatches('textview', ['android.widget.TextView', 'XCUIElementTypeTextView']);
66+
assert.equal(textView.score, 2);
67+
assert.deepEqual(
68+
textView.matches.map((node) => node.ref),
69+
['e1', 'e2'],
70+
);
71+
const editText = roleMatches('edittext', ['android.widget.EditText']);
72+
assert.equal(editText.score, 2);
73+
});
74+
75+
test('find role= keeps the historical substring score for legacy prefixes', () => {
76+
// Before #3021 the locator substring-matched the leaf (`statictext`.includes
77+
// ('static')); the alias window must keep that ranking, at score 1, not
78+
// promote it to an exact hit.
79+
const partial = roleMatches('static', ['XCUIElementTypeStaticText']);
80+
assert.equal(partial.score, 1);
81+
assert.equal(partial.matches[0]?.ref, 'e1');
82+
});
83+
84+
test('find role= substring-scores the leaf segment, never a dotted kind package path', () => {
85+
// An unrecognized custom class keeps its full dotted lowercase as its kind.
86+
// The retired locator substring-matched the LAST segment only, so a released
87+
// `find role=fenix` refused `org.mozilla.fenix.ReaderView`; substringing the
88+
// whole kind would silently pull such nodes into the best set. Exact-kind
89+
// scoring still answers the full spelling when the query names it.
90+
const type = 'org.mozilla.fenix.ReaderView';
91+
assert.equal(roleMatches('fenix', [type]).score, 0);
92+
assert.equal(roleMatches('readerview', [type]).score, 2); // windowed legacy leaf
93+
assert.equal(roleMatches(type.toLowerCase(), [type]).score, 2); // the exact kind
94+
});
95+
96+
test('find role= reaches AX-stripped queries through the exact window alias', () => {
97+
// Pinned parity decision (#3021 finding): the retired selector TERM stripped
98+
// a leading `AX` (`contracts` normalizeType) while the retired LOCATOR did
99+
// not, so `find role link click` on a macOS-helper `AXLink` node scored 1
100+
// there while `role=link` matched exactly. Adopting one shared authority
101+
// promotes the locator's AX-stripped query to the exact score — the locator
102+
// and the term cannot disagree about a node's role anymore.
103+
const axNode = roleMatches('link', ['AXLink']);
104+
assert.equal(axNode.score, 2);
105+
// The prefix-kept spelling keeps its historical exact score.
106+
assert.equal(roleMatches('axlink', ['AXLink']).score, 2);
107+
});
108+
109+
test('find role= exact-scores whitespace-normalized spellings like the term does', () => {
110+
// Same parity rule for the term's other normalization step: the selector
111+
// term compared `normalizeText` on both sides, so `role="some widget"`
112+
// always matched the class `Some Widget`; the retired locator scored that
113+
// query 0. One shared authority means the locator agrees — exact score, not
114+
// a substring rescue.
115+
assert.equal(roleMatches('some widget', ['Some Widget']).score, 2);
116+
});
117+
118+
test('find role= refuses sibling vocabulary words with no kind or alias match', () => {
119+
// Nearest-negative: a kind one dash away and an unrelated kind score 0.
120+
assert.equal(roleMatches('text-field', ['XCUIElementTypeStaticText']).score, 0);
121+
assert.equal(roleMatches('switch', ['XCUIElementTypeStaticText']).score, 0);
122+
});
123+
124+
test('find role= does not widen a legacy leaf onto kind siblings that never carried it', () => {
125+
// The node-scoped window at the locator: `linearlayout` and `framelayout`
126+
// share the coarse kind `group`, but the legacy leaf was the EXACT class
127+
// leaf, and widening it onto the kind's other members would silently change
128+
// what a released `find role linearlayout` click targets.
129+
const both = roleMatches('linearlayout', [
130+
'android.widget.LinearLayout',
131+
'android.widget.FrameLayout',
132+
]);
133+
assert.equal(both.score, 2);
134+
assert.deepEqual(
135+
both.matches.map((node) => node.ref),
136+
['e1'],
137+
);
138+
});
139+
140+
test('find role= and the role= selector term agree on every kind, alias, and negative', () => {
141+
// Locator/matcher parity is the issue's both-paths-reachable proof: the same
142+
// (query, type) pairs resolve through `resolveSelectorChain` exactly where
143+
// the locator scored them exactly 2.
144+
const cases: readonly [string, string][] = [
145+
['text', 'XCUIElementTypeStaticText'],
146+
['statictext', 'XCUIElementTypeStaticText'],
147+
['textview', 'android.widget.TextView'],
148+
['text-field', 'android.widget.EditText'],
149+
['edittext', 'android.widget.EditText'],
150+
['button', 'XCUIElementTypeButton'],
151+
['imagebutton', 'XCUIElementTypeOther'], // kind of Other is `other`; must NOT match
152+
['text-field', 'XCUIElementTypeStaticText'], // sibling kind: must NOT match
153+
];
154+
for (const [query, type] of cases) {
155+
const nodes = [makeTypedNode('e1', type)];
156+
const locatorScore = findBestMatchesByLocator(nodes, 'role', query).score;
157+
const chain = parseSelectorChain(`role=${query}`);
158+
const resolved = resolveSelectorChain(nodes, chain, {
159+
platform: 'android',
160+
requireUnique: true,
161+
});
162+
assert.equal(
163+
Boolean(resolved),
164+
locatorScore === 2,
165+
`query "${query}" against type "${type}" diverged: locator score ${locatorScore}, selector ${resolved ? 'matched' : 'missed'}`,
166+
);
167+
}
168+
});

‎packages/selectors/src/internal/find.ts‎

Lines changed: 35 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
22
import { AppError } from '@agent-device/kernel/errors';
3+
import { roleSpellingsOfNode } from './node.ts';
34
import { tryParseSelectorChain } from './parse.ts';
45

56
export const FIND_LOCATORS = ['any', 'text', 'label', 'value', 'role', 'id'] as const;
@@ -95,7 +96,7 @@ export function findBestMatchesByLocator(
9596
function matchNode(node: SnapshotNode, locator: FindLocator, query: string): number {
9697
switch (locator) {
9798
case 'role':
98-
return matchRole(node.type, query);
99+
return matchRole(node, query);
99100
case 'label':
100101
return matchText(node.label, query);
101102
case 'value':
@@ -121,24 +122,44 @@ function matchText(value: string | undefined, query: string): number {
121122
return 0;
122123
}
123124

124-
function matchRole(value: string | undefined, query: string): number {
125-
const normalized = normalizeRole(value ?? '');
126-
if (!normalized) return 0;
127-
if (normalized === query) return 2;
128-
if (normalized.includes(query)) return 1;
125+
function matchRole(node: SnapshotNode, query: string): number {
126+
// The package's one role-spelling reader (#3021, `node.ts`): the node's
127+
// canonical `kind` first, then its windowed legacy spelling. Exact scoring
128+
// is against the whole spellings; substring scoring stays on the leaf
129+
// segment because the retired locator substring-matched a last-`.`-segment
130+
// leaf, and an unrecognized dotted class keeps its package path in its
131+
// `kind` — substringing the full kind would let `find role=fenix` score
132+
// `org.mozilla.fenix.ReaderView` where the released query refused it.
133+
const spellings = roleSpellingsOfNode(node);
134+
// The query arrives `normalizeText`-ed, so the exact compare normalizes the
135+
// spellings the same way the selector term's `textEquals` does — that is the
136+
// locator/term agreement #3021 demands for whitespace-bearing class names.
137+
// Substring scoring stays on the raw spelling (the retired locator's leaf).
138+
if (spellings.some((spelling) => normalizeText(spelling) === query)) return 2;
139+
if (spellings.some((spelling) => roleLeafSegment(spelling).includes(query))) return 1;
129140
return 0;
130141
}
131142

132-
export function normalizeText(value: string): string {
133-
return value.trim().toLowerCase().replaceAll(/\s+/g, ' ');
143+
/**
144+
* The substring-scoring view of one role spelling: its last `.`-separated
145+
* segment. This is deliberately the RETIRED LOCATOR's leaf (`split('.').pop()`
146+
* only), not the retired term spelling (`normalizeType`) — the term also
147+
* reduced a trailing `/` segment, and mirroring that here would drop substring
148+
* hits the released locator kept (query `y/z` still scores 1 against kind
149+
* `x.y/z` today). The term's `/`-reduced spelling rides as the windowed exact
150+
* alias instead, so term/locator agreement comes from the exact side, and each
151+
* surface's historical substring meaning is preserved. An AX-stripped query
152+
* (`link` against an `AXLink` node) likewise reaches the exact score through
153+
* the windowed alias, not through substring — the retired term already matched
154+
* that spelling exactly, so promoting it here is the parity #3021 requires.
155+
*/
156+
function roleLeafSegment(spelling: string): string {
157+
const lastDot = spelling.lastIndexOf('.');
158+
return lastDot === -1 ? spelling : spelling.slice(lastDot + 1);
134159
}
135160

136-
function normalizeRole(value: string): string {
137-
let normalized = value.trim();
138-
if (!normalized) return '';
139-
const lastSegment = normalized.split('.').pop() ?? normalized;
140-
normalized = lastSegment.replaceAll(/XCUIElementType/gi, '').toLowerCase();
141-
return normalized;
161+
export function normalizeText(value: string): string {
162+
return value.trim().toLowerCase().replaceAll(/\s+/g, ' ');
142163
}
143164

144165
export type ParsedFindArgs = {

‎packages/selectors/src/internal/match.test.ts‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,121 @@ test('role selector normalization matches Android class names by leaf type', ()
7171
assert.equal(resolved.node.ref, 'a1');
7272
});
7373

74+
// ── #3021: `role=` shares `kind`'s vocabulary, with a legacy-alias window ──
75+
76+
// One tree covering the three vocabulary decisions at once: the coarse kind
77+
// `kind` publishes, the legacy leaf still windowed for it, and a sibling
78+
// vocabulary word that must match NOTHING here.
79+
const ROLE_VOCABULARY_NODES: SnapshotState['nodes'] = [
80+
{
81+
ref: 'text1',
82+
index: 0,
83+
type: 'XCUIElementTypeStaticText',
84+
kind: formatRole('XCUIElementTypeStaticText'),
85+
label: 'Total',
86+
rect: { x: 0, y: 0, width: 120, height: 20 },
87+
hittable: true,
88+
},
89+
{
90+
ref: 'field1',
91+
index: 1,
92+
type: 'android.widget.EditText',
93+
kind: formatRole('android.widget.EditText'),
94+
label: 'Email',
95+
rect: { x: 0, y: 30, width: 120, height: 44 },
96+
hittable: true,
97+
},
98+
{
99+
ref: 'button1',
100+
index: 2,
101+
type: 'XCUIElementTypeButton',
102+
kind: formatRole('XCUIElementTypeButton'),
103+
label: 'Continue',
104+
rect: { x: 0, y: 90, width: 120, height: 44 },
105+
hittable: true,
106+
},
107+
];
108+
109+
function resolveRole(role: string, platform: 'ios' | 'android' = 'ios') {
110+
const chain = parseSelectorChain(`role=${role}`);
111+
return resolveSelectorChain(ROLE_VOCABULARY_NODES, chain, {
112+
platform,
113+
requireRect: true,
114+
requireUnique: true,
115+
});
116+
}
117+
118+
test('role= matches the coarse kind vocabulary: kind=text and kind=text-field resolve', () => {
119+
// The migration's forward direction: these DID NOT match before #3021,
120+
// because the matcher saw the raw leaf (`statictext`, `edittext`).
121+
assert.equal(resolveRole('text')?.node.ref, 'text1');
122+
assert.equal(resolveRole('text-field', 'android')?.node.ref, 'field1');
123+
});
124+
125+
test('role= keeps matching legacy leaf spellings during the alias window', () => {
126+
// The compatibility direction: released scripts write these; they must keep
127+
// resolving to the same nodes they resolved to before #3021.
128+
assert.equal(resolveRole('statictext')?.node.ref, 'text1');
129+
assert.equal(resolveRole('edittext', 'android')?.node.ref, 'field1');
130+
assert.equal(resolveRole('button')?.node.ref, 'button1');
131+
// The window is the node's OWN old spelling: `textfield` was the old leaf of
132+
// the iOS `TextField` class, never of `EditText`, and must not widen onto it.
133+
assert.equal(resolveRole('textfield', 'android'), null);
134+
});
135+
136+
test('role= refuses a sibling vocabulary word that matches no kind or windowed alias', () => {
137+
// Nearest-negatives inside the same vocabulary: `text-view` is one dash from
138+
// `text`, and `switch`/`searchfield` are vocabulary entries (a kind and a
139+
// leaf windowed under kind `search`) that no node here carries.
140+
assert.equal(resolveRole('text-view'), null);
141+
assert.equal(resolveRole('switch'), null);
142+
assert.equal(resolveRole('searchfield'), null);
143+
});
144+
145+
test('role= does not widen a legacy leaf onto kind siblings that never carried it', () => {
146+
// The window is the node's OWN old spelling. `linearlayout` and
147+
// `framelayout` share the coarse kind `group`; matching the leaf may not
148+
// pull in the kind's other members, or every released `role=linearlayout`
149+
// script would silently start hitting FrameLayout rows.
150+
const layoutNodes: SnapshotState['nodes'] = [
151+
'android.widget.LinearLayout',
152+
'android.widget.FrameLayout',
153+
].map((type, idx) => ({
154+
ref: `row${idx + 1}`,
155+
index: idx,
156+
type,
157+
kind: formatRole(type),
158+
rect: { x: 0, y: idx * 50, width: 300, height: 48 },
159+
hittable: true,
160+
}));
161+
const chain = parseSelectorChain('role=linearlayout');
162+
const resolved = resolveSelectorChain(layoutNodes, chain, {
163+
platform: 'android',
164+
requireRect: true,
165+
requireUnique: true,
166+
});
167+
assert.ok(resolved);
168+
assert.equal(resolved.node.ref, 'row1');
169+
});
170+
171+
test('role= agrees with the published kind for every node it matches (#3021 parity)', () => {
172+
// The architecture pin: for every node in the tree, matching by that node's
173+
// own published `kind` must resolve to exactly that node. This is what makes
174+
// the selector/kind divergence that #3021 closes impossible to reintroduce
175+
// silently for any vocabulary entry.
176+
for (const node of ROLE_VOCABULARY_NODES) {
177+
assert.ok(node.kind, 'fixture nodes carry a published kind');
178+
const chain = parseSelectorChain(`role=${JSON.stringify(node.kind)}`);
179+
const resolved = resolveSelectorChain(ROLE_VOCABULARY_NODES, chain, {
180+
platform: 'ios',
181+
requireRect: true,
182+
requireUnique: false,
183+
});
184+
assert.ok(resolved, `role=${node.kind} resolved nothing`);
185+
assert.equal(resolved.node.ref, node.ref, `role=${node.kind} resolved the wrong node`);
186+
}
187+
});
188+
74189
test('focused selector matches snapshot focus state', () => {
75190
const tvNodes: SnapshotState['nodes'] = [
76191
{

‎packages/selectors/src/internal/match.ts‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import type { Platform, PublicPlatform } from '@agent-device/kernel/device';
22
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
3-
import { isNodeEditable, isNodeVisible } from './node.ts';
4-
import { extractNodeText, normalizeType } from '@agent-device/contracts/snapshot';
3+
import { isNodeEditable, isNodeVisible, roleSpellingsOfNode } from './node.ts';
4+
import { extractNodeText } from '@agent-device/contracts/snapshot';
55
import { normalizeText } from './find.ts';
66
import type { Selector, SelectorTerm } from './parse.ts';
77

@@ -22,7 +22,7 @@ function matchesTerm(
2222
case 'id':
2323
return textEquals(node.identifier, String(term.value));
2424
case 'role':
25-
return textEquals(normalizeType(node.type ?? ''), String(term.value));
25+
return matchesRoleKind(node, String(term.value));
2626
case 'label':
2727
return textEquals(node.label, String(term.value));
2828
case 'value':
@@ -55,3 +55,16 @@ function matchesTerm(
5555
function textEquals(value: string | undefined, query: string): boolean {
5656
return normalizeText(value ?? '') === normalizeText(query);
5757
}
58+
59+
/**
60+
* `role=` matches the canonical `kind` vocabulary (#3021): a term value equals
61+
* the node's role kind, or one of the legacy spellings still windowed for that
62+
* node. Both come from {@link roleSpellingsOfNode} — the same `kind`
63+
* `attachRefs` publishes and the same `normalizeType` that RECORDS selector
64+
* chains — so the selector, the `find role=` locator, snapshot output, and
65+
* recorded scripts cannot disagree about a node's role.
66+
*/
67+
function matchesRoleKind(node: SnapshotNode, query: string): boolean {
68+
const normalizedQuery = normalizeText(query);
69+
return roleSpellingsOfNode(node).some((spelling) => normalizeText(spelling) === normalizedQuery);
70+
}

0 commit comments

Comments
 (0)