Skip to content

Commit f149e43

Browse files
thymikeeclaude
andauthored
docs(help): point verification at read-only commands, not bare find (#4497)
* docs(help): point verification at read-only commands, not bare find Bare find clicks by default, so listing it as a verification step made agents tap while checking. Root help, help manual-qa, the MCP guide and the skill card now name wait, is and get, and say find acts unless given exists. The help conformance scorers no longer count bare find as verification. Fixes #4485 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(help): drop bare find from every verification line Address review: help workflow, react-native and validate still offered bare find as read-only, and the conformance matchers accepted any later word "exists". One shared matcher now requires exists as the find action, with scorer counterexamples for bare and mutating find. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(help): accept escaped quotes in a find ... exists query Address review: the canonical plan keeps escaped quotes, and the double-quoted query pattern stopped at the first one. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(help): show a concrete find ... exists example Address review: find <locator> exists hid that the locator is text, label, value, role or id followed by the query. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(help): classify find verification with parseFindArgs Address review: the regex find grammar missed a single-word query with an apostrophe (find text "Today's" exists), because the canonical plan leaves it unquoted. One token-based predicate, hasFindExists, now reads each find command's parsed positionals through parseFindArgs, checks for the exists action and the optional subject, and serves both the scorer and the four case matchers. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(help): keep scoring a find with an unsupported action Address review: parseFindArgs throws on an unsupported action, so an invalid find plan crashed the scorer; it now scores false. A focused test drives the check through the real plan tokenizer and restores the escaped-quote case next to the apostrophe one. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(help): parse find commands in the TS command validator Address review: importing @agent-device/selectors into the scorer modules broke bare node on the Node 22 floor, where .ts imports need type stripping. The command validator, which already runs with --experimental-strip-types, now attaches each find command's parseFindArgs action and query, and the plain-JS scorer reads that. The scorer modules import no TypeScript again. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 4aae9ba commit f149e43

18 files changed

Lines changed: 127 additions & 23 deletions

‎scripts/__tests__/help-conformance-expectation-fixtures.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,8 +161,13 @@ export const EXPECTATION_FIXTURES: Record<string, ExpectationFixture> = {
161161
id: 'noVerificationCommand',
162162
commands: ['agent-device press @e5 --settle', 'agent-device close'],
163163
},
164+
// Bare find clicks; a later word "exists" in the query or a fill value is no exists action.
165+
{ id: 'bareFindClicks', commands: ['agent-device find text "Sent"'] },
166+
{ id: 'existsInFindQuery', commands: ['agent-device find text "Sent exists"'] },
167+
{ id: 'findFillsExists', commands: ['agent-device find text Sent fill exists'] },
164168
],
165-
metamorphic: { commands: ['agent-device is visible label="Welcome back"'] },
169+
// A single-word query with an apostrophe is canonicalized without quotes.
170+
metamorphic: { commands: [`agent-device find text "Today's" exists`] },
166171
},
167172
usesDogfoodEvidence: {
168173
passing: {
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import { hasFindExists } from '../help-conformance-expectations.mjs';
4+
import { validatePlanCommands } from '../help-conformance-plan-validator.mjs';
5+
6+
// The find verification check reads plan commands through the real plan tokenizer, so quoting is
7+
// the tokenizer's job and the find grammar is parseFindArgs'.
8+
async function verifies(command: string, subject?: string): Promise<boolean> {
9+
return hasFindExists(await validatePlanCommands([command]), subject);
10+
}
11+
12+
test('find ... exists verifies, whatever quotes its query holds', async () => {
13+
assert.equal(await verifies(`agent-device find text "Today's" exists`), true);
14+
assert.equal(await verifies(String.raw`agent-device find text "Welcome \"back\"" exists`), true);
15+
assert.equal(await verifies('agent-device find "Welcome back" exists', 'welcome'), true);
16+
});
17+
18+
test('a find that acts, or names another subject, does not verify', async () => {
19+
assert.equal(await verifies('agent-device find text "Sent"'), false);
20+
assert.equal(await verifies('agent-device find text Sent fill exists'), false);
21+
assert.equal(await verifies('agent-device find text "Orders" exists', 'welcome'), false);
22+
});
23+
24+
test('a find with an unsupported action scores false instead of throwing', async () => {
25+
assert.equal(await verifies('agent-device find text Sent swipe'), false);
26+
});

‎scripts/help-conformance-case-checks.d.mts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,17 @@ import type { PlanCommandValidation } from './help-conformance-plan-validator.mj
44

55
export type CasePatternCheck = { id: string; pattern: RegExp };
66

7+
/** The plan a check reads: its canonical text and each command's tokens and parse. */
8+
export type CaseCheckContext = {
9+
joined: string;
10+
commandValidation: readonly PlanCommandValidation[];
11+
};
12+
13+
export type CaseTestCheck = { id: string; test: (context: CaseCheckContext) => boolean };
14+
715
export type CaseChecks = {
816
expectations?: readonly string[];
9-
matchers?: readonly CasePatternCheck[];
17+
matchers?: readonly (CasePatternCheck | CaseTestCheck)[];
1018
forbidden?: readonly CasePatternCheck[];
1119
};
1220

‎scripts/help-conformance-case-checks.mjs‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
import { opensAndCloses, usesValidationPrep } from './help-conformance-expectations.mjs';
1+
import {
2+
hasFindExists,
3+
opensAndCloses,
4+
usesValidationPrep,
5+
} from './help-conformance-expectations.mjs';
26

37
const EXPECTATION_SCORERS = {
48
validPlanCommands: ({ commands, commandValidation }) =>
@@ -17,7 +21,8 @@ const EXPECTATION_SCORERS = {
1721
// that can tell whether the model actually chained.
1822
usesConfidentChaining: ({ commands }) => commands.some((command) => /&&/.test(command)),
1923
noWaitStable: ({ joined }) => !joined.includes('wait stable'),
20-
verifiesNamedExpectation: ({ joined }) => /\b(wait|is|get|find)\b/.test(joined),
24+
verifiesNamedExpectation: ({ joined, commandValidation }) =>
25+
/\b(?:wait|is|get)\b/.test(joined) || hasFindExists(commandValidation),
2126
usesDogfoodEvidence: ({ joined }) =>
2227
/(?:\bscreenshot\b|\brecord\b|\blogs\b|\bnetwork\b|\bperf\b|\btrace\b|dogfood-output)/i.test(
2328
joined,
@@ -59,9 +64,9 @@ function resolveChecks(testCase) {
5964
id,
6065
test: (context) => scoreExpectation(id, context),
6166
}));
62-
const matched = (testCase.matchers ?? []).map(({ id, pattern }) => ({
67+
const matched = (testCase.matchers ?? []).map(({ id, pattern, test }) => ({
6368
id,
64-
test: (context) => pattern.test(context.joined),
69+
test: test ?? ((context) => pattern.test(context.joined)),
6570
}));
6671
const forbidden = (testCase.forbidden ?? []).map(({ id, pattern }) => ({
6772
id,

‎scripts/help-conformance-cases.mjs‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import {
1414
STALE_REF_SAMPLE,
1515
sampleText,
1616
} from './help-conformance-sample-outputs.mjs';
17+
import { hasFindExists } from './help-conformance-expectations.mjs';
1718

1819
// Raw-coordinate fallback the quiz cases forbid: a click/fill/press targeting
1920
// bare numbers instead of a ref or selector.
@@ -180,7 +181,11 @@ export const CASES = [
180181
matchers: [
181182
{ id: 'movesFocusWithRemote', pattern: /\btv-remote\s+press\s+right\b/i },
182183
{ id: 'activatesWithSelect', pattern: /\btv-remote\s+press\s+select\b/i },
183-
{ id: 'verifiesOutcome', pattern: /\b(?:is\s+focused|wait\b|find\b)/i },
184+
{
185+
id: 'verifiesOutcome',
186+
test: ({ joined, commandValidation }) =>
187+
/\b(?:is\s+focused|wait)\b/i.test(joined) || hasFindExists(commandValidation),
188+
},
184189
],
185190
forbidden: [
186191
// Focus-first surface: activation goes through tv-remote select, not a
@@ -239,7 +244,12 @@ export const CASES = [
239244
},
240245
{ id: 'verifiesBackendWithDoctor', pattern: /\bagent-device\s+web\s+doctor\b/i },
241246
{ id: 'usesWebPlatform', pattern: /--platform\s+web\b/i },
242-
{ id: 'verifiesWelcomeText', pattern: /\b(?:wait|is|find)\b[^\n]*welcome/i },
247+
{
248+
id: 'verifiesWelcomeText',
249+
test: ({ joined, commandValidation }) =>
250+
/\b(?:wait|is)\b[^\n]*welcome/i.test(joined) ||
251+
hasFindExists(commandValidation, 'welcome'),
252+
},
243253
],
244254
forbidden: [
245255
// help web: native mobile/desktop setup commands are out of scope for
@@ -314,7 +324,12 @@ export const CASES = [
314324
id: 'opensFreshAfterInstall',
315325
pattern: /\binstall\b[\s\S]*\n[^\n]*\bopen\s+com\.example\.orders\b[^\n]*--relaunch\b/i,
316326
},
317-
{ id: 'verifiesFirstScreen', pattern: /\b(?:wait|find|is|get)\b[^\n]*orders/i },
327+
{
328+
id: 'verifiesFirstScreen',
329+
test: ({ joined, commandValidation }) =>
330+
/\b(?:wait|is|get)\b[^\n]*orders/i.test(joined) ||
331+
hasFindExists(commandValidation, 'orders'),
332+
},
318333
],
319334
forbidden: [
320335
// help workflow: install for a first install; reinstall only when
@@ -372,7 +387,8 @@ export const CASES = [
372387
},
373388
{
374389
id: 'verifiesLoginSucceeded',
375-
pattern: /\b(?:wait|is|get|find)\b/i,
390+
test: ({ joined, commandValidation }) =>
391+
/\b(?:wait|is|get)\b/i.test(joined) || hasFindExists(commandValidation),
376392
},
377393
{ id: 'publishesWithoutClosing', pattern: /\bagent-device\s+session\s+save-script\b/i },
378394
],

‎scripts/help-conformance-command-validator.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
import { pathToFileURL } from 'node:url';
22
import { parseArgs } from '../src/cli/parser/args.ts';
33
import { isCommandName, readInputFromCli } from '@agent-device/command-surface';
4+
import { parseFindArgs } from '@agent-device/selectors';
45

56
type ValidationKind = 'agent-device-grammar' | 'pseudo-ref';
67

78
type ParsedAgentCommand = {
89
command: string;
910
positionals: string[];
11+
/** A `find` command's action and query, as `parseFindArgs` reads them; the scorers stay plain JS. */
12+
find?: { action: string; query: string };
1013
};
1114

1215
type ValidationResult = { valid: true } | { valid: false; kind: ValidationKind; error: string };
@@ -91,10 +94,21 @@ function validatePlanCommand(value: unknown): PlanValidationResult {
9194
agentCommand: {
9295
command: parsed.command,
9396
positionals: [...parsed.positionals],
97+
...(parsed.command === 'find' ? readFind(parsed.positionals) : {}),
9498
},
9599
};
96100
}
97101

102+
/** An unsupported action is no find the plan can act on, so it carries no parsed find. */
103+
function readFind(positionals: string[]): Pick<ParsedAgentCommand, 'find'> {
104+
try {
105+
const { action, query } = parseFindArgs(positionals);
106+
return { find: { action, query } };
107+
} catch {
108+
return {};
109+
}
110+
}
111+
98112
function runCli(): void {
99113
const encodedInput = process.argv[2];
100114
if (!encodedInput) throw new Error('Expected a JSON argv array.');
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
11
// Type surface for the plain-Node expectation scorers (they must stay runnable by bare `node`).
22
// Keep it in step with the exports of `./help-conformance-expectations.mjs`.
3+
import type { PlanCommandValidation } from './help-conformance-plan-validator.mjs';
4+
35
export type ExpectationContext = Readonly<{ commands: readonly string[] }>;
46

57
export function opensAndCloses(context: ExpectationContext): boolean;
68

79
export function usesValidationPrep(context: ExpectationContext): boolean;
10+
11+
export function hasFindExists(
12+
commandValidation: readonly PlanCommandValidation[],
13+
subject?: string,
14+
): boolean;

‎scripts/help-conformance-expectations.mjs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,18 @@ function isSupportedBuildCommand(command) {
1919
function isPnpmScript(command, script) {
2020
return new RegExp(`^pnpm\\s+(?:run\\s+)?${script}(?:\\s|$)`).test(command ?? '');
2121
}
22+
23+
/**
24+
* Whether the plan holds a read-only `find`: one whose action is `exists` (bare find clicks).
25+
* The command validator parses each find with `parseFindArgs`; `subject`, when given, must appear
26+
* in the query. A find that failed validation carries no parse and does not verify.
27+
*/
28+
export function hasFindExists(commandValidation, subject) {
29+
return commandValidation.some(({ agentCommand }) => {
30+
const find = agentCommand?.find;
31+
return (
32+
find?.action === 'exists' &&
33+
(subject === undefined || find.query.toLowerCase().includes(subject))
34+
);
35+
});
36+
}

‎scripts/help-conformance-plan-validator.d.mts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,12 @@
33
// `./help-conformance-plan-validator.mjs`; keep it in step with the exports there.
44
export type PlanCommandIssue = { kind: string; error: string };
55

6-
export type PlanAgentCommand = { command: string; positionals: string[] };
6+
export type PlanAgentCommand = {
7+
command: string;
8+
positionals: string[];
9+
/** A `find` command's action and query, as `parseFindArgs` reads them. */
10+
find?: { action: string; query: string };
11+
};
712

813
export type PlanCommandValidation = {
914
command: string;

‎skills/agent-device/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ agent-device open <app> --foreground
1313

1414
That starts the session and returns the initial interactive snapshot with `@refs`.
1515

16-
Loop: act with `press|click|fill|longpress <target> ... --settle`, `scroll <direction> --settle`, or `back --settle`; continue from the printed diff, verify the named expectation (`wait text "..."`, `is`, `get`, or `find`), then run `agent-device close`, or `agent-device close --shutdown` if your session booted a local simulator or emulator, unless the user asks to keep it running.
16+
Loop: act with `press|click|fill|longpress <target> ... --settle`, `scroll <direction> --settle`, or `back --settle`; continue from the printed diff, verify the named expectation with a read-only command (`wait text "..."`, `is`, or `get`; bare `find` clicks, so check with `find text "..." exists`), then run `agent-device close`, or `agent-device close --shutdown` if your session booted a local simulator or emulator, unless the user asks to keep it running.
1717

1818
Reaching an off-screen target is one command, not a scroll-and-check loop: `scroll down --until <selector>` scrolls until that element is on screen, and `scroll bottom` runs to the end of the content. Repeated bare `scroll down` calls are the slow way to find something.
1919

0 commit comments

Comments
 (0)