Skip to content

Commit 11a8e6d

Browse files
committed
bench: help-conformance case for settled scroll-to-find planning
The #1638 extension of the closed --settle grammar to scroll/back is the feature's entire payoff — collapsing scroll-then-observe into one call — and the closed command list is an enumerated N whose enumerator is this bench. The regex over the help text proves the sentence exists; this case checks whether a model plans differently because of it. One focused case, deliberately not coached: a pinned visible-first snapshot (rendered by formatSnapshotText, pinned by the sample-producers gate) whose wanted row is summarized off-screen with no ref anywhere in the output. The tempting pre-#1638 plan is `scroll` plus a separate `snapshot -i`; acceptance is the single settled call. Scoring was verified against eight plan shapes in both directions before recording. Model-backed record (claude-haiku-4-5, 3 trials, current help): 0/3 — but the decomposition is the finding. Settle eligibility GENERALIZED (3/3 trials put --settle on scroll unprompted; the mutation-suffix framing concern did not materialize) and the two-call habit is residual (1/3). All three trials failed on `scroll @e3 down --settle` — the pre-existing #1366 scroll-takes-no-target confusion, which the live CLI recovers with a dedicated hint but a single-shot bench cannot. The recorded gap is therefore a first-30 doc gap (nothing teaches that scroll takes no target), not a settle-eligibility gap; tuning the case until it passes would just delete the evidence.
1 parent 2c6955a commit 11a8e6d

3 files changed

Lines changed: 118 additions & 0 deletions

File tree

‎scripts/__tests__/help-conformance-sample-producers.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import {
55
BROWSERSTACK_CONNECT_SAMPLE,
66
DEVICE_IN_USE_SAMPLE,
77
NOT_SETTLED_SAMPLE,
8+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
89
PRIVATE_AX_RECOVERY_SAMPLE,
910
SETTLE_DIFF_SAMPLE,
1011
SETTLE_DIFF_SAMPLE_NOTES,
@@ -208,6 +209,64 @@ export const SAMPLE_PRODUCERS: SampleProducer[] = [
208209
}).trimEnd();
209210
},
210211
},
212+
{
213+
name: 'OFFSCREEN_TARGET_SNAPSHOT_SAMPLE',
214+
producer: 'the visible-first snapshot renderer with off-screen rows summarized',
215+
sample: OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
216+
render: () => {
217+
// A settings-style scrollable list in an 800pt viewport: five rows fit,
218+
// four more (Privacy & Security, Notifications, Wallpaper, Developer) are
219+
// laid out below it, so visible-first presentation summarizes them and
220+
// none of their refs reach the output.
221+
const row = (index: number, ref: string, label: string, y: number) => ({
222+
index,
223+
ref,
224+
parentIndex: 2,
225+
type: 'Cell',
226+
label,
227+
interactive: true,
228+
hittable: y < 800,
229+
rect: { x: 0, y, width: 390, height: 120 },
230+
});
231+
const nodes = [
232+
{
233+
index: 0,
234+
ref: 'e1',
235+
type: 'Application',
236+
label: 'Preferences',
237+
rect: { x: 0, y: 0, width: 390, height: 800 },
238+
},
239+
{
240+
index: 1,
241+
ref: 'e2',
242+
parentIndex: 0,
243+
type: 'Window',
244+
rect: { x: 0, y: 0, width: 390, height: 800 },
245+
},
246+
{
247+
index: 2,
248+
ref: 'e3',
249+
parentIndex: 1,
250+
type: 'CollectionView',
251+
interactive: true,
252+
rect: { x: 0, y: 60, width: 390, height: 740 },
253+
},
254+
row(3, 'e4', 'General', 60),
255+
row(4, 'e5', 'Display', 190),
256+
row(5, 'e6', 'Sounds', 320),
257+
row(6, 'e7', 'Focus', 450),
258+
row(7, 'e8', 'Screen Time', 580),
259+
row(8, 'e9', 'Privacy & Security', 900),
260+
row(9, 'e10', 'Notifications', 1030),
261+
row(10, 'e11', 'Wallpaper', 1160),
262+
row(11, 'e12', 'Developer', 1290),
263+
];
264+
return formatSnapshotText(
265+
{ nodes, backend: 'xctest', truncated: false },
266+
{ interactiveOnly: true },
267+
).trimEnd();
268+
},
269+
},
211270
{
212271
name: 'DEVICE_IN_USE_SAMPLE',
213272
producer: 'the real session-open by-session conflict producer',

‎scripts/help-conformance-cases.mjs‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
BROWSERSTACK_CONNECT_SAMPLE,
55
DEVICE_IN_USE_SAMPLE,
66
NOT_SETTLED_SAMPLE,
7+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
78
SETTLE_DIFF_SAMPLE,
89
SETTLE_DIFF_SAMPLE_NOTES,
910
SETTLE_TAIL_SAMPLE,
@@ -377,6 +378,46 @@ Use the output already shown to determine whether the feed-search UI is present,
377378
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
378379
],
379380
},
381+
{
382+
// #1638/#1650: the closed --settle grammar grew scroll and back, and this
383+
// extension IS the feature's payoff — collapsing scroll-then-observe into
384+
// one call. The old guidance framed settle as a mutation suffix, and
385+
// scroll reads as navigation, so eligibility generalizing is exactly what
386+
// this case checks. The task deliberately does not mention settle: the
387+
// wanted row is off-screen with no ref anywhere in the output, the
388+
// tempting pre-#1638 plan is `scroll` + a separate `snapshot -i`, and
389+
// acceptance is the single settled call.
390+
id: 'sample-output-offscreen-target-scrolls-settled',
391+
docs: ['--help:first30'],
392+
task: quiz(
393+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
394+
'The task is to open the Notifications row of this list. What command should run next?',
395+
),
396+
expectations: ['validPlanCommands', 'fullPrefix'],
397+
matchers: [
398+
{
399+
id: 'scrollsDownSettled',
400+
pattern: /(?:^|\n)(?:agent-device\s+)?scroll\s+down\b[^\n]*--settle\b/i,
401+
},
402+
],
403+
forbidden: [
404+
// The two-call habit this case exists to catch: a scroll line without
405+
// --settle means a separate observation call is coming.
406+
{
407+
id: 'noUnsettledScroll',
408+
pattern: /(?:^|\n)(?:agent-device\s+)?scroll\b(?:(?!--settle)[^\n])*(?=\n|$)/i,
409+
},
410+
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
411+
{ id: 'noWaitStable', pattern: /wait\s+stable/i },
412+
// Notifications never appears in the output, so any bare @eN press is a
413+
// guessed ref, not a resolved target.
414+
{
415+
id: 'noGuessedRef',
416+
pattern: /(?:^|\n)(?:agent-device\s+)?(?:press|click|fill|longpress)\s+@e\d/i,
417+
},
418+
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
419+
],
420+
},
380421
{
381422
id: 'sample-output-not-settled-needs-observe',
382423
docs: ['--help:first30'],

‎scripts/help-conformance-sample-outputs.mjs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,24 @@ settled after 480ms: +2 -0 (~11 unchanged)
4444
+ @e22 [text] "3 items"`,
4545
};
4646

47+
// Visible-first snapshot of a scrollable list whose remaining rows sit below
48+
// the viewport: the off-screen content is summarized, not listed as refs. The
49+
// scroll-to-find quiz case hangs off this — the wanted row exists but no ref
50+
// for it appears anywhere in the output.
51+
export const OFFSCREEN_TARGET_SNAPSHOT_SAMPLE = {
52+
command: 'agent-device snapshot -i',
53+
output: `Snapshot: 8 visible nodes (12 total)
54+
@e1 [application] "Preferences"
55+
@e2 [window]
56+
@e3 [collection]
57+
@e4 [cell] "General"
58+
@e5 [cell] "Display"
59+
@e6 [cell] "Sounds"
60+
@e7 [cell] "Focus"
61+
@e8 [cell] "Screen Time"
62+
[content below collection hidden]`,
63+
};
64+
4765
// Never-settled press: success response, no diff, NEVER_SETTLED_HINT attached.
4866
export const NOT_SETTLED_SAMPLE = {
4967
command: 'agent-device press @e12 --settle',

0 commit comments

Comments
 (0)