Skip to content

Commit 3937036

Browse files
authored
feat: support --settle on scroll and back (#1638) (#1650)
* feat: support --settle on scroll and back (#1638) Scroll-then-observe and back-then-observe are legitimate agent pairs, but the post-action observation registry never grew past the touch commands, so `--settle` on either was rejected with INVALID_ARGS — burning a tool call each in AppControlBench's bsky-16. Both commands now carry the `settle` descriptor trait, and every surface derives from it rather than a hand list: CLI allowed flags, MCP/SDK input fields, the flag-sourced timeout envelope, and MCP ref-pinning. The CLI flag/metadata helpers moved out of the interaction family into post-action-observation-grammar.ts (back is a system command), and SETTLE_REF_ISSUING_TOOLS became a derivation — a hand list would have silently stopped pinning the new commands' refs. settleAfterInteraction and the new settleObservationCommand are two entry points over one engine: same loop, storage, hints, and diff bounds, with the target-less path supplying its own baseline and no proximity point. The daemon reaches that command through the runtime surface, never by importing `commands/` (R2) — the same seam the touch handlers use for press/fill — and generic-settle.ts is loaded through a lazy `await import` returning a closure, so the interaction runtime subgraph stays out of this dispatcher's static graph (a static edge folded ~18 files into the daemon-server type cycle; R10 caught it). Both of generic-settle's orderings are load-bearing and tested: the baseline is frozen before dispatch (and before the Android dialog preflight), and the observation runs after markDeferredInteractionOutcome so settle's first capture folds in the #1542 stabilization rather than racing it. The ADR 0014 "a settled diff publishes refs" rule moved to settle-ref-issuance.ts, shared by both routes. One divergence is deliberate: scroll/back resolve no element, so the diff baseline is the session's STORED pre-action tree — "settled tree vs the last tree you observed" — not press's freshly resolved pre-action capture. Both commands also switch to preserve-daemon on timeout, which changes the non-settle path too: with --settle their dominant hang mode is now a wedged accessibility bridge, and a timed-out capture must not reset the daemon and lose every session (#1105). The reviewed-set gate records it. Live-validated on an iOS 26.2 simulator (Settings): scroll --settle settled in 1786ms with a +6/-6 diff carrying fresh refs; back --settle in 771ms with +15/-6. Alternating cost runs, one call vs the pair it replaces: scroll 2.9-3.0s vs 5.3-5.6s, back 3.1-3.2s vs 4.7-5.1s. Those include the #1627 deep-capture extension. * fix: render settled-diff refs paste-ready in CLI output A settled diff activates a PARTIAL ref frame (ADR 0014), which admits only the pinned `@eN~s<gen>` form of the refs it issued. The unchanged-interactive tail already rendered that way, but the diff's own added lines rendered the bare `@eN` embedded in the snapshot line — so a CLI caller who copied the ref the diff just handed them got `plain_ref_requires_complete_frame` and had to append the generation by hand. Added lines now render pinned when the response carries `refsGeneration`, exactly like the tail. Removed lines render verbatim: they name elements that just left the screen, and `SettleDiffLine` never gives them a ref. This is not new to scroll/back — press/click/fill/longpress had the same gap since #1101. MCP was never affected: its ref-pin store rewrites plain refs on the way in, which is why the model never sees a suffix. Live: `scroll down --settle` now emits `+ @E14~s218078 [cell] "Game Center"`, and `press @E14~s218078` copied straight out of that line taps successfully. * test: record the pinned-diff-ref bytes in the output-economy baseline Rendering added diff-line refs pinned costs 8 bytes in the two settle CLI text samples (two `~s<gen>` suffixes). The output-economy baseline is the tripwire for exactly this, so the increase takes an explicit reviewed waiver rather than a silent baseline bump — the same one the settled TAIL's pins already carry, for the same ADR 0014 reason. Only `bytes` moves: lines, refs, hints, and shape are unchanged, which is the evidence that this is a suffix on existing refs and not a new payload. Caught by CI, not locally: `pnpm test:unit` runs unit-core and subprocess-stub only, while the Coverage lane runs every vitest project. * test: prove the generic settle degrades when its runtime cannot be built `createGenericSettleRuntime` catches and returns undefined so an observation that cannot even start does not fail an action that already succeeded. That was a claim in a docstring with nothing behind it — the one changed line the coverage gate reported uncovered (95/96). The test puts the session in the state the catch exists for: the router handed us a session that is no longer in the store, so building the settle runtime throws SESSION_NOT_FOUND. The response keeps its scroll result and simply carries no settle payload. Removing the try/catch fails it. * build: teach fallow that vi.mock reaches pinOwnProcessStartTime dynamically Not from this PR: #1642 added `pinOwnProcessStartTime` on main, and its three consumers reach it the only way a Vitest module mock can — `vi.mock(path, async (importOriginal) => (await import('...')).pinOwnProcessStartTime(...))`. Dependency analysis cannot follow that dynamic import to a consumer, so the export reads as dead the moment any PR pulls that file into its audit scope. This PR is the one that did. The entry records the consumers by path and the reason, matching the daemon route-handler entry directly above it, which exists for the same dynamic-`import()` limitation. * refactor: adopt the best of the parallel #1653 implementation Two sessions independently built #1638 (PR #1650 and PR #1653) and converged on the same architecture — trait in the registry, one engine with two entry points, runtime-command seam, lazy import, preserve-daemon, stored-baseline honesty. #1650 continues; this folds in what #1653 did better: - The agent-facing help core loop (cli-help.ts) now names scroll and back as settle-capable. Without this, the benchmarked closed-grammar help line kept instructing agents that --settle is only for press/click/fill/longpress — actively steering the AppControlBench models away from what #1638 shipped. - issueSettleRefs moves into session-snapshot.ts, beside the partial-frame primitive it wraps, deleting the single-function settle-ref-issuance module. - Their seam tests: back reader→writer settle plumbing, back CLI settle rendering, and a trait-less generic command (home) ignoring a stray settle flag rather than observing or rejecting. What #1650 had that #1653 lacked, for the record: the SETTLE_REF_ISSUING_TOOLS registry derivation (without it, MCP never pins a scroll/back settle diff's refs and the partial frame rejects every follow-up), BackCommandResult.settle in contracts, back's MCP output schema, paste-ready pinned diff refs, and the docs/changelog/baseline surfaces. * bench: help-conformance case for settled scroll-to-find planning The #1638 extension of the closed --settle grammar to scroll/back is the feature's entire payoff — collapsing scroll-then-observe into one call — and the closed command list is an enumerated N whose enumerator is this bench. The regex over the help text proves the sentence exists; this case checks whether a model plans differently because of it. One focused case, deliberately not coached: a pinned visible-first snapshot (rendered by formatSnapshotText, pinned by the sample-producers gate) whose wanted row is summarized off-screen with no ref anywhere in the output. The tempting pre-#1638 plan is `scroll` plus a separate `snapshot -i`; acceptance is the single settled call. Scoring was verified against eight plan shapes in both directions before recording. Model-backed record (claude-haiku-4-5, 3 trials, current help): 0/3 — but the decomposition is the finding. Settle eligibility GENERALIZED (3/3 trials put --settle on scroll unprompted; the mutation-suffix framing concern did not materialize) and the two-call habit is residual (1/3). All three trials failed on `scroll @e3 down --settle` — the pre-existing #1366 scroll-takes-no-target confusion, which the live CLI recovers with a dedicated hint but a single-shot bench cannot. The recorded gap is therefore a first-30 doc gap (nothing teaches that scroll takes no target), not a settle-eligibility gap; tuning the case until it passes would just delete the evidence.
1 parent 8030fc1 commit 3937036

43 files changed

Lines changed: 1385 additions & 280 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.fallowrc.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,11 @@
7878
"file": "src/sdk/*.ts",
7979
"exports": ["*"]
8080
},
81+
{
82+
"comment": "#1642: its three consumers (src/__tests__/cli-device-status.test.ts, src/daemon/__tests__/device-claims.test.ts, src/daemon/__tests__/device-claim-prune.test.ts) all reach it the only way a Vitest module mock can — `vi.mock(path, async (importOriginal) => (await import('...host-process-mock.ts')).pinOwnProcessStartTime(importOriginal))`. Dependency analysis cannot follow that dynamic import to a consumer, the same limitation the daemon route-handler entry above records.",
83+
"file": "src/__tests__/test-utils/host-process-mock.ts",
84+
"exports": ["pinOwnProcessStartTime"]
85+
},
8186
{
8287
"comment": "Tool config default exports, loaded by the tool rather than imported.",
8388
"file": "{tsdown.config.ts,vitest.mutation.config.ts,website/rspress.config.ts}",

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## Unreleased
44

5+
- `scroll` and `back` now accept `--settle` (with `--settle-quiet` and `--timeout`), collapsing scroll-then-observe and back-then-observe into one call (#1638). The response carries the same settled payload the touch commands return — verdict, changed-lines diff with fresh refs on added lines, the unchanged-interactive tail, and `refsGeneration` when the settled tree was stored — and is best-effort: it never fails the action. One difference is deliberate: `scroll`/`back` resolve no element, so the diff baseline is the session's stored pre-action tree ("the last tree you observed") rather than a freshly resolved pre-action capture. Both commands now also preserve the daemon on timeout, like the other settle-capable commands.
56
- Security: repository `./agent-device.json` now accepts only project-safe automation defaults. It rejects daemon endpoint/auth/transport/server settings, tenant/run/lease selectors, provider/cloud and Metro connection fields, headers, executable reporter modules, local write destinations, and other operator-controlled values before local module loading or any daemon health/RPC request. Put remote endpoint and token together in protected CI environment variables, user config, an explicit `--config` file, or the existing `connect`/`--remote-config` workflow. Daemon auth tokens no longer travel in serialized command flags.
67
- `viewport` is now rejected during capability admission on Apple targets instead of reaching the device and failing inside dispatch. No Apple backend can resize a screen — simulator and device geometry is fixed by the selected device type — so `viewport` on iOS/iPadOS/tvOS/macOS now fails with `UNSUPPORTED_OPERATION`, `viewport is not supported on this device`, and a hint pointing at `--platform web` and at picking a different simulator. `capabilities` no longer advertises `viewport` on Apple targets. Web viewport resizing (`agent-device viewport 1280 900 --platform web`) is unchanged, and Android was already denied.
78
- `--save-script` is now accepted only by the commands that declare it — `open`, `close`, and `replay`. A hand-built daemon request (or a `batch` step) that set `saveScript` on any other command, such as `record` or `trace`, used to arm script publication and could write a `.ad` artifact; it is now rejected with `INVALID_ARGS` before the request reaches admission, the device, or any handler. CLI, Node, and MCP usage of `--save-script` on its documented commands is unchanged.

‎CONTEXT.md‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -131,11 +131,16 @@ task touches:
131131
eligibility gate, and the module owns only these post-action markers — never ADR 0014 ref-frame
132132
expiry or the ADR 0012/0016 staged protocols. Distinct from the same-response settled
133133
observation below.
134-
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress — the
135-
quiet-window stable loop re-captures until the UI settles, and the response carries the diff vs the
136-
pre-action tree (changed lines only, added lines with fresh refs, `refsGeneration` when the settled
137-
tree was stored). Best-effort: never fails the action; `settled: false` plus a hint on never-quiet
138-
content.
134+
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress and, on
135+
the generic route, scroll/back — the quiet-window stable loop re-captures until the UI settles, and
136+
the response carries the diff vs the pre-action tree (changed lines only, added lines with fresh
137+
refs, `refsGeneration` when the settled tree was stored). Best-effort: never fails the action;
138+
`settled: false` plus a hint on never-quiet content. Which commands support it is a descriptor
139+
trait (`postActionObservation`), and the CLI flags, MCP fields, timeout envelope, and ref-pinning
140+
all derive from it. The two routes differ in ONE way, deliberately: the touch commands diff against
141+
the freshly resolved pre-action capture, while scroll/back — which resolve nothing — diff against
142+
the session's stored pre-action tree, so their diff reads "settled tree vs the last tree you
143+
observed".
139144
- Resolution disclosure (ADR 0012 decision 2): additive `resolution` field on
140145
press/click/fill/longpress responses discloses how the acting path resolved its target —
141146
`runtime`/`unique` or `runtime`/`disambiguated` (with `matchCount`/`winnerDiagnostic`/`tiebreak`/

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ agent-device close
6161

6262
Use refs only from the latest output. Do not assume an earlier `@eN` still identifies the same element. After a command with `--settle`, use the refs in its diff. Take another snapshot only if the diff omits what you need.
6363

64+
`--settle` works the same way on `scroll` and `back`, so scroll-then-observe and back-then-observe are one call too.
65+
6466
Snapshots use the app's accessibility tree. Clear labels, roles, and test IDs make agent runs more reliable. Use screenshots and videos as evidence or when accessibility data is poor. Use refs and selectors for actions and assertions when you can.
6567

6668
![agent-device demo showing Codex using agent-device to create a new contact in the iOS Contacts app from a simple prompt](./website/docs/public/agent-device-contacts.gif)

‎packages/contracts/src/client-gesture.ts‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,10 @@ export type RotateGestureOptions = DeviceCommandBaseOptions & {
135135

136136
export type TransformGestureOptions = DeviceCommandBaseOptions & TransformGestureParams;
137137

138-
export type ScrollOptions = DeviceCommandBaseOptions & {
139-
direction: ScrollInputDirection;
140-
amount?: number;
141-
pixels?: number;
142-
durationMs?: number;
143-
};
138+
export type ScrollOptions = DeviceCommandBaseOptions &
139+
SettleCommandOptions & {
140+
direction: ScrollInputDirection;
141+
amount?: number;
142+
pixels?: number;
143+
durationMs?: number;
144+
};

‎packages/contracts/src/navigation.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import type { BackMode } from './back-mode.ts';
22
import type { DeviceRotation } from './device-rotation.ts';
3+
import type { SettleObservation } from './interaction.ts';
34
import type { TvRemoteButton } from './tv-remote.ts';
45

56
/**
@@ -18,11 +19,17 @@ export type HomeCommandResult = {
1819
message: string;
1920
};
2021

21-
/** `back` — `{ action: 'back', mode, message: 'Back' }`; `mode` defaults to `'in-app'`. */
22+
/**
23+
* `back` — `{ action: 'back', mode, message: 'Back' }`; `mode` defaults to
24+
* `'in-app'`. The one field the generic route may add on top of the dispatch
25+
* handler's literal return: `settle`, the opt-in `--settle` observation
26+
* (#1638), attached after the command by the generic dispatcher.
27+
*/
2228
export type BackCommandResult = {
2329
action: 'back';
2430
mode: BackMode;
2531
message: string;
32+
settle?: SettleObservation;
2633
};
2734

2835
/** `orientation` — `{ action: 'orientation', orientation, message: 'Rotated to <orientation>' }`. */

‎scripts/__tests__/help-conformance-sample-producers.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import {
55
BROWSERSTACK_CONNECT_SAMPLE,
66
DEVICE_IN_USE_SAMPLE,
77
NOT_SETTLED_SAMPLE,
8+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
89
PRIVATE_AX_RECOVERY_SAMPLE,
910
SETTLE_DIFF_SAMPLE,
1011
SETTLE_DIFF_SAMPLE_NOTES,
@@ -208,6 +209,64 @@ export const SAMPLE_PRODUCERS: SampleProducer[] = [
208209
}).trimEnd();
209210
},
210211
},
212+
{
213+
name: 'OFFSCREEN_TARGET_SNAPSHOT_SAMPLE',
214+
producer: 'the visible-first snapshot renderer with off-screen rows summarized',
215+
sample: OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
216+
render: () => {
217+
// A settings-style scrollable list in an 800pt viewport: five rows fit,
218+
// four more (Privacy & Security, Notifications, Wallpaper, Developer) are
219+
// laid out below it, so visible-first presentation summarizes them and
220+
// none of their refs reach the output.
221+
const row = (index: number, ref: string, label: string, y: number) => ({
222+
index,
223+
ref,
224+
parentIndex: 2,
225+
type: 'Cell',
226+
label,
227+
interactive: true,
228+
hittable: y < 800,
229+
rect: { x: 0, y, width: 390, height: 120 },
230+
});
231+
const nodes = [
232+
{
233+
index: 0,
234+
ref: 'e1',
235+
type: 'Application',
236+
label: 'Preferences',
237+
rect: { x: 0, y: 0, width: 390, height: 800 },
238+
},
239+
{
240+
index: 1,
241+
ref: 'e2',
242+
parentIndex: 0,
243+
type: 'Window',
244+
rect: { x: 0, y: 0, width: 390, height: 800 },
245+
},
246+
{
247+
index: 2,
248+
ref: 'e3',
249+
parentIndex: 1,
250+
type: 'CollectionView',
251+
interactive: true,
252+
rect: { x: 0, y: 60, width: 390, height: 740 },
253+
},
254+
row(3, 'e4', 'General', 60),
255+
row(4, 'e5', 'Display', 190),
256+
row(5, 'e6', 'Sounds', 320),
257+
row(6, 'e7', 'Focus', 450),
258+
row(7, 'e8', 'Screen Time', 580),
259+
row(8, 'e9', 'Privacy & Security', 900),
260+
row(9, 'e10', 'Notifications', 1030),
261+
row(10, 'e11', 'Wallpaper', 1160),
262+
row(11, 'e12', 'Developer', 1290),
263+
];
264+
return formatSnapshotText(
265+
{ nodes, backend: 'xctest', truncated: false },
266+
{ interactiveOnly: true },
267+
).trimEnd();
268+
},
269+
},
211270
{
212271
name: 'DEVICE_IN_USE_SAMPLE',
213272
producer: 'the real session-open by-session conflict producer',

‎scripts/help-conformance-cases.mjs‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
BROWSERSTACK_CONNECT_SAMPLE,
55
DEVICE_IN_USE_SAMPLE,
66
NOT_SETTLED_SAMPLE,
7+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
78
SETTLE_DIFF_SAMPLE,
89
SETTLE_DIFF_SAMPLE_NOTES,
910
SETTLE_TAIL_SAMPLE,
@@ -377,6 +378,46 @@ Use the output already shown to determine whether the feed-search UI is present,
377378
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
378379
],
379380
},
381+
{
382+
// #1638/#1650: the closed --settle grammar grew scroll and back, and this
383+
// extension IS the feature's payoff — collapsing scroll-then-observe into
384+
// one call. The old guidance framed settle as a mutation suffix, and
385+
// scroll reads as navigation, so eligibility generalizing is exactly what
386+
// this case checks. The task deliberately does not mention settle: the
387+
// wanted row is off-screen with no ref anywhere in the output, the
388+
// tempting pre-#1638 plan is `scroll` + a separate `snapshot -i`, and
389+
// acceptance is the single settled call.
390+
id: 'sample-output-offscreen-target-scrolls-settled',
391+
docs: ['--help:first30'],
392+
task: quiz(
393+
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
394+
'The task is to open the Notifications row of this list. What command should run next?',
395+
),
396+
expectations: ['validPlanCommands', 'fullPrefix'],
397+
matchers: [
398+
{
399+
id: 'scrollsDownSettled',
400+
pattern: /(?:^|\n)(?:agent-device\s+)?scroll\s+down\b[^\n]*--settle\b/i,
401+
},
402+
],
403+
forbidden: [
404+
// The two-call habit this case exists to catch: a scroll line without
405+
// --settle means a separate observation call is coming.
406+
{
407+
id: 'noUnsettledScroll',
408+
pattern: /(?:^|\n)(?:agent-device\s+)?scroll\b(?:(?!--settle)[^\n])*(?=\n|$)/i,
409+
},
410+
{ id: 'noSnapshot', pattern: /\bsnapshot\b/i },
411+
{ id: 'noWaitStable', pattern: /wait\s+stable/i },
412+
// Notifications never appears in the output, so any bare @eN press is a
413+
// guessed ref, not a resolved target.
414+
{
415+
id: 'noGuessedRef',
416+
pattern: /(?:^|\n)(?:agent-device\s+)?(?:press|click|fill|longpress)\s+@e\d/i,
417+
},
418+
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
419+
],
420+
},
380421
{
381422
id: 'sample-output-not-settled-needs-observe',
382423
docs: ['--help:first30'],

‎scripts/help-conformance-sample-outputs.mjs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,24 @@ settled after 480ms: +2 -0 (~11 unchanged)
4444
+ @e22 [text] "3 items"`,
4545
};
4646

47+
// Visible-first snapshot of a scrollable list whose remaining rows sit below
48+
// the viewport: the off-screen content is summarized, not listed as refs. The
49+
// scroll-to-find quiz case hangs off this — the wanted row exists but no ref
50+
// for it appears anywhere in the output.
51+
export const OFFSCREEN_TARGET_SNAPSHOT_SAMPLE = {
52+
command: 'agent-device snapshot -i',
53+
output: `Snapshot: 8 visible nodes (12 total)
54+
@e1 [application] "Preferences"
55+
@e2 [window]
56+
@e3 [collection]
57+
@e4 [cell] "General"
58+
@e5 [cell] "Display"
59+
@e6 [cell] "Sounds"
60+
@e7 [cell] "Focus"
61+
@e8 [cell] "Screen Time"
62+
[content below collection hidden]`,
63+
};
64+
4765
// Never-settled press: success response, no diff, NEVER_SETTLED_HINT attached.
4866
export const NOT_SETTLED_SAMPLE = {
4967
command: 'agent-device press @e12 --settle',

‎src/cli/parser/__tests__/cli-help-topics.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ test('usage includes agent workflows, config, environment, and examples footers'
9191
assert.match(usageText, /Default app loop: agent-device open <app>/);
9292
assert.match(
9393
usageText,
94-
/Use --settle only on planned press, click, fill, or longpress commands; never add it to open, snapshot, or close/,
94+
/Use --settle only on planned press, click, fill, longpress, scroll, or back commands; never add it to open, snapshot, or close/,
9595
);
9696
assert.match(usageText, /type never accepts --settle/);
9797
assert.match(usageText, /explicit success confirmation is visible, stop/);

0 commit comments

Comments
 (0)