Skip to content

Commit 5c5fa01

Browse files
authored
feat: --settle returns the settled diff in the interaction response (#1101) (#1106)
* feat: --settle returns the settled diff in the interaction response (#1101) press/click/fill/longpress --settle executes the action, waits for the UI to go quiet (wait stable's loop, shared via stable-capture.ts), and returns the settled diff vs the pre-action tree in the same response — one round trip instead of the interact -> observe pair. - payload: changed lines only (bounded), summary counts, added-line refs, refsGeneration; best-effort (settled:false + hint on never-quiet content, never an action failure); --verify shares the settle captures - ref issuance: the settled tree becomes the session snapshot; a diff-carrying settle response clears snapshotRefsStale and the MCP layer merge-only re-pins added-line refs at the settle generation - grammar: --settle + --settle-quiet <ms> + --timeout <ms> (flag-sourced descriptor budget with new envelope:'widen' semantics mirroring wait) - ADR 0011: new settleObservation guarantee classified on every path with contract scenarios per enforced/delegated cell * test: give the two contention-flaky doctor scenarios explicit budgets The doctor provider scenarios sit at ~5s of real daemon-harness work on a loaded host and flake at vitest's 5s default during full-suite runs (the known contention flake AGENTS.md documents). Same in-file precedent as the Metro-probe scenario's 10s budget. * fix: move SettleParams to contracts to satisfy the layering DAG daemon/handlers/interaction-flags.ts imported the type across the daemon -> commands boundary (R2 commands-floor). The tuning params are part of the interaction contract like SettleObservation, so they live in contracts/interaction.ts and both layers import from there. * feat: keep settle diffs content-first — drop Key nodes, added lines win the cap Bluesky dogfood: a fill that summons the iOS keyboard spent 49 of the 80 capped diff lines spelling out QWERTY keys, and a screen transition with 269 removals could starve out the added lines entirely. Key-type nodes are now filtered from both diff sides (the [keyboard] container line still signals presence), and under truncation added lines — the ones carrying fresh refs — win slots over removals. * docs: state the core loop in the top-level help starting point Benchmarked with headless haiku/sonnet agents given only --help: both models skipped the help-workflow pointer and started with plain snapshot (38KB payloads they then had to re-read from files). One core-loop line at the starting point is what teaches snapshot -i and --settle to models that never read a second help page. * fix: preserve settle digest refs for mcp * fix: reduce settle fallow complexity * fix: surface settle output in CLI text * fix: complete settle handling for longpress * refactor: localize daemon timeout envelopes * refactor: deepen post-action observation * refactor: centralize post-action observation planning * refactor: derive settle capability from descriptors * refactor: trim settle descriptor helpers
1 parent db69124 commit 5c5fa01

60 files changed

Lines changed: 3312 additions & 381 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CONTEXT.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
- Coverage manifest: `CONTRACT_COVERAGE` export beside each interaction contract test file claiming which matrix cells it proves; the coverage gate requires every enforced/delegated cell to be claimed and rejects overclaims of waived cells.
5454
- Delegation-on-error: a fast path falling back to the runtime path on semantic failure shapes. It closes failure-side guarantee cells only — never success-path parity.
5555
- Ref generation pin: optional `~s<n>` suffix on an @ref carrying the snapshot generation it was minted from. Accepted as input everywhere, emitted by no tree output (snapshot token budget), auto-appended by the MCP layer, stripped and ignored by replay.
56+
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress — the quiet-window stable loop re-captures until the UI settles, and the response carries the diff vs the pre-action tree (changed lines only, added lines with fresh refs, `refsGeneration` when the settled tree was stored). Best-effort: never fails the action; `settled: false` plus a hint on never-quiet content.
5657
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
5758
- Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
5859
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect. The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the next command reacquire the app through normal activation.

‎docs/adr/0008-command-descriptor-registry.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ every consumer table is **derived** by pure, parity-tested projection:
3131
- The descriptor composes a `surface` facet (owned by `src/commands/**`: identity, CLI schema/reader, MCP),
3232
a `capability` facet (owned by `src/core/capabilities`), and a `daemon` facet (route + request-policy
3333
traits, **owned under `src/daemon/`** per ADR 0003), plus a typed result.
34+
- Narrow command traits that affect multiple projections, such as interaction post-action observation
35+
(`--settle` / `--verify`), live on the descriptor so CLI flags, command metadata, timeout policy, and
36+
tests derive from one fact instead of repeating command-name lists.
3437
- The public catalog, capability matrix, daemon command registry, batch allowlist, MCP tool list, CLI
3538
schema, and the Node client surface become pure projections of the descriptor set. The
3639
`src/core/dispatch.ts` `switch` is replaced by a total map keyed on the command-name union, so a missing

‎scripts/integration-progress-model.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,9 @@ function summarizeProviderScenarioFlagCoverage(files) {
184184
['batchMaxSteps', 'batch max-step guard', ['maxSteps']],
185185
['findFirst', 'find first disambiguation'],
186186
['findLast', 'find last disambiguation'],
187-
['verify', 'post-action evidence capture on press/click/fill'],
187+
['verify', 'descriptor post-action evidence capture'],
188+
['settle', 'descriptor post-action settled-diff observation'],
189+
['settleQuietMs', 'settle quiet-window tuning'],
188190
];
189191
const sources = files.map((file) => fs.readFileSync(file, 'utf8')).join('\n');
190192
return flagTargets.map(([key, reason, aliases = []]) => {

‎src/cli/parser/cli-flags.ts‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ export type CliFlags = CloudProviderProfileFields &
106106
pixels?: number;
107107
doubleTap?: boolean;
108108
verify?: boolean;
109+
settle?: boolean;
110+
settleQuietMs?: number;
109111
clickButton?: ClickButton;
110112
backMode?: BackMode;
111113
pauseMs?: number;
@@ -211,6 +213,11 @@ export const REPEATED_TOUCH_FLAGS = flagKeys(
211213
'jitterPx',
212214
'doubleTap',
213215
);
216+
// Interaction commands with the descriptor post-action observation trait use
217+
// these flags for `--settle` (#1101). --timeout doubles as the settle deadline
218+
// (flag-sourced budget on the interaction descriptors, mirroring wait's
219+
// positional budget).
220+
export const SETTLE_FLAGS = flagKeys('settle', 'settleQuietMs', 'timeoutMs');
214221
export const REPLAY_FLAGS = flagKeys('replayUpdate', 'replayEnv');
215222

216223
const FLAG_DEFINITIONS: readonly FlagDefinition[] = [
@@ -848,6 +855,22 @@ const FLAG_DEFINITIONS: readonly FlagDefinition[] = [
848855
usageDescription:
849856
'Capture cheap post-action evidence (AX digest, node counts, changedFromBefore) instead of a follow-up snapshot',
850857
},
858+
{
859+
key: 'settle',
860+
names: ['--settle'],
861+
type: 'boolean',
862+
usageLabel: '--settle',
863+
usageDescription:
864+
'After the action, wait for the UI to go quiet and return the settled diff vs the pre-action tree in the same response (best-effort; never fails the action)',
865+
},
866+
{
867+
key: 'settleQuietMs',
868+
names: ['--settle-quiet'],
869+
type: 'int',
870+
min: 0,
871+
usageLabel: '--settle-quiet <ms>',
872+
usageDescription: 'Settle: quiet window the UI must hold to count as settled (default 500ms)',
873+
},
851874
{
852875
key: 'clickButton',
853876
names: ['--button'],
@@ -1051,7 +1074,7 @@ const FLAG_DEFINITIONS: readonly FlagDefinition[] = [
10511074
min: 1,
10521075
usageLabel: '--timeout <ms>',
10531076
usageDescription:
1054-
'Prepare/Replay/Snapshot/Test: maximum wall-clock time for the command or attempt',
1077+
'Prepare/Replay/Snapshot/Test: maximum wall-clock time for the command or attempt. With --settle: the settle-wait deadline (default 10s)',
10551078
},
10561079
{
10571080
key: 'retries',

‎src/cli/parser/cli-help.ts‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,18 +55,25 @@ const AGENT_START_LINES = [
5555
'Default to agent-device for installs, opens, snapshots, interactions, screenshots, logs, network/perf evidence, and verification.',
5656
'Use raw adb, simctl, xcrun, or platform scripts only when this help calls out a tool gap or platform setup step.',
5757
'Start with agent-device help workflow to understand the core loop and how to use the tool.',
58+
// Benchmarked 2026-07-05 (#1101): agents that only read --help skipped the
59+
// help-workflow pointer and fell into plain-snapshot loops; stating the loop
60+
// here is what makes small models pick snapshot -i and --settle unprompted.
61+
'Core loop: open <app> -> snapshot -i (interactive tree with @refs) -> press/click/fill <target> --settle (returns the settled diff with fresh @refs) -> repeat. Verify with diff snapshot -i.',
5862
] as const;
5963

6064
const AGENT_QUICKSTART_LINES = [
6165
'Planning output contract: when asked to plan commands, output command lines only: no prose, numbering, Markdown fences, pipes, or shell helpers.',
6266
'Default loop: devices/apps -> open -> snapshot -i -> press/fill/get/is/wait/find -> verify with diff snapshot -> close.',
6367
'Verify a mutation with diff snapshot (or diff snapshot -i), not a full snapshot: it prints only the added/removed/changed lines since the last snapshot in this session, so confirming an action costs a few lines instead of the whole tree.',
64-
'Use selectors or refs as positional targets: id="submit", label="Allow", or @e12 from snapshot -i.',
68+
'Collapse act+observe into one call with --settle on press/click/fill/longpress: the response waits for the UI to go quiet and carries the settled diff with fresh refs (settled: false plus a hint on never-quiet content; the action itself never fails). Tune with --settle-quiet <ms> and --timeout <ms>.',
69+
'Treat the --settle diff as the post-action observation: if it already shows the next target or final evidence, act on that fresh ref or answer from it instead of taking another snapshot.',
70+
'Network-backed or debounced results may arrive after the --settle quiet window; follow the settled action with wait text "Expected result" or wait <selector> instead of polling full snapshots.',
71+
'Use selectors or refs as positional targets: id="submit", label="Allow", role=button label="Send", or @e12 from snapshot -i. Do not write role names as selector keys, such as button="Search".',
6572
'Pin a ref to the snapshot that minted it with ~s<n> (n = refsGeneration in the snapshot response): press @e12~s4. Pinned refs get exact staleness warnings instead of the coarse tree-changed one; plain refs stay valid input.',
6673
'Plain snapshot reads state; snapshot -i refreshes current interactive refs only.',
6774
'Default snapshot text is an agent-facing, token-efficient view for planning and targeting actions.',
6875
'Read-only visible/state question: use snapshot/get/is/find; use snapshot -i only when refs are needed.',
69-
'Anti-pattern: snapshot -i followed by snapshot -i | grep ...; prior refs stay valid until app state changes, and --force-full is the explicit full re-read.',
76+
'Anti-pattern: snapshot -i followed by snapshot -i | grep ... or hiding command output with 2>/dev/null | jq ...; inspect raw agent-device output first, since hints and errors are part of the planning contract.',
7077
'Truncated text/input preview: expand first with snapshot -s @e12, not get text.',
7178
'React Native apps: read help react-native for Metro, DevTools routing, and RN-specific blockers; use react-native dismiss-overlay for LogBox/RedBox overlays.',
7279
'React Native JS memory leaks: read help cdp; use heap usage samples for a quick signal, then snapshot diff/leak-triplet for retained object proof.',
@@ -142,6 +149,7 @@ Fresh machine or first iOS run:
142149
Command shape:
143150
Plans should use agent-device commands, not raw platform tools, pseudo commands, package-manager aliases, or helper prose.
144151
If the user asks for a command plan, final output should be command lines only: no intro sentence, numbered list, Markdown fence, shell pipe, grep/head/tail helper, or explanatory bullets.
152+
While exploring, do not pipe or redirect agent-device output through jq/grep/head/tail or 2>/dev/null; raw output carries refs, warnings, hints, and diagnostics needed for the next step.
145153
Put subcommand first, then positionals, then flags:
146154
agent-device open com.example.app --session checkout --platform android --relaunch
147155
agent-device record start ./checkout.mp4 --session checkout
@@ -176,8 +184,9 @@ Snapshots and refs:
176184
@e14 [cell] label="Profiles" focused -> tvOS focus is currently on this row.
177185
[off-screen below] 4 items: "Privacy", "About" -> scroll down, then snapshot -i; those are hints, not refs.
178186
Re-snapshot after navigation, submit, typing/fill, modal/list/reload/dynamic changes when you need new refs.
179-
Anti-pattern: snapshot -i followed by snapshot -i | grep ...
187+
Anti-pattern: snapshot -i followed by snapshot -i | grep ..., or adding 2>/dev/null | jq ... before reading the raw command output.
180188
Refs from the first snapshot remain valid until you press, click, fill, type, scroll, go back, wait for async UI, or otherwise change app state.
189+
A --settle response is the observation for that mutation. If its diff already shows the next target or final evidence, use that fresh ref or answer from the diff instead of taking a follow-up snapshot.
181190
After a mutation, prefer a known selector/label directly (for example press 'label="Send"') because interaction commands refresh interactive state internally. If you need to discover the new control, use snapshot -i, or snapshot -i -s "Composer" when a stable container label/id can scope the refresh.
182191
If typing/fill opened the keyboard or changed layout and the next target has no stable selector, run snapshot -i, use the fresh ref, then verify with wait/find or diff snapshot -i.
183192
For a targeted query, use find/get/is. If you truly need the full tree again, pass --force-full.
@@ -188,6 +197,7 @@ Snapshots and refs:
188197
189198
Selectors:
190199
Use selectors as positional targets: id="field-email" or label="Allow".
200+
Selector terms are key=value filters such as id="submit", label="Search", text="Search", or role=button label="Search". Do not write role names as keys, such as button="Search"; use role=button label="Search" or the @ref from the latest snapshot/settle diff.
191201
Do not use CSS selectors, pseudo refs, --selector, --text, or raw x/y when refs/selectors exist.
192202
agent-device fill 'id="catalog-search"' "tart" --delay-ms 80
193203
agent-device press 'id="submit-order"'
@@ -224,6 +234,7 @@ Read-only and waits:
224234
agent-device wait 'label="Ready"' 3000
225235
agent-device wait stable
226236
agent-device wait stable 500 10000
237+
For network-backed search/typeahead, --settle confirms the local UI quieted after fill/press; use wait text "Expected result" or wait <result selector> for server-loaded content that can arrive later.
227238
agent-device find "Increment" press --json
228239
For async/list text presence, prefer wait text over is visible when no interaction is needed.
229240
wait stable [quietMs] [timeoutMs] (defaults 500/10000) replaces guessed sleeps between actions and snapshots in the core loop: it polls the interactive-only tree and resolves once two or more consecutive captures are unchanged for quietMs, or fails with the standard wait-timeout shape plus the last capture stats (captures, nodeCount) on timeout; use it after relaunch/navigation instead of a fixed wait <ms> before the next snapshot or action.

‎src/client/client-normalizers.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -343,6 +343,8 @@ export function buildFlags(options: InternalRequestOptions): CommandFlags {
343343
pixels: options.pixels,
344344
doubleTap: options.doubleTap,
345345
verify: options.verify,
346+
settle: options.settle,
347+
settleQuietMs: options.settleQuietMs,
346348
clickButton: options.clickButton,
347349
pauseMs: options.pauseMs,
348350
pattern: options.pattern,

‎src/client/client-types.ts‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -634,10 +634,25 @@ export type CaptureDiffOptions = DeviceCommandBaseOptions &
634634
out?: string;
635635
};
636636

637+
/**
638+
* Opt-in (#1101): after the action, wait for the UI to go quiet and return the
639+
* settled diff vs the pre-action tree (`settle` on the result) in the same
640+
* response. Best-effort — never fails the action. `settleQuietMs` tunes the
641+
* quiet window (default 500ms); `timeoutMs` bounds the settle wait (default
642+
* 10s) when `settle` is true. A bare `timeoutMs` without `settle` is ignored
643+
* for compatibility; `settleQuietMs` still requires `settle`.
644+
*/
645+
type SettleCommandOptions = {
646+
settle?: boolean;
647+
settleQuietMs?: number;
648+
timeoutMs?: number;
649+
};
650+
637651
export type ClickOptions = DeviceCommandBaseOptions &
638652
SelectorSnapshotCommandOptions &
639653
InteractionTarget &
640-
RepeatedPressOptions & {
654+
RepeatedPressOptions &
655+
SettleCommandOptions & {
641656
button?: ClickButton;
642657
/**
643658
* Opt-in (#1047): return cheap post-action evidence (AX digest, node counts,
@@ -650,13 +665,15 @@ export type ClickOptions = DeviceCommandBaseOptions &
650665
export type PressOptions = DeviceCommandBaseOptions &
651666
SelectorSnapshotCommandOptions &
652667
InteractionTarget &
653-
RepeatedPressOptions & {
668+
RepeatedPressOptions &
669+
SettleCommandOptions & {
654670
verify?: boolean;
655671
};
656672

657673
export type LongPressOptions = DeviceCommandBaseOptions &
658674
SelectorSnapshotCommandOptions &
659-
InteractionTarget & {
675+
InteractionTarget &
676+
SettleCommandOptions & {
660677
durationMs?: number;
661678
};
662679

@@ -702,7 +719,8 @@ export type TypeTextOptions = DeviceCommandBaseOptions & {
702719

703720
export type FillOptions = DeviceCommandBaseOptions &
704721
SelectorSnapshotCommandOptions &
705-
InteractionTarget & {
722+
InteractionTarget &
723+
SettleCommandOptions & {
706724
text: string;
707725
delayMs?: number;
708726
verify?: boolean;
@@ -933,6 +951,8 @@ type CommandExecutionOptions = Partial<ScreenshotRequestFlags> & {
933951
pixels?: number;
934952
doubleTap?: boolean;
935953
verify?: boolean;
954+
settle?: boolean;
955+
settleQuietMs?: number;
936956
clickButton?: ClickButton;
937957
pauseMs?: number;
938958
pattern?: SwipePattern;

‎src/commands/cli-grammar/common.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,17 @@ export function selectorSnapshotOptionsFromFlags(flags: CliFlags): SelectorSnaps
8888
};
8989
}
9090

91+
// Descriptor post-action observation commands use --settle (#1101).
92+
// --timeout doubles as the settle deadline only when --settle is present; a
93+
// bare --timeout stays compatible and is ignored by touch commands.
94+
export function settleInputFromFlags(flags: CliFlags): Record<string, unknown> {
95+
return compactRecord({
96+
settle: flags.settle,
97+
settleQuietMs: flags.settleQuietMs,
98+
timeoutMs: flags.timeoutMs,
99+
});
100+
}
101+
91102
export function repeatedInputFromFlags(flags: CliFlags): Record<string, unknown> {
92103
return compactRecord({
93104
count: flags.count,

0 commit comments

Comments
 (0)