You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5c5fa01
Browse filesBrowse the repository at this point in the historyBrowse files
feat: --settle returns the settled diff in the interaction response (#1101) (#1106)
* feat: --settle returns the settled diff in the interaction response (#1101)
press/click/fill/longpress --settle executes the action, waits for the UI
to go quiet (wait stable's loop, shared via stable-capture.ts), and returns
the settled diff vs the pre-action tree in the same response — one round
trip instead of the interact -> observe pair.
- payload: changed lines only (bounded), summary counts, added-line refs,
refsGeneration; best-effort (settled:false + hint on never-quiet content,
never an action failure); --verify shares the settle captures
- ref issuance: the settled tree becomes the session snapshot; a
diff-carrying settle response clears snapshotRefsStale and the MCP layer
merge-only re-pins added-line refs at the settle generation
- grammar: --settle + --settle-quiet <ms> + --timeout <ms> (flag-sourced
descriptor budget with new envelope:'widen' semantics mirroring wait)
- ADR 0011: new settleObservation guarantee classified on every path with
contract scenarios per enforced/delegated cell
* test: give the two contention-flaky doctor scenarios explicit budgets
The doctor provider scenarios sit at ~5s of real daemon-harness work on a
loaded host and flake at vitest's 5s default during full-suite runs (the
known contention flake AGENTS.md documents). Same in-file precedent as the
Metro-probe scenario's 10s budget.
* fix: move SettleParams to contracts to satisfy the layering DAG
daemon/handlers/interaction-flags.ts imported the type across the
daemon -> commands boundary (R2 commands-floor). The tuning params are
part of the interaction contract like SettleObservation, so they live
in contracts/interaction.ts and both layers import from there.
* feat: keep settle diffs content-first — drop Key nodes, added lines win the cap
Bluesky dogfood: a fill that summons the iOS keyboard spent 49 of the 80
capped diff lines spelling out QWERTY keys, and a screen transition with
269 removals could starve out the added lines entirely. Key-type nodes
are now filtered from both diff sides (the [keyboard] container line
still signals presence), and under truncation added lines — the ones
carrying fresh refs — win slots over removals.
* docs: state the core loop in the top-level help starting point
Benchmarked with headless haiku/sonnet agents given only --help: both
models skipped the help-workflow pointer and started with plain
snapshot (38KB payloads they then had to re-read from files). One
core-loop line at the starting point is what teaches snapshot -i and
--settle to models that never read a second help page.
* fix: preserve settle digest refs for mcp
* fix: reduce settle fallow complexity
* fix: surface settle output in CLI text
* fix: complete settle handling for longpress
* refactor: localize daemon timeout envelopes
* refactor: deepen post-action observation
* refactor: centralize post-action observation planning
* refactor: derive settle capability from descriptors
* refactor: trim settle descriptor helpers
Copy file name to clipboardExpand all lines: CONTEXT.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,6 +53,7 @@
53
53
- Coverage manifest: `CONTRACT_COVERAGE` export beside each interaction contract test file claiming which matrix cells it proves; the coverage gate requires every enforced/delegated cell to be claimed and rejects overclaims of waived cells.
54
54
- Delegation-on-error: a fast path falling back to the runtime path on semantic failure shapes. It closes failure-side guarantee cells only — never success-path parity.
55
55
- Ref generation pin: optional `~s<n>` suffix on an @ref carrying the snapshot generation it was minted from. Accepted as input everywhere, emitted by no tree output (snapshot token budget), auto-appended by the MCP layer, stripped and ignored by replay.
56
+
- Settled observation: opt-in (`--settle`) post-action payload on press/click/fill/longpress — the quiet-window stable loop re-captures until the UI settles, and the response carries the diff vs the pre-action tree (changed lines only, added lines with fresh refs, `refsGeneration` when the settled tree was stored). Best-effort: never fails the action; `settled: false` plus a hint on never-quiet content.
56
57
- Snapshot capture plan: per-strategy ordered chain of iOS snapshot capture backends (recursive tree, query sweep, private AX) run by one plan runner under a shared wall-clock budget; recovery ordering is declared data, never a per-call-site branch.
57
58
- Snapshot quality verdict: structured outcome (state, backend, reason code, effective depth, collapsed leaves) computed once by the plan runner and shipped with every planned snapshot payload; the daemon and CLI render it instead of re-deriving degradation from node shapes.
58
59
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect. The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the next command reacquire the app through normal activation.
'Capture cheap post-action evidence (AX digest, node counts, changedFromBefore) instead of a follow-up snapshot',
850
857
},
858
+
{
859
+
key: 'settle',
860
+
names: ['--settle'],
861
+
type: 'boolean',
862
+
usageLabel: '--settle',
863
+
usageDescription:
864
+
'After the action, wait for the UI to go quiet and return the settled diff vs the pre-action tree in the same response (best-effort; never fails the action)',
865
+
},
866
+
{
867
+
key: 'settleQuietMs',
868
+
names: ['--settle-quiet'],
869
+
type: 'int',
870
+
min: 0,
871
+
usageLabel: '--settle-quiet <ms>',
872
+
usageDescription: 'Settle: quiet window the UI must hold to count as settled (default 500ms)',
Copy file name to clipboardExpand all lines: src/cli/parser/cli-help.ts
+14-3Lines changed: 14 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -55,18 +55,25 @@ const AGENT_START_LINES = [
55
55
'Default to agent-device for installs, opens, snapshots, interactions, screenshots, logs, network/perf evidence, and verification.',
56
56
'Use raw adb, simctl, xcrun, or platform scripts only when this help calls out a tool gap or platform setup step.',
57
57
'Start with agent-device help workflow to understand the core loop and how to use the tool.',
58
+
// Benchmarked 2026-07-05 (#1101): agents that only read --help skipped the
59
+
// help-workflow pointer and fell into plain-snapshot loops; stating the loop
60
+
// here is what makes small models pick snapshot -i and --settle unprompted.
61
+
'Core loop: open <app> -> snapshot -i (interactive tree with @refs) -> press/click/fill <target> --settle (returns the settled diff with fresh @refs) -> repeat. Verify with diff snapshot -i.',
58
62
]asconst;
59
63
60
64
constAGENT_QUICKSTART_LINES=[
61
65
'Planning output contract: when asked to plan commands, output command lines only: no prose, numbering, Markdown fences, pipes, or shell helpers.',
62
66
'Default loop: devices/apps -> open -> snapshot -i -> press/fill/get/is/wait/find -> verify with diff snapshot -> close.',
63
67
'Verify a mutation with diff snapshot (or diff snapshot -i), not a full snapshot: it prints only the added/removed/changed lines since the last snapshot in this session, so confirming an action costs a few lines instead of the whole tree.',
64
-
'Use selectors or refs as positional targets: id="submit", label="Allow", or @e12 from snapshot -i.',
68
+
'Collapse act+observe into one call with --settle on press/click/fill/longpress: the response waits for the UI to go quiet and carries the settled diff with fresh refs (settled: false plus a hint on never-quiet content; the action itself never fails). Tune with --settle-quiet <ms> and --timeout <ms>.',
69
+
'Treat the --settle diff as the post-action observation: if it already shows the next target or final evidence, act on that fresh ref or answer from it instead of taking another snapshot.',
70
+
'Network-backed or debounced results may arrive after the --settle quiet window; follow the settled action with wait text "Expected result" or wait <selector> instead of polling full snapshots.',
71
+
'Use selectors or refs as positional targets: id="submit", label="Allow", role=button label="Send", or @e12 from snapshot -i. Do not write role names as selector keys, such as button="Search".',
65
72
'Pin a ref to the snapshot that minted it with ~s<n> (n = refsGeneration in the snapshot response): press @e12~s4. Pinned refs get exact staleness warnings instead of the coarse tree-changed one; plain refs stay valid input.',
'Default snapshot text is an agent-facing, token-efficient view for planning and targeting actions.',
68
75
'Read-only visible/state question: use snapshot/get/is/find; use snapshot -i only when refs are needed.',
69
-
'Anti-pattern: snapshot -i followed by snapshot -i | grep ...; prior refs stay valid until app state changes, and --force-full is the explicit full re-read.',
76
+
'Anti-pattern: snapshot -i followed by snapshot -i | grep ... or hiding command output with 2>/dev/null | jq ...; inspect raw agent-device output first, since hints and errors are part of the planning contract.',
70
77
'Truncated text/input preview: expand first with snapshot -s @e12, not get text.',
71
78
'React Native apps: read help react-native for Metro, DevTools routing, and RN-specific blockers; use react-native dismiss-overlay for LogBox/RedBox overlays.',
72
79
'React Native JS memory leaks: read help cdp; use heap usage samples for a quick signal, then snapshot diff/leak-triplet for retained object proof.',
@@ -142,6 +149,7 @@ Fresh machine or first iOS run:
142
149
Command shape:
143
150
Plans should use agent-device commands, not raw platform tools, pseudo commands, package-manager aliases, or helper prose.
144
151
If the user asks for a command plan, final output should be command lines only: no intro sentence, numbered list, Markdown fence, shell pipe, grep/head/tail helper, or explanatory bullets.
152
+
While exploring, do not pipe or redirect agent-device output through jq/grep/head/tail or 2>/dev/null; raw output carries refs, warnings, hints, and diagnostics needed for the next step.
145
153
Put subcommand first, then positionals, then flags:
146
154
agent-device open com.example.app --session checkout --platform android --relaunch
147
155
agent-device record start ./checkout.mp4 --session checkout
@@ -176,8 +184,9 @@ Snapshots and refs:
176
184
@e14 [cell] label="Profiles" focused -> tvOS focus is currently on this row.
177
185
[off-screen below] 4 items: "Privacy", "About" -> scroll down, then snapshot -i; those are hints, not refs.
178
186
Re-snapshot after navigation, submit, typing/fill, modal/list/reload/dynamic changes when you need new refs.
179
-
Anti-pattern: snapshot -i followed by snapshot -i | grep ...
187
+
Anti-pattern: snapshot -i followed by snapshot -i | grep ..., or adding 2>/dev/null | jq ... before reading the raw command output.
180
188
Refs from the first snapshot remain valid until you press, click, fill, type, scroll, go back, wait for async UI, or otherwise change app state.
189
+
A --settle response is the observation for that mutation. If its diff already shows the next target or final evidence, use that fresh ref or answer from the diff instead of taking a follow-up snapshot.
181
190
After a mutation, prefer a known selector/label directly (for example press 'label="Send"') because interaction commands refresh interactive state internally. If you need to discover the new control, use snapshot -i, or snapshot -i -s "Composer" when a stable container label/id can scope the refresh.
182
191
If typing/fill opened the keyboard or changed layout and the next target has no stable selector, run snapshot -i, use the fresh ref, then verify with wait/find or diff snapshot -i.
183
192
For a targeted query, use find/get/is. If you truly need the full tree again, pass --force-full.
@@ -188,6 +197,7 @@ Snapshots and refs:
188
197
189
198
Selectors:
190
199
Use selectors as positional targets: id="field-email" or label="Allow".
200
+
Selector terms are key=value filters such as id="submit", label="Search", text="Search", or role=button label="Search". Do not write role names as keys, such as button="Search"; use role=button label="Search" or the @ref from the latest snapshot/settle diff.
191
201
Do not use CSS selectors, pseudo refs, --selector, --text, or raw x/y when refs/selectors exist.
192
202
agent-device fill 'id="catalog-search"' "tart" --delay-ms 80
193
203
agent-device press 'id="submit-order"'
@@ -224,6 +234,7 @@ Read-only and waits:
224
234
agent-device wait 'label="Ready"' 3000
225
235
agent-device wait stable
226
236
agent-device wait stable 500 10000
237
+
For network-backed search/typeahead, --settle confirms the local UI quieted after fill/press; use wait text "Expected result" or wait <result selector> for server-loaded content that can arrive later.
227
238
agent-device find "Increment" press --json
228
239
For async/list text presence, prefer wait text over is visible when no interaction is needed.
229
240
wait stable [quietMs] [timeoutMs] (defaults 500/10000) replaces guessed sleeps between actions and snapshots in the core loop: it polls the interactive-only tree and resolves once two or more consecutive captures are unchanged for quietMs, or fails with the standard wait-timeout shape plus the last capture stats (captures, nodeCount) on timeout; use it after relaunch/navigation instead of a fixed wait <ms> before the next snapshot or action.
0 commit comments