Skip to content

Commit 3b6a9f7

Browse files
thymikeeclaude
andcommitted
fix(settle): settle on the full tree after a full snapshot
Address review: listing only additions against a full-tree baseline still reported the keyboard window as added, because the interactive view reads windows the full tree does not. A settle on refs from a full snapshot now captures the full tree too, so its diff compares one view and issues full-tree refs. The removals flag, hint and formatter change are gone. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 5443831 commit 3b6a9f7

7 files changed

Lines changed: 91 additions & 121 deletions

File tree

‎packages/command-output/src/settle-output.test.ts‎

Lines changed: 0 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -49,30 +49,3 @@ test('post-action observation CLI output formatters follow descriptor traits', a
4949
);
5050
}
5151
});
52-
53-
test('a settle that omitted removals does not print a removal count', async () => {
54-
const format = listCliOutputFormatters()[PUBLIC_COMMANDS.press]!;
55-
const settle = (removalsOmitted: boolean) => ({
56-
settled: true,
57-
waitedMs: 12,
58-
diff: {
59-
summary: { additions: 0, removals: 0, unchanged: 6 },
60-
lines: [],
61-
...(removalsOmitted ? { removalsOmitted: true } : {}),
62-
},
63-
});
64-
const text = async (removalsOmitted: boolean) =>
65-
String(
66-
(
67-
await format({
68-
env: process.env,
69-
stderrIsTTY: false,
70-
input: {},
71-
result: { message: 'ok', settle: settle(removalsOmitted) },
72-
})
73-
).text,
74-
);
75-
76-
assert.match(await text(true), /settled after 12ms: \+0 removals not listed \(~6 unchanged\)/);
77-
assert.match(await text(false), /settled after 12ms: \+0 -0 \(~6 unchanged\)/);
78-
});

‎packages/command-output/src/settle-output.ts‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@ type SettleTextView = {
1919
summary?: { additions?: number; removals?: number; unchanged?: number };
2020
lines?: Array<{ kind?: string; text?: string; ref?: string }>;
2121
truncated?: boolean;
22-
removalsOmitted?: boolean;
2322
};
2423
tail?: Array<{ ref?: string; role?: string; label?: string }>;
2524
tailTruncated?: boolean;
@@ -141,7 +140,5 @@ function formatSettleVerdict(view: SettleTextView): string {
141140
const verdict = view.settled === true ? 'settled' : 'not settled';
142141
const summary = view.diff?.summary;
143142
if (!summary) return `${verdict} after ${view.waitedMs ?? 0}ms`;
144-
const removals =
145-
view.diff?.removalsOmitted === true ? 'removals not listed' : `-${summary.removals ?? 0}`;
146-
return `${verdict} after ${view.waitedMs ?? 0}ms: +${summary.additions ?? 0} ${removals} (~${summary.unchanged ?? 0} unchanged)`;
143+
return `${verdict} after ${view.waitedMs ?? 0}ms: +${summary.additions ?? 0} -${summary.removals ?? 0} (~${summary.unchanged ?? 0} unchanged)`;
147144
}

‎packages/command-runtime/src/commands/interaction/runtime/settle-view-baseline.test.ts‎

Lines changed: 69 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ import { createMemorySessionStore } from '../../../command-runtime/runtime-facto
1010
import { ref, selector } from './selector-read-utils.ts';
1111
import { createFakeClock } from './__tests__/settle-device-fixtures.ts';
1212

13-
// #4486: a full `snapshot` keeps wrapper nodes that the interactive view the settle loop reads
14-
// omits. A press that changes nothing must not report those wrappers as removed.
13+
// #4486: a full `snapshot` keeps wrapper nodes the interactive view omits. A settle on its refs reads
14+
// the full tree too, so an unchanged screen reports no change and wrappers are never false removals.
1515
//
1616
// Budgets are injected (fake clock) — no real waiting.
1717

@@ -44,16 +44,29 @@ function interactiveTree(labels: readonly string[]): SnapshotState {
4444
return { ...makeSnapshotState(dialogNodes(labels)), presentation: { interactiveOnly: true } };
4545
}
4646

47-
function createDevice(params: {
48-
refFrame: SnapshotState;
49-
capture: (call: number) => SnapshotState;
50-
}) {
51-
let calls = 0;
52-
return createAgentDevice({
47+
type Screen = { full: SnapshotState; interactive: SnapshotState };
48+
49+
const DIALOG: Screen = { full: fullDialogTree(), interactive: interactiveTree(DIALOG_LABELS) };
50+
51+
/**
52+
* A backend that answers each capture in the view it was asked for. `screens` is the screen before
53+
* the press and after it; `views` records the view of every capture taken after the press.
54+
*/
55+
function createDevice(params: { refFrame: SnapshotState; before: Screen; after: Screen }) {
56+
let pressed = false;
57+
const views: Array<'full' | 'interactive'> = [];
58+
const device = createAgentDevice({
5359
backend: {
5460
platform: 'android',
55-
captureSnapshot: async () => ({ snapshot: params.capture((calls += 1)) }),
56-
tap: async () => ({ ok: true }),
61+
captureSnapshot: async (_context, options) => {
62+
const screen = pressed ? params.after : params.before;
63+
if (pressed) views.push(options?.interactiveOnly === true ? 'interactive' : 'full');
64+
return { snapshot: options?.interactiveOnly === true ? screen.interactive : screen.full };
65+
},
66+
tap: async () => {
67+
pressed = true;
68+
return { ok: true };
69+
},
5770
} satisfies AgentDeviceBackend,
5871
artifacts: createLocalArtifactAdapter(),
5972
sessions: createMemorySessionStore([
@@ -62,68 +75,77 @@ function createDevice(params: {
6275
policy: localCommandPolicy(),
6376
clock: createFakeClock(),
6477
});
78+
return { device, views };
79+
}
80+
81+
function removedLabels(lines: Array<{ kind: string; text: string }> | undefined): string[] {
82+
return (lines ?? []).filter((line) => line.kind === 'removed').map((line) => line.text);
6583
}
6684

67-
test('a ref press after a full snapshot reports no removals the view change caused', async () => {
68-
const device = createDevice({
69-
refFrame: fullDialogTree(),
70-
capture: () => interactiveTree(DIALOG_LABELS),
85+
test('a ref press after a full snapshot settles on the full tree and reports no false removals', async () => {
86+
const { device, views } = createDevice({
87+
refFrame: DIALOG.full,
88+
before: DIALOG,
89+
after: DIALOG,
7190
});
7291

7392
const result = await device.interactions.press(ref('@e3'), { session: 'default', settle: {} });
7493

75-
assert.equal(result.settle?.diff?.summary.removals, 0);
76-
assert.equal(result.settle?.diff?.removalsOmitted, true);
77-
// The tiny-tree readiness hint the settled screen earns is kept beside the view-change hint.
78-
assert.match(result.settle?.hint ?? '', /previous snapshot was a full tree.*nearly-empty tree/);
94+
assert.ok(views.length > 0 && views.every((view) => view === 'full'));
95+
assert.deepEqual(result.settle?.diff?.summary, { additions: 0, removals: 0, unchanged: 4 });
7996
});
8097

81-
test('a ref press after a full snapshot still lists what the press added', async () => {
82-
const device = createDevice({
83-
refFrame: fullDialogTree(),
84-
// The press opens another screen: its controls are new, the dialog is gone.
85-
capture: () => interactiveTree(['Next', 'Back', 'Home', 'Menu', 'Help']),
98+
test('a ref press after a full snapshot still reports what changed on the full tree', async () => {
99+
const next = ['Next', 'Back'];
100+
const { device } = createDevice({
101+
refFrame: DIALOG.full,
102+
before: DIALOG,
103+
after: {
104+
full: {
105+
...makeSnapshotState([
106+
{ index: 0, depth: 0, type: 'Group', rect: { x: 0, y: 0, width: 400, height: 800 } },
107+
...dialogNodes(next, 1),
108+
]),
109+
presentation: {},
110+
},
111+
interactive: interactiveTree(next),
112+
},
86113
});
87114

88115
const result = await device.interactions.press(ref('@e3'), { session: 'default', settle: {} });
89116

90-
assert.equal(result.settle?.diff?.summary.additions, 5);
91-
assert.equal(result.settle?.diff?.summary.removals, 0);
92-
assert.equal(result.settle?.diff?.lines.length, 5);
93-
assert.ok(result.settle?.diff?.lines.every((line) => line.kind === 'added' && line.ref));
117+
assert.equal(result.settle?.diff?.summary.additions, 2);
118+
assert.equal(result.settle?.diff?.summary.removals, 3);
119+
assert.ok(removedLabels(result.settle?.diff?.lines).some((text) => text.includes('Device name')));
94120
});
95121

96-
test('a ref press after an interactive snapshot still reports what left the screen', async () => {
97-
const device = createDevice({
98-
refFrame: interactiveTree(DIALOG_LABELS),
99-
capture: () => interactiveTree(['Cancel', 'OK']),
122+
test('a ref press after an interactive snapshot settles on the interactive view', async () => {
123+
const titleGone = interactiveTree(['Cancel', 'OK']);
124+
const { device, views } = createDevice({
125+
refFrame: DIALOG.interactive,
126+
before: DIALOG,
127+
after: { full: DIALOG.full, interactive: titleGone },
100128
});
101129

102130
const result = await device.interactions.press(ref('@e1'), { session: 'default', settle: {} });
103131

104-
assert.equal(result.settle?.diff?.summary.removals, 1);
105-
assert.doesNotMatch(result.settle?.hint ?? '', /full tree/);
132+
assert.ok(views.length > 0 && views.every((view) => view === 'interactive'));
133+
assert.deepEqual(removedLabels(result.settle?.diff?.lines).length, 1);
106134
});
107135

108-
test('a selector press keeps its diff although its resolution read a full tree', async () => {
109-
const device = createDevice({
110-
refFrame: interactiveTree(DIALOG_LABELS),
111-
// Capture 1 resolves the selector on a full tree; the settle loop then reads the interactive
112-
// view, where the dialog title is gone after the press.
113-
capture: (call) => (call === 1 ? fullDialogTree() : interactiveTree(['Cancel', 'OK'])),
136+
test('a selector press settles on the interactive view although its resolution read a full tree', async () => {
137+
const titleGone = interactiveTree(['Cancel', 'OK']);
138+
const { device, views } = createDevice({
139+
refFrame: DIALOG.interactive,
140+
before: DIALOG,
141+
after: { full: DIALOG.full, interactive: titleGone },
114142
});
115143

116144
const result = await device.interactions.press(selector('label=OK'), {
117145
session: 'default',
118146
settle: {},
119147
});
120148

121-
// The genuine removal is listed; this route keeps comparing removals (#4486 leaves it open).
122-
assert.ok(
123-
result.settle?.diff?.lines.some(
124-
(line) => line.kind === 'removed' && line.text.includes('Device name'),
125-
),
126-
);
127-
assert.equal(result.settle?.diff?.removalsOmitted, undefined);
128-
assert.doesNotMatch(result.settle?.hint ?? '', /full tree/);
149+
assert.ok(views.length > 0 && views.every((view) => view === 'interactive'));
150+
assert.ok(removedLabels(result.settle?.diff?.lines).some((text) => text.includes('Device name')));
129151
});

‎packages/command-runtime/src/commands/interaction/runtime/settle.ts‎

Lines changed: 10 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,6 @@ const MAX_SETTLE_TAIL_ENTRIES = 20;
8080
export const NEVER_SETTLED_HINT =
8181
'The UI kept changing for the whole settle budget (animation, carousel, or ticker?), so no settled diff is shown. Raise --timeout, wait for specific content, or take a fresh snapshot.';
8282

83-
const FULL_VIEW_BASELINE_HINT =
84-
'The previous snapshot was a full tree and settle reads the interactive view, so this diff lists only added lines; removals are not listed. A settle on refs from this response compares interactive views.';
85-
8683
const SETTLE_CAPTURE_STALLED_HINT =
8784
'A snapshot capture stalled past the settle budget, so no settled diff is shown. The action itself succeeded; observe with wait stable or snapshot.';
8885

@@ -210,16 +207,13 @@ async function readSettledOutcome(
210207
// advisory state.
211208
...(diffShown
212209
? buildSettleDiffAndTail(
213-
params.baseline,
210+
params.baseline.nodes,
214211
settledNodes,
215212
params.actionPoint,
216213
session?.appBundleId,
217214
)
218215
: {}),
219-
...withFullViewBaselineHint(
220-
resolveSettleHint(outcome, stored, settledNodes.length, surfaceChange),
221-
diffShown && params.baseline.fullView === true,
222-
),
216+
...resolveSettleHint(outcome, stored, settledNodes.length, surfaceChange),
223217
},
224218
settledCapture,
225219
};
@@ -268,8 +262,8 @@ function authorizedRefBaseline(
268262

269263
/**
270264
* The ref frame the agent last read. A frame read as a full tree (`snapshot` without `-i`) is marked
271-
* `fullView`, because the settle loop reads the interactive view. Selector captures are full trees
272-
* too, but no agent chose that view, so only the ref frame carries the mark.
265+
* `fullView`, so the settle loop reads the full tree too and the diff compares one view. Selector
266+
* captures are full trees as well, but no agent chose that view, so only the ref frame carries it.
273267
*/
274268
function refFrameBaseline(snapshot: SnapshotState | undefined): SurfaceScopedNodes {
275269
const baseline = surfaceScopedNodes(snapshot);
@@ -298,7 +292,7 @@ function nonEmptyBaseline(
298292
}
299293

300294
function buildSettleDiff(
301-
baseline: SurfaceScopedNodes,
295+
baselineNodes: SnapshotNode[],
302296
settledNodes: SnapshotNode[],
303297
appBundleId: string | undefined,
304298
): NonNullable<SettleObservation['diff']> {
@@ -308,36 +302,30 @@ function buildSettleDiff(
308302
// snapshot), extra baseline-only lines surface as removals — advisory noise,
309303
// the same baseline caveat --verify's changedFromBefore already accepts.
310304
const diff = buildSnapshotDiff(
311-
withoutSettleChrome(baseline.nodes, appBundleId),
305+
withoutSettleChrome(baselineNodes, appBundleId),
312306
withoutSettleChrome(settledNodes, appBundleId),
313307
{ flatten: true, withRefs: true },
314308
);
315-
// A full-tree baseline holds nodes the interactive settle view never reads, so its removals are
316-
// not evidence and are left out. Added lines stay: on Android the interactive view adds no node
317-
// of its own, while an iOS row rewrite can also read as added, as it did before #4486.
318-
const changed = diff.lines.filter(
319-
(line) => line.kind === 'added' || (line.kind === 'removed' && baseline.fullView !== true),
320-
);
309+
const changed = diff.lines.filter((line) => line.kind !== 'unchanged');
321310
const lines = capSettleDiffLines(changed).map((line) => ({
322311
kind: line.kind as 'added' | 'removed',
323312
text: line.text,
324313
...(line.ref ? { ref: line.ref } : {}),
325314
}));
326315
return {
327-
summary: baseline.fullView === true ? { ...diff.summary, removals: 0 } : diff.summary,
316+
summary: diff.summary,
328317
lines,
329-
...(baseline.fullView === true ? { removalsOmitted: true as const } : {}),
330318
...(changed.length > lines.length ? { truncated: true as const } : {}),
331319
};
332320
}
333321

334322
function buildSettleDiffAndTail(
335-
baseline: SurfaceScopedNodes,
323+
baselineNodes: SnapshotNode[],
336324
settledNodes: SnapshotNode[],
337325
actionPoint: Point | undefined,
338326
appBundleId: string | undefined,
339327
): Pick<SettleObservation, 'diff' | 'tail' | 'tailTruncated'> {
340-
const diff = buildSettleDiff(baseline, settledNodes, appBundleId);
328+
const diff = buildSettleDiff(baselineNodes, settledNodes, appBundleId);
341329
return { diff, ...buildSettleTail(diff, settledNodes, actionPoint, appBundleId) };
342330
}
343331

@@ -466,16 +454,6 @@ function capSettleDiffLines<T extends { kind: string }>(changed: T[]): T[] {
466454
return kept;
467455
}
468456

469-
function withFullViewBaselineHint(
470-
resolved: { hint?: string },
471-
fullViewBaselineReset: boolean,
472-
): { hint?: string } {
473-
if (!fullViewBaselineReset) return resolved;
474-
return {
475-
hint: resolved.hint ? `${FULL_VIEW_BASELINE_HINT} ${resolved.hint}` : FULL_VIEW_BASELINE_HINT,
476-
};
477-
}
478-
479457
/**
480458
* The settled observation's hint. A surface change (#2438) is stated alongside whatever the loop
481459
* itself reports rather than in place of it: the transition explains the missing diff, and the

‎packages/command-runtime/src/commands/interaction/runtime/stable-capture.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,10 @@ export async function runStableCaptureLoop(
103103
quietMs?: number;
104104
timeoutMs: number;
105105
resetBudgetOnPrivateAxRecovery?: boolean;
106-
/** Immutable pre-action projection; the stored session may advance before settle begins. */
106+
/**
107+
* Immutable pre-action projection; the stored session may advance before settle begins. The loop
108+
* reads the view the baseline was read in, so a full-view baseline is diffed full against full.
109+
*/
107110
broadTransitionBaseline?: SurfaceScopedNodes;
108111
},
109112
): Promise<StableCaptureLoopResult> {
@@ -154,6 +157,7 @@ export async function runStableCaptureLoop(
154157
runtime,
155158
options,
156159
deadlineMs - now(runtime),
160+
params.broadTransitionBaseline?.fullView !== true,
157161
preferredBackend,
158162
);
159163
if (read.kind !== 'captured') return unfinished(read);
@@ -543,6 +547,7 @@ async function captureStableSignalWithinDeadline(
543547
runtime: AgentDeviceRuntime,
544548
options: CommandContext & SelectorSnapshotOptions,
545549
remainingMs: number,
550+
interactiveOnly: boolean,
546551
preferredBackend?: SnapshotPreferredBackend,
547552
): Promise<StableCaptureRead> {
548553
try {
@@ -552,7 +557,7 @@ async function captureStableSignalWithinDeadline(
552557
{ ...options, signal },
553558
{
554559
updateSession: false,
555-
interactiveOnly: true,
560+
interactiveOnly,
556561
preferredBackend,
557562
},
558563
);

‎packages/contracts/src/interaction.ts‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -161,9 +161,9 @@ export type SurfaceScopedNodes = {
161161
*/
162162
presentedSurface?: PresentedSurface;
163163
/**
164-
* The nodes are a ref frame the agent read as a full tree (`snapshot` without `-i`). A settle reads
165-
* the interactive view, so a diff against these nodes would report every node that view omits as
166-
* removed. Only the settle baseline sets it.
164+
* The nodes are a ref frame the agent read as a full tree (`snapshot` without `-i`). The settle
165+
* loop then reads the full tree too, because diffing these nodes against the interactive view
166+
* reports every node that view omits as removed. Only the settle baseline sets it.
167167
*/
168168
fullView?: true;
169169
};
@@ -350,11 +350,6 @@ export type SettleObservation = {
350350
lines: SettleDiffLine[];
351351
/** Present (true) when lines were capped to the response bound. */
352352
truncated?: boolean;
353-
/**
354-
* Present (true) when the baseline was a ref frame read as a full tree: removals were not
355-
* compared, so `summary.removals` is 0 and `lines` holds only additions.
356-
*/
357-
removalsOmitted?: true;
358353
};
359354
/**
360355
* Unchanged interactive refs tail: benchmarks (July 2026) showed 27% of

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -494,7 +494,7 @@ agent-device gesture tap 80 300 200 450 --count 20 --pause-ms 100 # repeated two
494494
495495
If rotation stops repeated iOS presses, inspect the screen before continuing with the remaining presses.
496496
497-
A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait <selector>`. The settled diff compares interactive views. After a full `snapshot` (no `-i`), a `--settle` action on one of its `@ref`s has no interactive baseline. That settle lists only added lines, sets `diff.removalsOmitted`, and says so in its hint. A settle on refs from that response compares interactive views. On iOS, a row the interactive view rewrites can still be listed as added. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command).
497+
A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait <selector>`. After a full `snapshot` (no `-i`), a `--settle` action on one of its `@ref`s reads the full tree too, so its diff compares one view and its refs are full-tree refs. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command).
498498
499499
A `click` or `press` whose target the daemon resolves to a point (`@ref` or selector) re-checks that target's position before dispatching when the session's previous action animated the screen, and dispatches where it came to rest. When that check cannot confirm a stable position — the target was still moving, or a re-check could not measure it — the command refuses with `target_still_moving` and `dispatched: no`: nothing was dispatched, the point the command resolved is in `error.details.rect`, and where the target was last seen is in `error.details.movedToRect` when that last look measured one. Run `wait <selector>` on the target and retry the same command, or add `--settle` so the action waits for the UI to go quiet. A refused `@ref` press needs a fresh `snapshot -i` and a selector, because the wait retires the ref. Two dispatch paths hand off before the check: a press at raw coordinates has no resolved target to re-check, and a dispatch the backend performs on the element itself — `click @ref` on a backend with element dispatch, or a Maestro non-hittable coordinate fallback — goes to that backend rather than through a resolved point. The re-check stops when the request is canceled or reaches its deadline, and nothing is dispatched (`dispatched: no`): a canceled request fails with `details.reason: request_canceled`, and a request out of time with `details.reason: daemon_request_timeout`.
500500

0 commit comments

Comments
 (0)