Skip to content

Commit d387a45

Browse files
thymikeeclaude
andauthored
fix(settle): compare post-action trees in one view across every route (#4500)
* fix(settle): never diff a settle against a full-tree snapshot After a full snapshot (no -i), the settle diff compared that tree with the interactive view the settle loop reads, so every wrapper node the view omits read as removed while nothing changed. The baseline now carries whether it is a full view, and such a settle diffs from its own settled capture and says so in its hint. Later settles diff interactive against interactive. Fixes #4486 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): keep loop hints beside the full-view baseline hint Address review: the full-view hint replaced the loop hint (such as the tiny-tree readiness hint) instead of joining it, the wording could read as diffing against the full view, and the interactive-baseline test used a tree that held the full-view wrapper. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): mark only a full-view ref frame, not selector captures Address review: selector resolution captures are full trees and fed the settle baseline through preAction, so marking every non-interactive presentation reset every selector-targeted settle to +0 -0. Only the ref frame the agent read as a full snapshot is marked now; selector and generic-route baselines keep their diff. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(settle): keep resolveSettleHint beside its doc comment Address review: the new hint helper sat between resolveSettleHint and its JSDoc. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): list additions against a full-view ref frame Address review: resetting the baseline to the settled capture made every settle after a full snapshot report +0 -0, even when the press navigated. The diff now runs against the full ref frame and lists only added lines: the interactive view adds no node of its own, so additions are evidence, while removals of nodes that view never reads are not. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(settle): pin the added-line count after a full-view baseline Address review: the every() check passed on an empty line list. The settle comment now states that only Android's interactive view adds no node of its own; an iOS row rewrite can still read as added, as before. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): mark omitted removals with a typed flag Address review: a forced removals count of 0 read as a fact. The settle diff now carries removalsOmitted, the text verdict prints "removals not listed", the hint no longer promises a baseline the ref frame may not advance to, and the docs note the iOS row-rewrite limit. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): settle on the full tree after a full snapshot Address review: listing only additions against a full-tree baseline still reported the keyboard window as added, because the interactive view reads windows the full tree does not. A settle on refs from a full snapshot now captures the full tree too, so its diff compares one view and issues full-tree refs. The removals flag, hint and formatter change are gone. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): mark only a whole full-tree ref frame, without a tail Address review: a scoped, depth-limited or raw frame is a projection the settle capture does not reproduce, so only an unprojected full snapshot is marked fullView. Point taps keep the session baseline unmarked. A full-tree settle carries no tail, because the tail names interactive targets. Comments now describe the full-tree path. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): keep the tail on a full-tree settle An unchanged full-tree settle without a tail handed back no refs, so the next @ref press hit an expired frame. The tail stays, listing addressable refs from the full-tree view the agent chose. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): leave --actions ref frames on the interactive settle view Address review: custom actions are another projection the settle capture does not reproduce, like scope, depth and raw. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(settle): compare post-action trees in one view across every route A settle diff compared whatever pre-action tree it found with an interactive settle capture, so a full snapshot, a selector fallback or a full tree a selector read stored showed up as false removals (#4486). One rule now owns every post-action comparison: two trees are compared only when they were read in the same presentation. capture-kit's postActionBaseline picks the baseline in the agent's view, the presentation of the ref frame it last read, and SurfaceScopedNodes carries that presentation. The settle loop captures in it for press, click, fill, longpress, scroll and back, on refs, selectors and coordinates. --verify captures in the pre-action tree's view, and diff snapshot starts a new baseline when the stored tree was read in another view. Fixes #4486 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(snapshot): preserve view options and lazy baseline imports * fix(snapshot): retain default views and surface transition evidence --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 38b40be commit d387a45

18 files changed

Lines changed: 527 additions & 72 deletions

File tree

‎packages/capture-kit/src/snapshot-baseline.test.ts‎

Lines changed: 54 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { expect, test } from 'vitest';
22
import type { SnapshotState } from '@agent-device/contracts/snapshot-model';
3-
import { surfaceScopedNodes } from './snapshot-baseline.ts';
3+
import type { SurfaceScopedNodes } from '@agent-device/contracts/interaction';
4+
import { postActionBaseline, surfaceScopedNodes } from './snapshot-baseline.ts';
45

56
const nodes = [{ index: 0, type: 'Application', ref: 'e1' }];
67

@@ -33,6 +34,58 @@ test('a session with no stored snapshot still mints an empty baseline', () => {
3334
expect(surfaceScopedNodes(undefined)).toEqual({ nodes: [] });
3435
});
3536

37+
test('the mint carries the presentation the capture read', () => {
38+
expect(surfaceScopedNodes(snapshot({ presentation: { interactiveOnly: true } }))).toEqual({
39+
nodes,
40+
presentation: { interactiveOnly: true },
41+
});
42+
});
43+
44+
const INTERACTIVE = { interactiveOnly: true };
45+
const FULL = {};
46+
const frame = (presentation: SurfaceScopedNodes['presentation']) => tree('frame', presentation);
47+
48+
// #4486: a settle diff compares one view, and the agent's view is its ref frame's presentation.
49+
test.each([
50+
['interactive frame, interactive candidate', INTERACTIVE, INTERACTIVE, 'candidate'],
51+
['interactive frame, full candidate', INTERACTIVE, FULL, 'frame'],
52+
['full frame, interactive candidate', FULL, INTERACTIVE, 'frame'],
53+
['full frame, full candidate', FULL, FULL, 'candidate'],
54+
[
55+
'interactive frame, scoped interactive candidate',
56+
INTERACTIVE,
57+
{ interactiveOnly: true, scope: 'Composer' },
58+
'frame',
59+
],
60+
['interactive frame, candidate with default full view', INTERACTIVE, undefined, 'frame'],
61+
['full frame, candidate with default full view', FULL, undefined, 'candidate'],
62+
] as const)(
63+
'%s: the %s answers',
64+
async (_case, framePresentation, candidatePresentation, expected) => {
65+
const baseline = await postActionBaseline({
66+
frame: frame(framePresentation),
67+
candidates: [tree('candidate', candidatePresentation)],
68+
});
69+
expect(baseline.nodes[0]?.label).toBe(expected);
70+
},
71+
);
72+
73+
test('without a frame the first non-empty candidate answers, in its own view', async () => {
74+
const baseline = await postActionBaseline({
75+
frame: undefined,
76+
candidates: [undefined, { nodes: [] }, tree('candidate', FULL)],
77+
});
78+
expect(baseline).toEqual(tree('candidate', FULL));
79+
expect(await postActionBaseline({ frame: undefined, candidates: [] })).toEqual({ nodes: [] });
80+
});
81+
82+
function tree(label: string, presentation: SurfaceScopedNodes['presentation']): SurfaceScopedNodes {
83+
return {
84+
nodes: [{ index: 0, type: 'Button', label, ref: 'e1' }],
85+
...(presentation ? { presentation } : {}),
86+
};
87+
}
88+
3689
function snapshot(fields: Partial<SnapshotState>): SnapshotState {
3790
return { nodes, createdAt: 0, presentationKey: '', ...fields } as SnapshotState;
3891
}

‎packages/capture-kit/src/snapshot-baseline.ts‎

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,18 @@
11
import type { SnapshotState } from '@agent-device/contracts/snapshot-model';
22
import type { SurfaceScopedNodes } from '@agent-device/contracts/interaction';
3+
import type { SnapshotPresentation } from '@agent-device/kernel/snapshot';
4+
5+
/** Reproduce a populated baseline's view; without one, capture the interactive view. */
6+
export function postActionPresentation(
7+
baseline: SurfaceScopedNodes | undefined,
8+
): SnapshotPresentation {
9+
return baseline?.presentation ?? (baseline?.nodes.length ? {} : { interactiveOnly: true });
10+
}
311

412
/**
513
* Mints the one baseline value a post-action comparison carries: the captured nodes together with the
6-
* surface they describe and the presentation the capture read (#2438, #3565).
14+
* surface they describe, the presentation the capture read (#2438, #3565), and the view it was read
15+
* in (#4486).
716
*
817
* Both settle routes take their baseline from here — the element-targeted interaction route and the
918
* generic daemon route (`scroll`/`back --settle`) — so a fact a capture established cannot be carried
@@ -16,7 +25,10 @@ import type { SurfaceScopedNodes } from '@agent-device/contracts/interaction';
1625
*/
1726
export function surfaceScopedNodes(
1827
snapshot:
19-
| Pick<SnapshotState, 'nodes' | 'iosSystemSurfaceBundleId' | 'presentedSurface'>
28+
| Pick<
29+
SnapshotState,
30+
'nodes' | 'iosSystemSurfaceBundleId' | 'presentedSurface' | 'presentation'
31+
>
2032
| undefined,
2133
): SurfaceScopedNodes {
2234
return {
@@ -25,5 +37,35 @@ export function surfaceScopedNodes(
2537
? { surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
2638
: {}),
2739
...(snapshot?.presentedSurface ? { presentedSurface: snapshot.presentedSurface } : {}),
40+
...(snapshot?.presentation ? { presentation: snapshot.presentation } : {}),
2841
};
2942
}
43+
44+
/**
45+
* The pre-action tree a settle diff runs against, and through its `presentation` the view the settle
46+
* captures in (#4486). Two trees compare only within one presentation, and the agent's view is the
47+
* presentation of the ref frame it last read: `snapshot -i`, a full `snapshot`, a scoped one, or the
48+
* refs a settle issued. So the first candidate read in that view answers. A candidate read in another
49+
* view, such as a selector resolution that fell back to the full tree or a full tree a selector read
50+
* stored, is skipped, and the frame itself answers when no candidate qualifies. Without a frame, the
51+
* first candidate answers.
52+
*
53+
* Every settle route picks its baseline here, so none can compare an interactive read with a full one.
54+
*/
55+
export async function postActionBaseline(params: {
56+
/** The ref frame, as the caller admits it as a baseline. */
57+
frame: SurfaceScopedNodes | undefined;
58+
/** Pre-action trees in the caller's order of preference. */
59+
candidates: readonly (SurfaceScopedNodes | undefined)[];
60+
}): Promise<SurfaceScopedNodes> {
61+
const { isSameSnapshotPresentation } = await import('@agent-device/kernel/snapshot');
62+
const { frame } = params;
63+
const view = frame && frame.nodes.length > 0 ? frame.presentation : undefined;
64+
const candidate = params.candidates.find(
65+
(baseline) =>
66+
baseline !== undefined &&
67+
baseline.nodes.length > 0 &&
68+
(!frame?.nodes.length || isSameSnapshotPresentation(baseline.presentation, view)),
69+
);
70+
return candidate ?? frame ?? { nodes: [] };
71+
}

‎packages/command-runtime/src/commands/capture/runtime/snapshot.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import {
88
type SnapshotDiagnosticsSummary,
99
} from '@agent-device/contracts/capture';
1010
import { AppError } from '@agent-device/kernel/errors';
11+
import { isSameSnapshotPresentation } from '@agent-device/kernel/snapshot';
1112
import { normalizeType } from '@agent-device/contracts/snapshot';
1213
import { snapshotProducerFacts } from '@agent-device/contracts/snapshot-producers';
1314
import type {
@@ -135,7 +136,11 @@ export const diffSnapshotCommand: RuntimeCommand<
135136
}
136137
const capture = await captureRuntimeSnapshot(runtime, options);
137138
const flattenForDiff = options.interactiveOnly === true;
138-
const previousSnapshot = capture.session?.snapshot;
139+
const stored = capture.session?.snapshot;
140+
const previousSnapshot =
141+
stored && isSameSnapshotPresentation(stored.presentation, capture.snapshot.presentation)
142+
? stored
143+
: undefined;
139144
const nextSession = nextSnapshotSession(options.session, capture);
140145

141146
if (!previousSnapshot) {

‎packages/command-runtime/src/commands/interaction/runtime/interaction-snapshot-capture.ts‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,7 @@ import { toBackendContext } from '../../runtime-common.ts';
66
/**
77
* A leaf shared by every interaction runtime module that needs a fresh (or session-cached) tree:
88
* `resolution.ts`'s ref/selector resolution, `selector-target-resolution.ts`'s acting selector lookup,
9-
* `moving-target-gate.ts`'s re-observation, `gestures.ts`'s viewport resolution, and
10-
* `post-action-observation.ts`'s `--verify` baseline.
9+
* `moving-target-gate.ts`'s re-observation, and `gestures.ts`'s viewport resolution.
1110
* Deliberately self-contained — it imports no sibling of this directory, so those siblings can
1211
* import it without ever risking a cycle back.
1312
*/

‎packages/command-runtime/src/commands/interaction/runtime/post-action-observation.ts‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,16 @@ import type {
55
SettleObservation,
66
SettleParams,
77
} from '@agent-device/contracts/interaction';
8-
import { captureInteractionSnapshot } from './interaction-snapshot-capture.ts';
9-
import { summarizePostActionEvidence } from './post-action-surface.ts';
10-
import { surfaceScopedNodes } from '@agent-device/capture-kit/snapshot-baseline';
8+
import { isSameSnapshotPresentation } from '@agent-device/kernel/snapshot';
9+
import {
10+
resolvePostActionSurfaceChange,
11+
summarizePostActionEvidence,
12+
} from './post-action-surface.ts';
13+
import {
14+
postActionPresentation,
15+
surfaceScopedNodes,
16+
} from '@agent-device/capture-kit/snapshot-baseline';
17+
import { captureSelectorSnapshot } from './selector-read-shared.ts';
1118
import { settleAfterInteraction, settleEvidence } from './settle.ts';
1219

1320
type ObservedResult<T extends object> = T & {
@@ -81,9 +88,17 @@ async function observeAfterInteraction(
8188
...params.settle,
8289
resolved,
8390
});
84-
const evidence = params.verify
85-
? settleEvidence(outcome.settledCapture, resolved.preAction)
86-
: undefined;
91+
// A surface transition is evidence in every view; node digests compare only within one view.
92+
const baseline =
93+
outcome.settledCapture &&
94+
(resolvePostActionSurfaceChange(resolved.preAction, outcome.settledCapture) ||
95+
isSameSnapshotPresentation(
96+
resolved.preAction?.presentation,
97+
outcome.settledCapture.presentation,
98+
))
99+
? resolved.preAction
100+
: undefined;
101+
const evidence = params.verify ? settleEvidence(outcome.settledCapture, baseline) : undefined;
87102
return { settle: outcome.observation, ...(evidence ? { evidence } : {}) };
88103
}
89104
if (!params.verify) return {};
@@ -92,8 +107,8 @@ async function observeAfterInteraction(
92107
}
93108

94109
/**
95-
* Post-action side of `--verify` (#1047): one interactive-only capture through
96-
* the same capture helper the resolution path already uses, digested and then
110+
* Post-action side of `--verify` (#1047): one capture in the pre-action tree's presentation
111+
* (interactive-only by default), so the comparison reads one view (#4486), digested and then
97112
* discarded. The node tree itself is never attached to the result, only the
98113
* cheap summary. A missing baseline yields no `changedFromBefore` claim —
99114
* no baseline, no comparison.
@@ -104,7 +119,11 @@ async function captureVerifyEvidence(
104119
resolved: ResolvedInteractionTarget,
105120
): Promise<InteractionEvidence | undefined> {
106121
try {
107-
const capture = await captureInteractionSnapshot(runtime, options, true);
122+
const capture = await captureSelectorSnapshot(runtime, options, {
123+
updateSession: true,
124+
includeRects: true,
125+
presentation: postActionPresentation(resolved.preAction),
126+
});
108127
return summarizePostActionEvidence(surfaceScopedNodes(capture.snapshot), resolved.preAction);
109128
} catch {
110129
return undefined;

‎packages/command-runtime/src/commands/interaction/runtime/post-action-surface.test.ts‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -117,12 +117,13 @@ test('press --settle attaches no diff across a sheet-to-app surface change and d
117117
assert.match(settle.hint ?? '', /different surfaces/);
118118
});
119119

120-
test('press --settle --verify reports one app-to-sheet surface change on both payloads', async () => {
121-
const before = buttonSnapshot();
122-
const sheet = webSignInSheetSnapshot(WEB_SIGN_IN_SHEET_LABELS);
120+
test('press --settle --verify reports a surface change even when the selector and frame views differ', async () => {
121+
const before = { ...buttonSnapshot(), presentation: { interactiveOnly: true } };
122+
const frame = { ...before, presentation: {} };
123+
const sheet = { ...webSignInSheetSnapshot(WEB_SIGN_IN_SHEET_LABELS), presentation: {} };
123124
let captures = 0;
124125
const device = createSettleDevice({
125-
stored: before,
126+
stored: frame,
126127
captureSnapshot: () => {
127128
captures += 1;
128129
return { snapshot: captures === 1 ? before : sheet };

‎packages/command-runtime/src/commands/interaction/runtime/selector-read-shared.ts‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
type SnapshotCommandOptionFields,
1616
type SnapshotNode,
1717
type SnapshotPreferredBackend,
18+
type SnapshotPresentation,
1819
} from '@agent-device/kernel/snapshot';
1920
import type { SnapshotState } from '@agent-device/contracts/snapshot-model';
2021
import {
@@ -72,6 +73,11 @@ export async function captureSelectorSnapshot(
7273
interactiveOnly?: boolean;
7374
includeHiddenContentHints?: boolean;
7475
preferredBackend?: SnapshotPreferredBackend;
76+
/**
77+
* The whole presentation to capture in, overriding `interactiveOnly`, `scope` and the command's
78+
* depth and raw flags: a post-action read reproduces its baseline's view (#4486).
79+
*/
80+
presentation?: SnapshotPresentation;
7581
} = {
7682
updateSession: true,
7783
},
@@ -82,11 +88,22 @@ export async function captureSelectorSnapshot(
8288
}
8389
const sessionName = options.session ?? 'default';
8490
const session = await runtime.sessions.get(sessionName);
91+
const view = captureOptions.presentation;
8592
const result = await captureSnapshot(toBackendContext(runtime, options), {
86-
interactiveOnly: captureOptions.interactiveOnly ?? false,
87-
depth: options.depth,
88-
scope: captureOptions.scope ?? options.scope,
89-
raw: options.raw,
93+
...(view
94+
? {
95+
interactiveOnly: view.interactiveOnly === true,
96+
depth: view.depth,
97+
scope: view.scope,
98+
raw: view.raw,
99+
...(view.customActions === true ? { customActions: true } : {}),
100+
}
101+
: {
102+
interactiveOnly: captureOptions.interactiveOnly ?? false,
103+
depth: options.depth,
104+
scope: captureOptions.scope ?? options.scope,
105+
raw: options.raw,
106+
}),
90107
includeRects: captureOptions.includeRects,
91108
...(captureOptions.preferredBackend
92109
? { preferredBackend: captureOptions.preferredBackend }

0 commit comments

Comments
 (0)