Skip to content

Commit 9a495bd

Browse files
committed
fix(daemon): judge post-gesture movement by identity, not by the intersection
Fixes #1569. The post-gesture baseline check asked "did anything in the intersection of the two captures move". On a scroll that is the wrong question: a scroll REPLACES content rather than sliding shared elements around, so the intersection is whatever sat still. Measured on a live checkout-form `scroll down 0.6` that moved the entire form, the baseline and the post-scroll capture had exactly five identifiers in common and all five were tab-bar icons, which cannot move. What carried the verdict instead was anonymous layout containers matched by ordinal position among other anonymous nodes. That real pair had 8 of them on one side and 10 on the other, and the six "moved" entries reported deltas of -920, +37 and -7 px for a ~500px scroll. Removing them from the same pair flipped the verdict to 'unchanged' — for a scroll that replaced every element on screen. That is the saturation in #1569 finding 1: the oracle was reading capture composition, not the screen. Three changes: - Signature entries carry an `identity` (identifier|label|value|type) alongside the existing `key`. The key keeps folding in hittable/enabled/selected and an occurrence index, which is right for "did two back-to-back captures agree" and wrong across a gesture: scrolling flips `hittable` the moment a node's centre leaves the viewport, evicting exactly the elements whose movement was the evidence. Anonymous nodes get no identity and leave the comparison. - The verdict is set membership. Content leaving AND arriving is replacement, so 'changed'. One-sided difference is scope drift between differently scoped captures, so 'unchanged' — the subset tolerance the old rule needed, kept. Rect movement of a survivor is still 'changed'. - The loop records the baseline's snapshot backend and re-baselines rather than concluding when it changes. Backends do not return comparable views: on one live screen private AX returned 139 nodes including 43 scrolled off-viewport where the tree backend returned 48. A plan fallback or the XCTest-channel penalty can swap backends mid-poll at any time. The distrust budget is untouched — with an oracle that can see movement, a real scroll settles on the first quiet pair instead of running to the cap. Counterfactuals, each failing the pin it belongs to: putting `hittable` back into identity fails the hittability-flip test; admitting anonymous nodes fails the fluctuating-container test; removing the backend guard fails the re-baseline test.
1 parent 3a64ee7 commit 9a495bd

5 files changed

Lines changed: 476 additions & 75 deletions

File tree

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import type { SnapshotNode } from '@agent-device/kernel/snapshot';
4+
import {
5+
buildInteractionSurfaceSignature,
6+
classifyBaselineSurfaceEvidence,
7+
} from '../interaction-outcome-policy.ts';
8+
9+
// #1569: node shapes below are transcribed from a live iPhone 17 Pro capture of
10+
// examples/test-app's checkout form, before and after `scroll down 0.6`. Two
11+
// properties of that real screen are what these tests exist to hold:
12+
//
13+
// - the scroll REPLACED the content: nothing from the form section survives
14+
// into the post capture, and the only identifiers common to both are the
15+
// tab-bar entries, which by construction never move;
16+
// - the captures disagree about which anonymous layout containers exist (the
17+
// live pair had 8 and 10 of them), so any rule that matches those by
18+
// ordinal position is reading noise.
19+
20+
const tabBar = (): SnapshotNode[] =>
21+
[
22+
{ label: 'Home', type: 'Other', rect: { x: 21, y: 791, width: 360, height: 62 } },
23+
{ label: 'Home', type: 'Button', rect: { x: 25, y: 795, width: 74, height: 54 } },
24+
{
25+
identifier: 'house.fill',
26+
label: 'home',
27+
type: 'Image',
28+
rect: { x: 45, y: 801, width: 33, height: 28 },
29+
},
30+
] as SnapshotNode[];
31+
32+
const anonymousContainers = (count: number, baseY: number): SnapshotNode[] =>
33+
Array.from({ length: count }, (_, index) => ({
34+
type: 'Other',
35+
rect: { x: 0, y: baseY + index * 40, width: 402, height: 120 },
36+
})) as SnapshotNode[];
37+
38+
const formSection = (): SnapshotNode[] =>
39+
[
40+
{ label: 'Full name', type: 'StaticText', rect: { x: 34, y: 319, width: 333, height: 17 } },
41+
{
42+
identifier: 'field-name',
43+
label: 'Full name',
44+
type: 'TextField',
45+
rect: { x: 35, y: 344, width: 333, height: 51 },
46+
},
47+
{ label: 'Email', type: 'StaticText', rect: { x: 34, y: 410, width: 333, height: 17 } },
48+
{
49+
identifier: 'field-email',
50+
label: 'Email',
51+
type: 'TextField',
52+
rect: { x: 35, y: 435, width: 333, height: 51 },
53+
},
54+
] as SnapshotNode[];
55+
56+
const deliverySection = (): SnapshotNode[] =>
57+
[
58+
{
59+
identifier: 'shipping-pickup',
60+
label: 'Pickup',
61+
type: 'Button',
62+
rect: { x: 126, y: 137, width: 75, height: 38 },
63+
},
64+
{
65+
identifier: 'checkbox-agree',
66+
label: 'I confirm the order details',
67+
type: 'Other',
68+
rect: { x: 34, y: 543, width: 333, height: 24 },
69+
},
70+
] as SnapshotNode[];
71+
72+
const classify = (before: SnapshotNode[], after: SnapshotNode[]) =>
73+
classifyBaselineSurfaceEvidence(
74+
buildInteractionSurfaceSignature(before),
75+
buildInteractionSurfaceSignature(after),
76+
);
77+
78+
test('a scroll that replaces the content is changed, though every shared element sat still', () => {
79+
// The live shape of #1569. Shared identifiers: only the tab bar, all at dy=0.
80+
// Asking those whether the screen moved is asking the one part of the surface
81+
// guaranteed not to.
82+
assert.equal(
83+
classify(
84+
[...formSection(), ...anonymousContainers(8, 60), ...tabBar()],
85+
[...deliverySection(), ...anonymousContainers(10, 40), ...tabBar()],
86+
),
87+
'changed',
88+
);
89+
});
90+
91+
test('the verdict does not depend on which anonymous containers a capture happened to include', () => {
92+
// The defect: the previous rule matched anonymous nodes by ordinal position,
93+
// so its entire movement evidence came from containers that alias. Strip them
94+
// and it reported 'unchanged' for the very same pair of screens. Every
95+
// composition of the same two screens must agree.
96+
const compositions: [string, SnapshotNode[], SnapshotNode[]][] = [
97+
['no containers', [...formSection(), ...tabBar()], [...deliverySection(), ...tabBar()]],
98+
[
99+
'matched container counts',
100+
[...formSection(), ...anonymousContainers(8, 60), ...tabBar()],
101+
[...deliverySection(), ...anonymousContainers(8, 60), ...tabBar()],
102+
],
103+
[
104+
'mismatched container counts',
105+
[...formSection(), ...anonymousContainers(8, 60), ...tabBar()],
106+
[...deliverySection(), ...anonymousContainers(10, 40), ...tabBar()],
107+
],
108+
];
109+
110+
for (const [label, before, after] of compositions) {
111+
assert.equal(classify(before, after), 'changed', label);
112+
}
113+
});
114+
115+
test('an inert gesture stays unchanged while the anonymous container count fluctuates', () => {
116+
// The live pair carried 8 anonymous containers on one side and 10 on the
117+
// other for the SAME screen. Admitting them to the comparison makes that
118+
// fluctuation look like content both leaving and arriving — a false 'changed'
119+
// on a screen where nothing happened, which would settle a genuinely stale
120+
// capture on the strength of layout noise.
121+
assert.equal(
122+
classify(
123+
[...formSection(), ...anonymousContainers(8, 60), ...tabBar()],
124+
[...formSection(), ...anonymousContainers(10, 40), ...tabBar()],
125+
),
126+
'unchanged',
127+
);
128+
});
129+
130+
test('an inert gesture is unchanged even when volatile hittability flipped', () => {
131+
// `hittable` is viewport-derived, so it flips without the element moving.
132+
// Keying identity on it would drop this element from the comparison entirely.
133+
const before = [
134+
...formSection().map((node) => ({ ...node, hittable: true })),
135+
...tabBar(),
136+
] as SnapshotNode[];
137+
const after = [
138+
...formSection().map((node) => ({ ...node, hittable: false })),
139+
...tabBar(),
140+
] as SnapshotNode[];
141+
142+
assert.equal(classify(before, after), 'unchanged');
143+
});
144+
145+
test('a narrower capture of the same screen is scope drift, not movement', () => {
146+
// Baseline from a broad text search, quiet capture interactive-only: the
147+
// second is a strict subset. Nothing arrived, so nothing was replaced.
148+
assert.equal(classify([...formSection(), ...tabBar()], [...formSection()]), 'unchanged');
149+
assert.equal(classify([...formSection()], [...formSection(), ...tabBar()]), 'unchanged');
150+
});
151+
152+
test('movement of a surviving element is still changed', () => {
153+
const moved = formSection().map((node) => ({
154+
...node,
155+
rect: { ...node.rect!, y: node.rect!.y - 120 },
156+
})) as SnapshotNode[];
157+
158+
assert.equal(classify([...formSection(), ...tabBar()], [...moved, ...tabBar()]), 'changed');
159+
});
160+
161+
test('sharing only the tab bar is ambiguous once the content sets are disjoint on one side', () => {
162+
// Chrome alone carries no evidence: it reads identically whatever happened.
163+
assert.equal(classify(tabBar(), tabBar()), 'unchanged');
164+
assert.equal(classify([...formSection()], [...deliverySection()]), 'ambiguous');
165+
});

‎src/daemon/__tests__/post-gesture-stabilization.test.ts‎

Lines changed: 132 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -158,12 +158,18 @@ test('capturePostGestureStabilizedResult keeps polling past the normal deadline
158158
assert.ok(captureCount > 8, `expected sustained polling, saw ${captureCount} captures`);
159159
});
160160

161-
test('a replaced list under fixed chrome accepts stale but never claims no-effect (#1601 P1)', async () => {
162-
// The reviewer's counterexample: a SUCCESSFUL scroll swapped every list
163-
// cell while the tab-bar chrome (discriminating, shared, unmoved) kept the
164-
// subset-tolerant classifier at 'unchanged'. The loop may still accept the
165-
// stale read — but the agent-facing no-effect claim must be vetoed by the
166-
// unmatched discriminating cells on both sides.
161+
test('a replaced list under fixed chrome now settles outright, and still claims no no-effect (#1601 P1, #1569)', async () => {
162+
// The reviewer's counterexample: a SUCCESSFUL scroll swapped every list cell
163+
// while the tab-bar chrome (discriminating, shared, unmoved) kept the
164+
// classifier at 'unchanged'. #1601 could only veto the agent-facing claim and
165+
// had to let the loop accept the stale read — `staleAccepts` was 1 here.
166+
//
167+
// #1569 removed the premise: the classifier compares identified content by
168+
// set membership, so cells leaving AND arriving is 'changed'. The loop now
169+
// trusts the capture instead of polling to the cap, which is what made every
170+
// checkout-form scroll pay the full distrust budget on live hardware. The
171+
// no-effect claim stays absent — now because there is no accept-stale to
172+
// corroborate at all, which is the stronger reason.
167173
vi.useFakeTimers();
168174
const session = makeSession('ios');
169175
session.snapshot = chromeWithListSnapshot(['row-1', 'row-2']);
@@ -186,6 +192,45 @@ test('a replaced list under fixed chrome accepts stale but never claims no-effec
186192
await vi.advanceTimersByTimeAsync(10_000);
187193
const { result, staleAccepts } = await resultPromise;
188194

195+
assert.equal(staleAccepts, 0);
196+
assert.equal(result.gestureNoEffect, undefined);
197+
});
198+
199+
test('scope drift accepts stale but is vetoed from claiming no-effect (#1601 P1 gate)', async () => {
200+
// #1601's full-surface gate stays load-bearing, and #1569 makes it more so.
201+
// The classifier treats a one-sided difference as scope drift rather than
202+
// movement, so a narrower quiet capture of an unmoved screen still reaches
203+
// accept-stale — and the agent-facing claim must not follow it there, because
204+
// the missing rows are unexamined, not proven absent.
205+
vi.useFakeTimers();
206+
const session = makeSession('ios');
207+
session.snapshot = chromeWithListSnapshot(['row-1', 'row-2']);
208+
markPostGestureStabilization(session, 'scroll');
209+
210+
// Same screen, but the quiet captures see only the chrome — the shape a
211+
// broad baseline followed by an interactive-only capture produces.
212+
const narrowed = makeSnapshotState(
213+
chromeWithListSnapshot(['row-1', 'row-2']).nodes.filter(
214+
(node) => node.type !== 'Cell',
215+
) as never,
216+
);
217+
const capture = vi.fn(async () => narrowed);
218+
219+
const resultPromise = withDiagnosticsScope({}, async () => {
220+
const result = await capturePostGestureStabilizedResult({
221+
session,
222+
capture,
223+
readSnapshot: (snapshot) => snapshot,
224+
});
225+
return {
226+
result,
227+
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
228+
};
229+
});
230+
231+
await vi.advanceTimersByTimeAsync(10_000);
232+
const { result, staleAccepts } = await resultPromise;
233+
189234
assert.equal(staleAccepts, 1);
190235
assert.equal(result.gestureNoEffect, undefined);
191236
});
@@ -431,3 +476,84 @@ test('capturePostGestureStabilizedResult trusts immediately (no cap tax) when th
431476
assert.equal(staleAccepts, 0);
432477
assert.equal(capture.mock.calls.length, 2);
433478
});
479+
480+
// --- #1569: a backend swap mid-poll is not comparable evidence ---
481+
482+
test('capturePostGestureStabilizedResult re-baselines instead of concluding when the backend changes (iOS)', async () => {
483+
// The capture plan can fall back, or the XCTest-channel penalty can pre-empt
484+
// it, at any point during the poll. The backends do not agree on which nodes
485+
// exist — on one live checkout screen private AX returned 139 nodes including
486+
// 43 scrolled off-viewport where the tree backend returned 48 — so a quiet
487+
// capture from a different backend says nothing about the gesture. It must
488+
// become the new baseline, never a verdict.
489+
vi.useFakeTimers();
490+
const session = makeSession('ios');
491+
session.snapshot = makeSnapshotState(pickupSnapshot(500).nodes, {
492+
snapshotQuality: { state: 'healthy', backend: 'tree' },
493+
});
494+
markPostGestureStabilization(session, 'scroll');
495+
496+
// Every capture shows the pre-gesture surface, but on the OTHER backend.
497+
const capture = vi.fn(async () =>
498+
makeSnapshotState(pickupSnapshot(500).nodes, {
499+
snapshotQuality: { state: 'healthy', backend: 'private-ax' },
500+
}),
501+
);
502+
503+
const resultPromise = withDiagnosticsScope({}, async () => {
504+
await capturePostGestureStabilizedResult({
505+
session,
506+
capture,
507+
readSnapshot: (snapshot) => snapshot,
508+
});
509+
return {
510+
rebased: countDiagnosticEventsByPhase(['post_gesture_snapshot_baseline_rebased']),
511+
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
512+
settled: countDiagnosticEventsByPhase(['post_gesture_snapshot_stabilized']),
513+
};
514+
});
515+
516+
await vi.advanceTimersByTimeAsync(6_000);
517+
const { rebased, staleAccepts, settled } = await resultPromise;
518+
519+
// Rebased once onto private-ax, then compared same-backend and settled. A
520+
// cross-backend comparison would have produced a verdict from incomparable
521+
// node sets instead.
522+
assert.equal(rebased, 1);
523+
assert.equal(staleAccepts + settled, 1);
524+
});
525+
526+
test('capturePostGestureStabilizedResult still distrusts a same-backend baseline match (iOS)', async () => {
527+
// The guard above must not become a blanket escape hatch: when the backend is
528+
// stable, an unchanged surface is still the stale-read signal #1542 added.
529+
vi.useFakeTimers();
530+
const session = makeSession('ios');
531+
session.snapshot = makeSnapshotState(pickupSnapshot(500).nodes, {
532+
snapshotQuality: { state: 'healthy', backend: 'tree' },
533+
});
534+
markPostGestureStabilization(session, 'scroll');
535+
536+
const capture = vi.fn(async () =>
537+
makeSnapshotState(pickupSnapshot(500).nodes, {
538+
snapshotQuality: { state: 'healthy', backend: 'tree' },
539+
}),
540+
);
541+
542+
const resultPromise = withDiagnosticsScope({}, async () => {
543+
await capturePostGestureStabilizedResult({
544+
session,
545+
capture,
546+
readSnapshot: (snapshot) => snapshot,
547+
});
548+
return {
549+
rebased: countDiagnosticEventsByPhase(['post_gesture_snapshot_baseline_rebased']),
550+
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
551+
};
552+
});
553+
554+
await vi.advanceTimersByTimeAsync(6_000);
555+
const { rebased, staleAccepts } = await resultPromise;
556+
557+
assert.equal(rebased, 0);
558+
assert.equal(staleAccepts, 1);
559+
});

0 commit comments

Comments
 (0)