Skip to content

Commit 3a64ee7

Browse files
authored
fix(daemon): surface proven no-effect gestures to the agent (#1601)
* fix(daemon): surface proven no-effect gestures to the agent (#1600) The #1542 post-gesture stabilization loop already PROVES when a scroll, swipe, or pan moved nothing: its accept-stale verdict fires only when the quiet post-gesture capture still equals the pre-gesture baseline after the distrust cap. But the verdict went to the diagnostics stream only — the agent-facing response reported plain success, and the benchmark showed the cost: element-18 burned ~40 tool calls re-issuing scrolls the daemon knew did nothing ("Scrolled up by 1200px" fifteen times over a byte-identical tree) before stumbling onto raw swipe. The stabilization loop now returns the verdict alongside the capture, and the snapshot handler appends a warning through the existing annotations channel: it names the exact gesture, admits the at-edge ambiguity the platform cannot resolve, and hands over the raw-drag escape hatch that moved the stuck list in the field. Live-verified on the seeded Bluesky feed: scroll up at top warns, scroll down with real room moves silently, a fling at the bottom edge warns — all three truthful. * fix: require full-surface evidence before the no-effect claim (#1601 review P1) accept-stale alone is subset-tolerant by design: a successful scroll that replaced every list cell under fixed chrome still classifies 'unchanged' on the shared chrome alone, and #1573 has live evidence of that shape. The agent-facing claim now additionally requires every discriminating entry of the quiet capture to match the baseline exactly, in both directions (haveIdenticalDiscriminatingSurfaces): any appeared or vanished real element — including scope drift — vetoes the warning. Silence is the safe failure mode for a message that steers the agent's next move. Red evidence: the new fixed-chrome + replaced-list regression test fails on the previous PR commit (gestureNoEffect wrongly present), passes with the gate. * refactor: extract the corroborated-accept result builder capturePostGestureStabilizedResult crossed the complexity threshold (14 cyclomatic) after the #1601 review gate; the accept/veto construction now lives in buildAcceptedStabilizedResult, which also keeps the corroboration rule stated in one place.
1 parent 8ba5f9b commit 3a64ee7

6 files changed

Lines changed: 227 additions & 11 deletions

File tree

‎src/daemon/__tests__/post-gesture-stabilization-fixtures.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,45 @@ export function pickupSnapshotWithExtraText(y = 500) {
6262
};
6363
}
6464

65+
/**
66+
* Fixed tab-bar chrome + a list whose visible cells sit at stable row rects.
67+
* Two captures with DIFFERENT cell ids model a successful scroll that
68+
* replaced every list row while the chrome (a discriminating, shared,
69+
* unmoved entry) stayed put: `classifyBaselineSurfaceEvidence` reads
70+
* 'unchanged' from the shared chrome alone (#1601 review P1) — the exact
71+
* accept-stale false negative that must never surface as an agent-facing
72+
* no-effect claim.
73+
*/
74+
export function chromeWithListSnapshot(cellIds: [string, string]) {
75+
return makeSnapshotState([
76+
{ index: 0, type: 'Application', label: 'App', rect: { x: 0, y: 0, width: 390, height: 844 } },
77+
{
78+
index: 1,
79+
parentIndex: 0,
80+
type: 'Button',
81+
identifier: 'tab-home',
82+
label: 'Home',
83+
rect: { x: 0, y: 800, width: 195, height: 44 },
84+
},
85+
{
86+
index: 2,
87+
parentIndex: 0,
88+
type: 'Cell',
89+
identifier: cellIds[0],
90+
label: cellIds[0],
91+
rect: { x: 0, y: 100, width: 390, height: 60 },
92+
},
93+
{
94+
index: 3,
95+
parentIndex: 0,
96+
type: 'Cell',
97+
identifier: cellIds[1],
98+
label: cellIds[1],
99+
rect: { x: 0, y: 160, width: 390, height: 60 },
100+
},
101+
]);
102+
}
103+
65104
export function applicationRootNode() {
66105
return {
67106
ref: 'e-root',

‎src/daemon/__tests__/post-gesture-stabilization.test.ts‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,11 @@ import { countDiagnosticEventsByPhase, withDiagnosticsScope } from '../../utils/
55
import { buildInteractionSurfaceSignature } from '../interaction-outcome-policy.ts';
66
import {
77
capturePostGestureStabilizedResult,
8+
formatGestureNoEffectWarning,
89
markPostGestureStabilization,
910
} from '../post-gesture-stabilization.ts';
1011
import {
12+
chromeWithListSnapshot,
1113
deliverySnapshot,
1214
keyboardWindowNodes,
1315
makeSession,
@@ -143,16 +145,64 @@ test('capturePostGestureStabilizedResult keeps polling past the normal deadline
143145
});
144146

145147
await vi.advanceTimersByTimeAsync(10_000);
146-
const { staleAccepts, settled } = await resultPromise;
148+
const { result, staleAccepts, settled } = await resultPromise;
147149

148150
assert.equal(staleAccepts, 1);
149151
assert.equal(settled, 0);
150152
assert.equal(session.postGestureStabilization, undefined);
153+
// #1600: a stale-accept is the daemon PROVING the gesture moved nothing —
154+
// that verdict must reach the caller, not only the diagnostics stream.
155+
assert.equal(result.gestureNoEffect?.action, 'scroll');
151156
// Proves it kept polling well past the OLD 1.5s accept point (2 attempts,
152157
// ~200ms) instead of trusting the first quiet match.
153158
assert.ok(captureCount > 8, `expected sustained polling, saw ${captureCount} captures`);
154159
});
155160

161+
test('a replaced list under fixed chrome accepts stale but never claims no-effect (#1601 P1)', async () => {
162+
// The reviewer's counterexample: a SUCCESSFUL scroll swapped every list
163+
// cell while the tab-bar chrome (discriminating, shared, unmoved) kept the
164+
// subset-tolerant classifier at 'unchanged'. The loop may still accept the
165+
// stale read — but the agent-facing no-effect claim must be vetoed by the
166+
// unmatched discriminating cells on both sides.
167+
vi.useFakeTimers();
168+
const session = makeSession('ios');
169+
session.snapshot = chromeWithListSnapshot(['row-1', 'row-2']);
170+
markPostGestureStabilization(session, 'scroll');
171+
172+
const capture = vi.fn(async () => chromeWithListSnapshot(['row-3', 'row-4']));
173+
174+
const resultPromise = withDiagnosticsScope({}, async () => {
175+
const result = await capturePostGestureStabilizedResult({
176+
session,
177+
capture,
178+
readSnapshot: (snapshot) => snapshot,
179+
});
180+
return {
181+
result,
182+
staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']),
183+
};
184+
});
185+
186+
await vi.advanceTimersByTimeAsync(10_000);
187+
const { result, staleAccepts } = await resultPromise;
188+
189+
assert.equal(staleAccepts, 1);
190+
assert.equal(result.gestureNoEffect, undefined);
191+
});
192+
193+
test('formatGestureNoEffectWarning names the gesture and the raw-drag escape hatch', () => {
194+
const scrollWarning = formatGestureNoEffectWarning('scroll', ['down', '1']);
195+
assert.match(scrollWarning, /scroll down produced no visible change/);
196+
assert.match(scrollWarning, /swipe x1 y1 x2 y2/);
197+
assert.match(scrollWarning, /already at its edge/);
198+
199+
const gestureWarning = formatGestureNoEffectWarning('gesture', ['swipe', 'left']);
200+
assert.match(gestureWarning, /gesture swipe left produced no visible change/);
201+
202+
const bareWarning = formatGestureNoEffectWarning('swipe', []);
203+
assert.match(bareWarning, /swipe produced no visible change/);
204+
});
205+
156206
test('capturePostGestureStabilizedResult trusts a quiet signature once content genuinely differs from the baseline (iOS)', async () => {
157207
vi.useFakeTimers();
158208
const session = makeSession('ios');
@@ -175,10 +225,12 @@ test('capturePostGestureStabilizedResult trusts a quiet signature once content g
175225
});
176226

177227
await vi.advanceTimersByTimeAsync(1_000);
178-
const { staleAccepts, settled } = await resultPromise;
228+
const { result, staleAccepts, settled } = await resultPromise;
179229

180230
assert.equal(settled, 1);
181231
assert.equal(staleAccepts, 0);
232+
// A genuine settle carries no no-effect claim.
233+
assert.equal(result.gestureNoEffect, undefined);
182234
// Accepted at the first quiet match (initial capture + one poll = 2
183235
// attempts): no distrust cost for a genuine settle.
184236
assert.equal(capture.mock.calls.length, 2);

‎src/daemon/handlers/snapshot-capture.ts‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,10 @@ import {
4545
getActivePendingInteractionOutcome,
4646
retryPendingInteractionOutcome,
4747
} from '../interaction-outcome-policy.ts';
48-
import { capturePostGestureStabilizedResult } from '../post-gesture-stabilization.ts';
48+
import {
49+
capturePostGestureStabilizedResult,
50+
formatGestureNoEffectWarning,
51+
} from '../post-gesture-stabilization.ts';
4952
import { presentIosInteractiveSnapshot } from '../snapshot-presentation/ios/index.ts';
5053
import type { SessionState } from '../types.ts';
5154
import { errorResponse, type DaemonFailureResponse } from './response.ts';
@@ -153,12 +156,13 @@ async function captureInteractionOutcomeAwareSnapshot(
153156
}
154157

155158
clearPendingInteractionOutcome(session);
156-
latest = await capturePostGestureStabilizedResult({
159+
const stabilized = await capturePostGestureStabilizedResult({
157160
session,
158161
initial: latest,
159162
capture: async () => await capturePostActionSnapshotAttempt(params),
160163
readSnapshot: (attempt) => attempt.snapshot,
161164
});
165+
latest = stabilized.value;
162166
if (outcome.change !== 'ambiguous' && latest.annotations.freshness?.staleAfterRetries !== true) {
163167
clearAndroidSnapshotFreshness(session);
164168
}
@@ -175,7 +179,7 @@ async function captureInteractionOutcomeAwareSnapshot(
175179

176180
return {
177181
snapshot: latest.snapshot,
178-
...latest.annotations,
182+
...withGestureNoEffectWarning(latest.annotations, stabilized.gestureNoEffect),
179183
};
180184
}
181185

@@ -288,14 +292,35 @@ async function captureAndroidFreshnessAwareAttempt(
288292
async function capturePostGestureAwareSnapshot(
289293
params: CaptureSnapshotParams & { session: SessionState },
290294
): Promise<CaptureSnapshotResult> {
291-
const latest = await capturePostGestureStabilizedResult({
295+
const stabilized = await capturePostGestureStabilizedResult({
292296
session: params.session,
293297
capture: async () => await capturePostActionSnapshotAttempt(params),
294298
readSnapshot: (attempt) => attempt.snapshot,
295299
});
300+
const latest = stabilized.value;
296301
return {
297302
snapshot: latest.snapshot,
298-
...latest.annotations,
303+
...withGestureNoEffectWarning(latest.annotations, stabilized.gestureNoEffect),
304+
};
305+
}
306+
307+
/**
308+
* #1600: a proven no-effect gesture must reach the agent inside the very
309+
* response it reads next, not only the diagnostics stream. Warnings ride the
310+
* existing annotations channel so every renderer that already prints capture
311+
* warnings picks this up with no new plumbing.
312+
*/
313+
function withGestureNoEffectWarning(
314+
annotations: SnapshotCaptureAnnotations,
315+
gestureNoEffect: { action: string; positionals: string[] } | undefined,
316+
): SnapshotCaptureAnnotations {
317+
if (!gestureNoEffect) return annotations;
318+
return {
319+
...annotations,
320+
warnings: [
321+
...(annotations.warnings ?? []),
322+
formatGestureNoEffectWarning(gestureNoEffect.action, gestureNoEffect.positionals),
323+
],
299324
};
300325
}
301326

‎src/daemon/interaction-outcome-policy.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,44 @@ export function classifyBaselineSurfaceEvidence(
251251
return discriminatingOverlap > 0 ? 'unchanged' : 'ambiguous';
252252
}
253253

254+
/**
255+
* Full-surface agreement over DISCRIMINATING entries, in BOTH directions —
256+
* the stronger bar an agent-facing no-effect claim needs (#1601 review P1).
257+
*
258+
* `classifyBaselineSurfaceEvidence` is deliberately subset-tolerant for the
259+
* distrust loop, where a false 'unchanged' only buys extra polling. But a
260+
* fixed-chrome screen whose list cells were fully replaced by a SUCCESSFUL
261+
* scroll classifies 'unchanged' on the shared chrome alone — new cells are
262+
* absent from the baseline and silently ignored. Requiring the discriminating
263+
* entry sets to match exactly (same keys both ways, every rect within
264+
* tolerance) vetoes that shape: any appeared or vanished real element kills
265+
* the claim. Scope drift between baseline and capture vetoes too — silence
266+
* is the safe failure mode for a message that steers the agent's next move.
267+
*/
268+
export function haveIdenticalDiscriminatingSurfaces(
269+
left: InteractionSurfaceSignature,
270+
right: InteractionSurfaceSignature,
271+
): boolean {
272+
const leftEntries = left.filter((entry) => entry.discriminating);
273+
const rightEntries = right.filter((entry) => entry.discriminating);
274+
if (leftEntries.length === 0 || leftEntries.length !== rightEntries.length) return false;
275+
// Keys carry an occurrence ordinal (`|#N`), so a map by key is lossless.
276+
const rightByKey = new Map(rightEntries.map((entry) => [entry.key, entry]));
277+
for (const entry of leftEntries) {
278+
const other = rightByKey.get(entry.key);
279+
if (!other) return false;
280+
if (
281+
Math.abs(entry.x - other.x) > RECT_TOLERANCE_PX ||
282+
Math.abs(entry.y - other.y) > RECT_TOLERANCE_PX ||
283+
Math.abs(entry.width - other.width) > RECT_TOLERANCE_PX ||
284+
Math.abs(entry.height - other.height) > RECT_TOLERANCE_PX
285+
) {
286+
return false;
287+
}
288+
}
289+
return true;
290+
}
291+
254292
function supportsInteractionOutcomePolicy(session: SessionState): boolean {
255293
return isMobilePlatform(session.device);
256294
}

‎src/daemon/post-gesture-stabilization.ts‎

Lines changed: 63 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import {
77
areInteractionSurfaceSignaturesStable,
88
buildInteractionSurfaceSignature,
99
classifyBaselineSurfaceEvidence,
10+
haveIdenticalDiscriminatingSurfaces,
1011
type InteractionSurfaceSignature,
1112
} from './interaction-outcome-policy.ts';
1213
import type { SessionState } from './types.ts';
@@ -41,6 +42,7 @@ export function markPostGestureStabilization(
4142
if (!isPostGestureStabilizingAction(action, positionals, flags)) return;
4243
session.postGestureStabilization = {
4344
action,
45+
positionals,
4446
markedAt: Date.now(),
4547
// No extra capture: `session.snapshot` is still whatever was captured
4648
// before this gesture dispatched (this call happens post-dispatch,
@@ -150,16 +152,31 @@ function emitPostGestureSettleDiagnostic(
150152
});
151153
}
152154

155+
export type PostGestureStabilizedResult<T> = {
156+
value: T;
157+
/**
158+
* Present ONLY when the accept-stale verdict is corroborated by full-surface
159+
* evidence: every discriminating entry of the quiet capture matches the
160+
* pre-gesture baseline exactly, in both directions
161+
* (`haveIdenticalDiscriminatingSurfaces`). The bare verdict is NOT enough —
162+
* it is subset-tolerant by design, and a successful scroll that replaced
163+
* every list cell under fixed chrome still reads accept-stale (#1601 review
164+
* P1). Callers surface this to the agent: a diagnostics-only signal let one
165+
* benchmark run burn 40 calls re-issuing scrolls that moved nothing (#1600).
166+
*/
167+
gestureNoEffect?: { action: string; positionals: string[] };
168+
};
169+
153170
export async function capturePostGestureStabilizedResult<T>(params: {
154171
session: SessionState | undefined;
155172
capture: () => Promise<T>;
156173
readSnapshot: (result: T) => SnapshotState;
157174
initial?: T;
158-
}): Promise<T> {
175+
}): Promise<PostGestureStabilizedResult<T>> {
159176
const { session, capture, readSnapshot } = params;
160177
const pending = session?.postGestureStabilization;
161178
if (!session || !supportsPostGestureStabilization(session.device) || !pending) {
162-
return params.initial ?? (await capture());
179+
return { value: params.initial ?? (await capture()) };
163180
}
164181

165182
const needsBaselineDistrust = requiresPostGestureBaselineDistrust(session.device);
@@ -190,7 +207,7 @@ export async function capturePostGestureStabilizedResult<T>(params: {
190207
}
191208
clearPostGestureStabilization(session);
192209
emitPostGestureSettleDiagnostic(verdict, pending.action, attempts, elapsedMs);
193-
return current.value;
210+
return buildAcceptedStabilizedResult(verdict, pending, current);
194211
}
195212
previous = current;
196213
}
@@ -205,7 +222,49 @@ export async function capturePostGestureStabilizedResult<T>(params: {
205222
durationMs: Date.now() - startedAt,
206223
},
207224
});
208-
return previous.value;
225+
return { value: previous.value };
226+
}
227+
228+
/**
229+
* The agent-facing wording for a proven no-effect gesture. Names the exact
230+
* gesture, admits the honest ambiguity (at-edge is a legitimate no-op the
231+
* platform cannot distinguish), and hands over the one escape hatch that
232+
* moved a stuck list when synthesized scrolls did not (#1600, element-18:
233+
* raw `swipe` worked where scroll/fling/pan all silently no-opped).
234+
*/
235+
export function formatGestureNoEffectWarning(action: string, positionals: string[]): string {
236+
const gesture = [action, ...positionals.filter((value) => !/^[\d.-]+$/.test(value))]
237+
.join(' ')
238+
.trim();
239+
return (
240+
`${gesture} produced no visible change: the tree still matches its pre-gesture state. ` +
241+
'Either the container is already at its edge, or it ignores synthesized scrolls — ' +
242+
'a raw drag moves such lists: swipe x1 y1 x2 y2 (start inside the list).'
243+
);
244+
}
245+
246+
/**
247+
* The no-effect claim needs BOTH the accept-stale verdict AND full-surface
248+
* corroboration (`haveIdenticalDiscriminatingSurfaces`): the verdict alone is
249+
* subset-tolerant, and a successful scroll that replaced every list cell
250+
* under fixed chrome still reads accept-stale (#1601 review P1).
251+
*/
252+
function buildAcceptedStabilizedResult<T>(
253+
verdict: 'trust' | 'accept-stale',
254+
pending: NonNullable<SessionState['postGestureStabilization']>,
255+
current: CapturedSurface<T>,
256+
): PostGestureStabilizedResult<T> {
257+
const corroborated =
258+
verdict === 'accept-stale' &&
259+
haveIdenticalDiscriminatingSurfaces(pending.baselineSignature ?? [], current.signature);
260+
if (!corroborated) return { value: current.value };
261+
return {
262+
value: current.value,
263+
gestureNoEffect: {
264+
action: pending.action,
265+
positionals: pending.positionals ?? [],
266+
},
267+
};
209268
}
210269

211270
function isPostGestureStabilizingAction(

‎src/daemon/types.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,9 @@ export type AndroidSnapshotFreshness = {
198198

199199
export type PostGestureStabilization = {
200200
action: string;
201+
/** The gesture's own positionals (e.g. scroll direction) — wording input for
202+
* the #1600 no-effect warning; never re-dispatched. */
203+
positionals?: string[];
201204
markedAt: number;
202205
/**
203206
* Pre-gesture interaction-surface signature, captured from the session's

0 commit comments

Comments
 (0)