Skip to content

Commit 95daea0

Browse files
thymikeethymikee
committed
fix(interaction): carry one post-gesture outcome on the tree and disclose it on every route
A selector click, press, or fill right after a scroll whose surface never settled missed on that moving tree and answered a plain selector_not_found, and find lost the fact when a sparse capture was recovered by a re-capture. A proven no-effect gesture reached only snapshot, through annotations that selector reads and interactions drop. Both facts are one field, SnapshotState.postGestureOutcome ({ kind: 'unsettled' | 'no-effect', gesture }), stamped by post-gesture stabilization. A re-capture taken at once to recover or widen that tree inherits it. Every route discloses it through withCaptureDisclosures; interactions pass their request capture proof as the consumed tree. A failure's disclosure sentence now lands in error.hint, the hint request finalization keeps. withGestureNoEffectWarning is deleted. Closes #2853 Closes #2856
1 parent 7286206 commit 95daea0

36 files changed

Lines changed: 490 additions & 261 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,15 @@
2020
- Fixed (mobile): a read taken right after a `scroll`, `swipe`, or `gesture swipe` no longer reports
2121
a definite miss when the surface never settled. When post-gesture stabilization ran out of budget
2222
on a surface still moving, `is visible` answered a plain `selector_not_found` and `is absent`
23-
passed. That capture now carries `unsettledGesture`: `is`, `get`, `find`, and `wait` report it (in
24-
`error.details` or `data`) with an appended warning, `snapshot` appends the warning, `is absent`
25-
refuses with `observation: "unsettled"`, `wait absent` keeps polling, and the next read captures
26-
afresh. Click, press, and fill by selector do not disclose it yet. A failed read now also carries
27-
`targetActivation` in `error.details`, the same place as `unsettledGesture`.
23+
passed. The capture now carries `postGestureOutcome` (`{ kind, gesture: { action, positionals } }`)
24+
with `kind: "unsettled"`, and so does a re-capture taken at once to recover or widen it. A proven
25+
no-effect gesture rides the same field with `kind: "no-effect"`; before, its warning reached only
26+
`snapshot`. `is`, `get`, `find`, `wait`, and every interaction that captured it (`click`, `press`,
27+
`fill`, and the other touch and gesture commands) report the field in `data` or `error.details`
28+
with an appended warning; `snapshot` appends the warning. `is absent` refuses an unsettled capture
29+
with `observation: "unsettled"`, `wait absent` keeps polling, and the next read captures afresh.
30+
A failed read also carries `targetActivation` in `error.details`, and a failed interaction now
31+
keeps the disclosure sentences in its hint.
2832
- Fixed (ios): `open` on a local Simulator now waits for the launched app's discovery before it
2933
decides whether the app is observable. On a loaded host `simctl spawn launchctl list` outlasts one
3034
1.5 s discovery wait slice, and the launch observation read that slice as an unobservable app, so

‎packages/capture-kit/src/post-gesture-stability.ts‎

Lines changed: 26 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { emitDiagnostic } from '@agent-device/host-kit/diagnostics';
22
import { sleep } from '@agent-device/host-kit/retry';
3-
import type { PostGestureAction } from '@agent-device/kernel/snapshot';
3+
import type { PostGestureAction, PostGestureOutcome } from '@agent-device/kernel/snapshot';
44

55
/**
66
* Pure post-gesture stability mechanics: the quiet-window polling loop and the
@@ -59,41 +59,37 @@ export type PostGestureStabilityHooks<T, S extends readonly unknown[]> = {
5959

6060
export type PostGestureStabilityOutcome<T> = {
6161
value: T;
62-
/** Present when the deadline expired while the last two captures still disagreed. */
63-
unsettledGesture?: PostGestureAction;
6462
/**
65-
* Present ONLY when the accept-stale verdict is corroborated by full-surface
66-
* evidence (`surfacesIdentical`). The bare verdict is NOT enough — it is
67-
* subset-tolerant by design, and a successful scroll that replaced every
68-
* list cell under fixed chrome still reads accept-stale (#1601 review P1).
69-
* Callers surface this to the agent: a diagnostics-only signal let one
70-
* benchmark run burn 40 calls re-issuing scrolls that moved nothing (#1600).
63+
* `unsettled` when the deadline expired while the last two captures still disagreed.
64+
* `no-effect` ONLY when the accept-stale verdict is corroborated by full-surface evidence
65+
* (`surfacesIdentical`): the bare verdict is subset-tolerant by design, and a successful scroll
66+
* that replaced every list cell under fixed chrome still reads accept-stale (#1601 review P1).
67+
* Callers surface it to the agent: a diagnostics-only signal let one benchmark run burn 40 calls
68+
* re-issuing scrolls that moved nothing (#1600).
7169
*/
72-
gestureNoEffect?: PostGestureAction;
70+
postGestureOutcome?: PostGestureOutcome;
7371
};
7472

75-
function describePostGestureAction(gesture: PostGestureAction): string {
76-
return [gesture.action, ...gesture.positionals].join(' ').trim();
77-
}
78-
7973
/**
80-
* The agent-facing wording for a proven no-effect gesture. Names the exact
81-
* gesture, admits the honest ambiguity (at-edge is a legitimate no-op the
82-
* platform cannot distinguish), and hands over the one escape hatch that
83-
* moved a stuck list when synthesized scrolls did not (#1600, element-18:
84-
* raw `swipe` worked where scroll/fling/pan all silently no-opped).
74+
* The agent-facing sentence for a post-gesture outcome, true whether the read that carries it found
75+
* its target or not. A no-effect gesture admits the honest ambiguity (at-edge is a legitimate no-op
76+
* the platform cannot distinguish) and hands over the escape hatch that moved a stuck list when
77+
* synthesized scrolls did not (#1600: raw `swipe` worked where scroll/fling/pan all no-opped).
8578
*/
86-
export function formatGestureNoEffectWarning(action: string, positionals: string[]): string {
87-
return (
88-
`${describePostGestureAction({ action, positionals })} produced no visible change: the tree still matches its pre-gesture state. ` +
89-
'Either the container is already at its edge, or it ignores synthesized scrolls — ' +
90-
'a raw drag moves such lists: swipe x1 y1 x2 y2 (start inside the list).'
91-
);
79+
export function formatPostGestureOutcomeWarning({ kind, gesture }: PostGestureOutcome): string {
80+
const named = [gesture.action, ...gesture.positionals].join(' ').trim();
81+
return kind === 'unsettled'
82+
? `The surface was still changing after ${named} when this tree was read, so it may not match where the surface comes to rest: an element missing from it is not proof of absence.`
83+
: `${named} produced no visible change: the tree still matches its pre-gesture state. ` +
84+
'Either the container is already at its edge, or it ignores synthesized scrolls — ' +
85+
'a raw drag moves such lists: swipe x1 y1 x2 y2 (start inside the list).';
9286
}
9387

94-
/** A miss on a tree read while a gesture's surface was still changing is not proof of absence. */
95-
export function formatGestureUnsettledWarning(gesture: PostGestureAction): string {
96-
return `The surface was still changing after ${describePostGestureAction(gesture)} when this was read, so an element missing from it may still be on screen. Read again before treating it as absent.`;
88+
function postGestureOutcome(
89+
kind: PostGestureOutcome['kind'],
90+
pending: PostGestureAction,
91+
): PostGestureOutcome {
92+
return { kind, gesture: { action: pending.action, positionals: pending.positionals } };
9793
}
9894

9995
/**
@@ -199,10 +195,7 @@ export async function runPostGestureStabilityLoop<T, S extends readonly unknown[
199195
},
200196
});
201197
if (lastPairAgreed) return { value: previous.value };
202-
return {
203-
value: previous.value,
204-
unsettledGesture: { action: pending.action, positionals: pending.positionals },
205-
};
198+
return { value: previous.value, postGestureOutcome: postGestureOutcome('unsettled', pending) };
206199
}
207200

208201
type CapturedSurface<T, S> = {
@@ -266,13 +259,7 @@ function buildAcceptedOutcome<T, S extends readonly unknown[]>(
266259
baselineSignature !== undefined &&
267260
hooks.surfacesIdentical(baselineSignature, current.signature)
268261
) {
269-
return {
270-
value: current.value,
271-
gestureNoEffect: {
272-
action: pending.action,
273-
positionals: pending.positionals,
274-
},
275-
};
262+
return { value: current.value, postGestureOutcome: postGestureOutcome('no-effect', pending) };
276263
}
277264
emitDiagnostic({
278265
level: 'info',

‎packages/kernel/src/snapshot.ts‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -502,13 +502,34 @@ export type SnapshotState = {
502502
* the foreground instead (#2682). Consumers that surface this tree disclose the repair.
503503
*/
504504
targetActivation?: IosTargetActivation;
505-
/** The gesture whose surface was still changing when stabilization gave up on this capture. */
506-
unsettledGesture?: PostGestureAction;
505+
/** What post-gesture stabilization proved about the gesture before this capture. */
506+
postGestureOutcome?: PostGestureOutcome;
507507
} & SnapshotStateProvenance;
508508

509509
/** The gesture a post-gesture outcome fact names: the command and its positionals. */
510510
export type PostGestureAction = { action: string; positionals: string[] };
511511

512+
/**
513+
* `unsettled`: the surface was still changing when the stabilization deadline expired.
514+
* `no-effect`: the settled surface still matches the pre-gesture tree (#1600).
515+
*/
516+
export type PostGestureOutcome = {
517+
kind: 'unsettled' | 'no-effect';
518+
gesture: PostGestureAction;
519+
};
520+
521+
/**
522+
* A capture taken at once to recover or widen `previous` reads the same moment after the same
523+
* gesture, so it carries that capture's outcome.
524+
*/
525+
export function inheritPostGestureOutcome<T extends SnapshotState>(
526+
previous: SnapshotState,
527+
recapture: T,
528+
): T {
529+
recapture.postGestureOutcome ??= previous.postGestureOutcome;
530+
return recapture;
531+
}
532+
512533
export type SnapshotUnchanged = {
513534
ageMs: number;
514535
nodeCount: number;

‎packages/selectors/src/absence-observation.ts‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ export function absenceCaptureOptionMessage(
7272
export function classifyAbsenceObservation(
7373
snapshot: Pick<
7474
SnapshotState,
75-
'backend' | 'nodes' | 'snapshotQuality' | 'truncated' | 'unsettledGesture'
75+
'backend' | 'nodes' | 'snapshotQuality' | 'truncated' | 'postGestureOutcome'
7676
>,
7777
matches: readonly SnapshotNode[],
7878
): AbsenceObservation {
@@ -100,7 +100,10 @@ export function classifyAbsenceObservation(
100100
};
101101
}
102102
if (matchCount === 0) {
103-
return { kind: snapshot.unsettledGesture ? 'unsettled' : 'absent', matches: 0 };
103+
return {
104+
kind: snapshot.postGestureOutcome?.kind === 'unsettled' ? 'unsettled' : 'absent',
105+
matches: 0,
106+
};
104107
}
105108
return { kind: 'present', matches: matchCount, firstMatch: firstMatch! };
106109
}

‎src/commands/capture/runtime/snapshot.test.ts‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@ import {
1313
type CommandSessionStore,
1414
} from '../../../runtime.ts';
1515
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
16-
import { formatGestureUnsettledWarning } from '@agent-device/capture-kit/post-gesture-stability';
16+
import type { PostGestureOutcome } from '@agent-device/kernel/snapshot';
17+
import { formatPostGestureOutcomeWarning } from '@agent-device/capture-kit/post-gesture-stability';
1718

1819
test('runtime snapshot captures nodes and updates the session baseline', async () => {
1920
let stored: Parameters<CommandSessionStore['set']>[0] | undefined;
@@ -789,17 +790,20 @@ test('runtime snapshot leaves the keyboard band off when the backend measured no
789790
});
790791

791792
test('runtime snapshot warns when its tree was read on a surface still moving after a gesture', async () => {
792-
const gesture = { action: 'scroll', positionals: ['down'] };
793+
const outcome: PostGestureOutcome = {
794+
kind: 'unsettled',
795+
gesture: { action: 'scroll', positionals: ['down'] },
796+
};
793797
const device = createSnapshotOnlyDevice({
794798
snapshot: {
795799
...makeSnapshotState([{ index: 0, depth: 0, type: 'Window', label: 'Home' }], {
796800
backend: 'xctest',
797801
}),
798-
unsettledGesture: gesture,
802+
postGestureOutcome: outcome,
799803
},
800804
});
801805

802806
const result = await device.capture.snapshot({ session: 'default' });
803807

804-
assert.deepEqual(result.warnings, [formatGestureUnsettledWarning(gesture)]);
808+
assert.deepEqual(result.warnings, [formatPostGestureOutcomeWarning(outcome)]);
805809
});

‎src/commands/capture/runtime/snapshot.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import {
2626
renderSnapshotQualityWarnings,
2727
truncatedCaptureWarning,
2828
} from '@agent-device/capture-kit/quality-warnings';
29-
import { formatGestureUnsettledWarning } from '@agent-device/capture-kit/post-gesture-stability';
29+
import { formatPostGestureOutcomeWarning } from '@agent-device/capture-kit/post-gesture-stability';
3030
import { buildSnapshotVisibility } from '@agent-device/capture-kit/snapshot-visibility';
3131
import { ANDROID_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/android-system-surface-disclosure';
3232
import { formatReactNativeOverlayWarning } from '../../react-native/overlay.ts';
@@ -261,8 +261,8 @@ function buildSnapshotWarnings(params: {
261261
);
262262
}
263263
warnings.push(...truncatedCaptureWarning(snapshotTruncationForResult(params.snapshot)));
264-
if (params.snapshot.unsettledGesture) {
265-
warnings.push(formatGestureUnsettledWarning(params.snapshot.unsettledGesture));
264+
if (params.snapshot.postGestureOutcome) {
265+
warnings.push(formatPostGestureOutcomeWarning(params.snapshot.postGestureOutcome));
266266
}
267267
warnings.push(...buildEmptyAndroidInteractiveWarnings(params));
268268
if (!params.annotations.quality) {

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,11 @@ import type {
55
SnapshotNode,
66
SnapshotState,
77
} from '@agent-device/kernel/snapshot';
8-
import { findNodeByRef, normalizeRef } from '@agent-device/kernel/snapshot';
8+
import {
9+
findNodeByRef,
10+
inheritPostGestureOutcome,
11+
normalizeRef,
12+
} from '@agent-device/kernel/snapshot';
913
import { resolveRectCenter } from '@agent-device/kernel/rect-center';
1014
import type {
1115
AgentDeviceRuntime,
@@ -390,7 +394,9 @@ async function resolveSelectorInteractionTarget(
390394
params.pipeline,
391395
);
392396
if ((!resolved || !resolved.node.rect) && params.requireInteractive) {
397+
const interactive = capture.snapshot;
393398
capture = await captureInteractionSnapshot(runtime, options, false);
399+
inheritPostGestureOutcome(interactive, capture.snapshot);
394400
resolved = resolveActionSelector(
395401
capture.snapshot.nodes,
396402
selectorExpression,

‎src/commands/interaction/runtime/wait-absent.test.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,10 @@ test('wait absent does not take a miss on a surface still moving after a gesture
145145
const unsettled = {
146146
snapshot: {
147147
...makeSnapshotState([]),
148-
unsettledGesture: { action: 'scroll', positionals: ['down'] },
148+
postGestureOutcome: {
149+
kind: 'unsettled' as const,
150+
gesture: { action: 'scroll', positionals: ['down'] },
151+
},
149152
},
150153
};
151154
const device = absentDevice([unsettled, snapshot('Removed')]);

‎src/daemon/__tests__/capture-disclosure-target-activation.test.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ test('a failure response appends the repair to its hint and keeps the original d
5454
assert.equal(response.error.message, 'selector missed');
5555
assert.equal(response.error.code, 'COMMAND_FAILED');
5656
assert.equal(response.error.details?.blockedBy, 'android_foreground_surface');
57-
assert.match(String(response.error.details?.hint), /prior state runningBackground/);
57+
assert.match(String(response.error.hint), /prior state runningBackground/);
5858
});
5959

6060
test('a capture with no repair leaves the response byte-identical', () => {
@@ -64,7 +64,7 @@ test('a capture with no repair leaves the response byte-identical', () => {
6464
});
6565

6666
/**
67-
* A failure carries the sentence in `error.details.hint`. That carrier is exactly where a route could
67+
* A failure carries the sentence in `error.hint`. That carrier is exactly where a route could
6868
* cheaply borrow the previous command's repair off the stored snapshot and blame it on this request,
6969
* so the repair travels only on the request's own proof (#2682).
7070
*/
@@ -77,7 +77,7 @@ test('a failure does not borrow a repair the stored snapshot happens to carry',
7777
const response = withCaptureDisclosures({
7878
response: failed,
7979
consumedTree: { targetActivation: FACT },
80-
activationProof: {},
80+
captureProof: {},
8181
});
8282

8383
assert.equal(response, failed);
@@ -95,7 +95,7 @@ test('surface and foreground disclosures ride one response together', () => {
9595
iosSystemSurfaceBundleId: 'com.apple.SafariViewService',
9696
targetActivation: FACT,
9797
},
98-
activationProof: { state: { targetActivation: FACT } },
98+
captureProof: { targetActivation: FACT },
9999
});
100100
const data = dataOf(response);
101101
assert.match(String(data.warning), /system web sign-in sheet/);
@@ -117,22 +117,22 @@ test('a repair that passes through two wrappers is named once in the failure hin
117117
details: { hint: 'Use snapshot to see the current tree.' },
118118
},
119119
};
120-
const proof = { state: { targetActivation: FACT } };
120+
const proof = { targetActivation: FACT };
121121

122122
const once = withCaptureDisclosures({
123123
response: missed,
124124
consumedTree: { targetActivation: FACT },
125-
activationProof: proof,
125+
captureProof: proof,
126126
});
127127
const twice = withCaptureDisclosures({
128128
response: once,
129129
consumedTree: { targetActivation: FACT },
130-
activationProof: proof,
130+
captureProof: proof,
131131
});
132132

133133
assert.equal(twice.ok, false);
134134
if (twice.ok) return;
135-
const hint = String(twice.error.details?.hint);
135+
const hint = String(twice.error.hint);
136136
assert.equal(
137137
hint.split(iosTargetActivationDisclosure(FACT)).length - 1,
138138
1,

‎src/daemon/__tests__/capture-disclosure.test.ts‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ test('wait timeout for app text hidden behind a system surface discloses the occ
148148
expect(response.ok).toBe(false);
149149
if (response.ok) return;
150150
expect(response.error.message).toMatch(/wait timed out for text: Bakery list/);
151-
expect(String(response.error.details?.hint)).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
151+
expect(String(response.error.hint)).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
152152
});
153153

154154
test('sessionless read-only find still discloses the occluding system surface', async () => {
@@ -204,7 +204,7 @@ test('disclosure appends after an existing failure hint instead of replacing it'
204204
);
205205
expect(response.ok).toBe(false);
206206
if (response.ok) return;
207-
const hint = String(response.error.details?.hint);
207+
const hint = String(response.error.hint);
208208
expect(hint).toContain('prior hint text');
209209
expect(hint).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
210210
expect(hint.indexOf('prior hint text')).toBeLessThan(
@@ -261,7 +261,7 @@ test('sessionless wait timeout still discloses the occluding system surface', as
261261
expect(response.ok).toBe(false);
262262
if (response.ok) return;
263263
expect(sessionStore.get('default')).toBeUndefined();
264-
expect(String(response.error.details?.hint)).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
264+
expect(String(response.error.hint)).toContain(ANDROID_SYSTEM_SURFACE_DISCLOSURE);
265265
});
266266

267267
// --- #2438: an in-place iOS system surface (web sign-in sheet) discloses on the same shared seam ---
@@ -350,7 +350,7 @@ test('mutating find that misses on an in-place system surface still discloses it
350350

351351
expect(response?.ok).toBe(false);
352352
if (response?.ok) return;
353-
expect(String(response?.error.details?.hint)).toContain(WEB_SIGN_IN_DISCLOSURE);
353+
expect(String(response?.error.hint)).toContain(WEB_SIGN_IN_DISCLOSURE);
354354
});
355355

356356
test('the shared disclosure helper reports an iOS system surface on both outcomes', () => {
@@ -368,13 +368,13 @@ test('the shared disclosure helper reports an iOS system surface on both outcome
368368
);
369369
expect(failed.ok).toBe(false);
370370
if (failed.ok) return;
371-
expect(String(failed.error.details?.hint)).toContain(WEB_SIGN_IN_DISCLOSURE);
371+
expect(String(failed.error.hint)).toContain(WEB_SIGN_IN_DISCLOSURE);
372372
});
373373

374374
/**
375375
* A timed-out `wait text` polled the device and the runner had to re-activate the session app to
376376
* answer those polls (#2682). The disclosure arrives on the failure the same way the occlusion one
377-
* does: in `error.details.hint`.
377+
* does: in `error.hint`.
378378
*/
379379
test('wait timeout whose polls required a foreground repair discloses the repair', async () => {
380380
const sessionStore = makeSessionStore();
@@ -412,7 +412,7 @@ test('wait timeout whose polls required a foreground repair discloses the repair
412412

413413
expect(response.ok).toBe(false);
414414
if (response.ok) return;
415-
expect(String(response.error.details?.hint)).toContain(
415+
expect(String(response.error.hint)).toContain(
416416
iosTargetActivationDisclosure(TARGET_ACTIVATION_FACT),
417417
);
418418
});

0 commit comments

Comments
 (0)