Skip to content

Commit da29d7f

Browse files
authored
feat(interaction): add the moving-target gate the tap path consults (#3471) (#3538)
* feat(interaction): add the moving-target gate the tap path consults (#3471) A press resolves a point from one capture, and nothing between that capture and the dispatch looked at the surface twice. A press issued right after an action that animated the UI can therefore resolve a point the control has already vacated and report success on the vacancy — measured on ADB Android, whose helper reports frames in flight. This adds the judgement: a pure window/stability decision in @agent-device/contracts, the same-node identity lookup it screens with in @agent-device/ad-script, and the runtime seam that spends the window in front of the tap, re-resolving once and tapping where the target came to rest. The session does not yet record the history the gate reads, so every press still answers `idle` and pays nothing. * chore(gates): export the moving-target decision module (#3471) * fix(selectors): keep the moving-target reason in the vocabulary the adapters read (#3471) The eager-closure gate (ADR 0027, `scripts/__tests__/eager-closure-budgets.test.ts`) holds every façade that reads an interaction reason to "no more modules than the merge-base evaluated". Importing `TARGET_STILL_MOVING_REASON` from `@agent-device/contracts/moving-target-gate` pulled that module and its kernel dependencies into six closures, `packages/maestro/src/daemon-port/daemon-runtime-port.ts` and `packages/replay-port/src/daemon-port/session-replay-maestro-runtime.ts` among them. The reason is a literal in `INTERACTION_ERROR_REASONS` again, the one vocabulary every adapter already reads, so the leaf imports nothing; the hint stays in contracts, which is on no façade path.
1 parent f70224a commit da29d7f

12 files changed

Lines changed: 949 additions & 9 deletions

File tree

‎packages/ad-script/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ export {
3939
localIdentitiesEqual,
4040
matchesAncestryPrefix,
4141
matchesLocalIdentity,
42+
nodeRectByIdentity,
4243
readNodeLocalIdentity,
4344
readNodeStructuralDenotation,
4445
siblingOrdinal,

‎packages/ad-script/src/internal/__tests__/target-annotation-identity.test.ts‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import {
66
idMatchCountInTree,
77
matchesAncestryPrefix,
88
matchesLocalIdentity,
9+
nodeRectByIdentity,
910
readNodeLocalIdentity,
1011
} from '../target-annotation-identity.ts';
1112

@@ -124,3 +125,48 @@ test('demoteNonUniqueLocalIdentity passes through identities without an id', ()
124125
assert.equal(identity.id, undefined);
125126
assert.deepEqual(demoteNonUniqueLocalIdentity(identity, nodes), identity);
126127
});
128+
129+
// ---------------------------------------------------------------------------
130+
// nodeRectByIdentity: the fail-closed same-node lookup #3471's gate screens with
131+
// ---------------------------------------------------------------------------
132+
133+
const RECT = { x: 10, y: 240, width: 120, height: 50 };
134+
135+
test('nodeRectByIdentity reports the position a list names once', () => {
136+
const nodes: RawSnapshotNode[] = [{ index: 0, type: 'Button', label: 'Continue', rect: RECT }];
137+
138+
assert.deepEqual(nodeRectByIdentity(nodes, nodes[0]!), RECT);
139+
});
140+
141+
test('nodeRectByIdentity answers nothing for an absent or unmeasurable node', () => {
142+
const target: RawSnapshotNode = { index: 0, type: 'Button', label: 'Continue', rect: RECT };
143+
const frameless: RawSnapshotNode = { index: 0, type: 'Button', label: 'Continue' };
144+
145+
assert.equal(nodeRectByIdentity([], target), undefined);
146+
assert.equal(nodeRectByIdentity([frameless], target), undefined);
147+
});
148+
149+
test('nodeRectByIdentity answers nothing when an identity names several nodes', () => {
150+
// Local identity is role/id/label, so two sibling rows both reading `label="Item"` are ONE
151+
// identity. Guessing between them is positional coincidence — the same fail-closed direction
152+
// `demoteNonUniqueLocalIdentity` takes for a non-unique id.
153+
const rows: RawSnapshotNode[] = [
154+
{ index: 0, type: 'Cell', label: 'Item', rect: { x: 0, y: 100, width: 400, height: 60 } },
155+
{ index: 1, type: 'Cell', label: 'Item', rect: { x: 0, y: 160, width: 400, height: 60 } },
156+
];
157+
158+
assert.equal(nodeRectByIdentity(rows, rows[0]!), undefined);
159+
});
160+
161+
test('nodeRectByIdentity keys on the exact identity triple, so a relabel is a different node', () => {
162+
const before: RawSnapshotNode = {
163+
index: 0,
164+
type: 'Button',
165+
identifier: 'save-action',
166+
label: 'Save',
167+
rect: RECT,
168+
};
169+
const after: RawSnapshotNode = { ...before, label: 'Saved' };
170+
171+
assert.equal(nodeRectByIdentity([before], after), undefined);
172+
});

‎packages/ad-script/src/internal/target-annotation-identity.ts‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
* here, not through `@agent-device/ad-replay`'s façade (#1555 review).
1616
*/
1717

18-
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
18+
import type { RawSnapshotNode, Rect } from '@agent-device/kernel/snapshot';
1919
import { normalizeType } from '@agent-device/contracts/snapshot';
2020
import type {
2121
LocalIdentity,
@@ -120,6 +120,27 @@ export function demoteNonUniqueLocalIdentity(
120120
return { role, ...(label !== undefined ? { label } : {}) };
121121
}
122122

123+
/**
124+
* Where `nodes` puts the node carrying `node`'s local identity, or nothing when the list cannot
125+
* answer: absent, named by more than one node, or named by a node carrying no rect. Local identity is
126+
* role/id/label, NOT element identity, so two sibling rows both reading `label="Item"` are one
127+
* identity — and an ambiguous identity answers nothing, the same fail-closed direction
128+
* `demoteNonUniqueLocalIdentity` takes for a non-unique id. `#3471`'s moving-target gate pairs this
129+
* with `movingTargetRectsStable` to ask whether one node held its position between two trees.
130+
*/
131+
export function nodeRectByIdentity(
132+
nodes: readonly IdentityPositionedNode[],
133+
node: IdentityPositionedNode,
134+
): Rect | undefined {
135+
const identity = readNodeLocalIdentity(node);
136+
const carriers = nodes.filter((candidate) =>
137+
localIdentitiesEqual(identity, readNodeLocalIdentity(candidate)),
138+
);
139+
return carriers.length === 1 ? carriers[0]?.rect : undefined;
140+
}
141+
142+
type IdentityPositionedNode = IdentityTreeNode & { rect?: Rect };
143+
123144
type StructuralNode = Pick<RawSnapshotNode, 'index' | 'parentIndex'>;
124145

125146
/** Zero-based ordinal among the node's own parent's children, in document order. */

‎packages/contracts/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,10 @@
316316
"types": "./src/managed-web-backend.ts",
317317
"default": "./src/managed-web-backend.ts"
318318
},
319+
"./moving-target-gate": {
320+
"types": "./src/moving-target-gate.ts",
321+
"default": "./src/moving-target-gate.ts"
322+
},
319323
"./navigation": {
320324
"types": "./src/navigation.ts",
321325
"default": "./src/navigation.ts"
Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
import assert from 'node:assert/strict';
2+
import { test } from 'vitest';
3+
import {
4+
MOVING_TARGET_GATE_RECT_TOLERANCE_PX,
5+
MOVING_TARGET_GATE_SAMPLE_GAP_MS,
6+
MOVING_TARGET_GATE_WINDOW_MS,
7+
movingTargetGateDecision,
8+
movingTargetRectsStable,
9+
targetStillMovingHint,
10+
} from './moving-target-gate.ts';
11+
12+
// #3471: the two judgements the moving-target gate makes, isolated from both callers so the window
13+
// and the stability rule have exactly one reading. The runtime half (what a press does with these
14+
// answers) is `src/commands/interaction/runtime/moving-target-gate.ts`; the Layer-3 scenarios behind
15+
// the ADR 0011 `targetStability` cells drive the whole press.
16+
17+
const RECT = { x: 100, y: 200, width: 120, height: 44 };
18+
19+
function shifted(over: Partial<typeof RECT> = {}) {
20+
return { ...RECT, ...over };
21+
}
22+
23+
// --- is a second observation owed? ---
24+
25+
test('a session with no mutation clock is idle and owes nothing', () => {
26+
// `targetShownAtRest` is deliberately the disturbing kind here: with no clock there is nothing to
27+
// wait out, and an absent clock must not read as "a mutation just finished".
28+
const decision = movingTargetGateDecision({
29+
baselineFinishedAtMs: undefined,
30+
targetShownAtRest: false,
31+
nowMs: 10_000,
32+
});
33+
34+
assert.deepEqual(decision, { kind: 'idle' });
35+
});
36+
37+
test('a target the previous action left in place is answered without a re-observation', () => {
38+
const decision = movingTargetGateDecision({
39+
baselineFinishedAtMs: 9_900,
40+
targetShownAtRest: true,
41+
nowMs: 10_000,
42+
});
43+
44+
assert.deepEqual(decision, { kind: 'stationary' });
45+
});
46+
47+
test('the window is measured from the mutation END, so only what is LEFT of it is waited out', () => {
48+
// The action finished 400 ms ago; a full window of waiting would be a 900 ms pause on a press
49+
// whose surface has already been quiet for 400 ms.
50+
const decision = movingTargetGateDecision({
51+
baselineFinishedAtMs: 9_600,
52+
targetShownAtRest: false,
53+
nowMs: 10_000,
54+
});
55+
56+
assert.deepEqual(decision, { kind: 'recheck', waitMs: 100 });
57+
});
58+
59+
test('the declared window is the 500 ms transition budget, and its edge decides the branch', () => {
60+
// The edge is exclusive: at exactly the budget the resolved point already stands, because an
61+
// action that finished a full window ago has no claim on this press.
62+
assert.equal(MOVING_TARGET_GATE_WINDOW_MS, 500);
63+
const atEdge = movingTargetGateDecision({
64+
baselineFinishedAtMs: 10_000 - MOVING_TARGET_GATE_WINDOW_MS,
65+
targetShownAtRest: false,
66+
nowMs: 10_000,
67+
});
68+
const inside = movingTargetGateDecision({
69+
baselineFinishedAtMs: 9_600,
70+
targetShownAtRest: false,
71+
nowMs: 10_000,
72+
});
73+
const outside = movingTargetGateDecision({
74+
baselineFinishedAtMs: 9_400,
75+
targetShownAtRest: false,
76+
nowMs: 10_000,
77+
});
78+
79+
assert.deepEqual(atEdge, { kind: 'not-recent' });
80+
assert.equal(inside.kind, 'recheck');
81+
assert.equal(outside.kind, 'not-recent');
82+
});
83+
84+
test('inside the window, only a tree that shows the target at rest excuses the re-observation', () => {
85+
// One bit carries the whole screening verdict, because that is all this branch can use: absent,
86+
// ambiguous, and moved all mean the same thing here — the session cannot show where the target
87+
// came to rest, so it owes the look.
88+
for (const targetShownAtRest of [false, true] as const) {
89+
const decision = movingTargetGateDecision({
90+
baselineFinishedAtMs: 9_950,
91+
targetShownAtRest,
92+
nowMs: 10_000,
93+
});
94+
95+
assert.equal(
96+
decision.kind,
97+
targetShownAtRest ? 'stationary' : 'recheck',
98+
`targetShownAtRest ${String(targetShownAtRest)}`,
99+
);
100+
}
101+
});
102+
103+
test('the wait never drops below one sample gap, even as the window closes', () => {
104+
// Re-observing immediately would compare two captures the same animation frame could produce,
105+
// and agreement would then be a timing accident rather than the target having stopped.
106+
const decision = movingTargetGateDecision({
107+
baselineFinishedAtMs: 9_520,
108+
targetShownAtRest: false,
109+
nowMs: 10_000,
110+
});
111+
112+
assert.deepEqual(decision, { kind: 'recheck', waitMs: MOVING_TARGET_GATE_SAMPLE_GAP_MS });
113+
});
114+
115+
test('a clock skew ahead of the mutation does not produce a negative wait', () => {
116+
const decision = movingTargetGateDecision({
117+
baselineFinishedAtMs: 10_100,
118+
targetShownAtRest: false,
119+
nowMs: 10_000,
120+
});
121+
122+
assert.ok(decision.kind === 'recheck');
123+
assert.ok(decision.waitMs >= MOVING_TARGET_GATE_SAMPLE_GAP_MS);
124+
});
125+
126+
// --- has the target stopped? ---
127+
128+
test('two observations at the same rect are a resting position', () => {
129+
assert.equal(movingTargetRectsStable(RECT, shifted()), true);
130+
});
131+
132+
test('capture rounding within the declared tolerance is not motion', () => {
133+
assert.equal(MOVING_TARGET_GATE_RECT_TOLERANCE_PX, 1);
134+
assert.equal(movingTargetRectsStable(RECT, shifted({ x: RECT.x + 1 })), true);
135+
assert.equal(movingTargetRectsStable(RECT, shifted({ height: RECT.height + 1 })), true);
136+
});
137+
138+
test('one axis moving is enough to still be moving', () => {
139+
// The slide-in case from the #3471 repro (ADB Android): a translate animation keeps width and
140+
// height constant, so only one axis says the target moved.
141+
assert.equal(movingTargetRectsStable(RECT, shifted({ x: RECT.x + 2 })), false);
142+
assert.equal(movingTargetRectsStable(RECT, shifted({ y: RECT.y + 2 })), false);
143+
assert.equal(movingTargetRectsStable(RECT, shifted({ width: RECT.width + 2 })), false);
144+
assert.equal(movingTargetRectsStable(RECT, shifted({ height: RECT.height + 2 })), false);
145+
});
146+
147+
test('an observation that could not measure the target never agrees with one that could', () => {
148+
// Absence is the surface having changed under the point, which is the exposure the gate exists
149+
// to catch — so it can never be the observation that certifies rest.
150+
assert.equal(movingTargetRectsStable(RECT, undefined), false);
151+
assert.equal(movingTargetRectsStable(undefined, RECT), false);
152+
assert.equal(movingTargetRectsStable(undefined, undefined), false);
153+
assert.equal(movingTargetRectsStable(RECT, shifted({ width: Number.NaN })), false);
154+
});
155+
156+
test('the stability comparison is symmetric', () => {
157+
assert.equal(
158+
movingTargetRectsStable(shifted({ y: 205 }), RECT),
159+
movingTargetRectsStable(RECT, shifted({ y: 205 })),
160+
);
161+
});
162+
163+
// --- the refusal's published recovery ---
164+
165+
test('a refused selector press names the wait that recovers it', () => {
166+
const hint = targetStillMovingHint({ selector: 'label="Slide target"' });
167+
168+
assert.match(hint, /wait 'label="Slide target"'/);
169+
assert.match(hint, /same selector/);
170+
assert.match(hint, /--settle/);
171+
assert.match(hint, /nothing was tapped/);
172+
});
173+
174+
test('a refused @ref press is sent to a selector, because the wait would retire its frame', () => {
175+
// ADR 0014: the capture a `wait` takes expires the ref frame, so telling the caller to retry the
176+
// same ref would trade a motion refusal for a stale-ref refusal.
177+
const hint = targetStillMovingHint({ selector: undefined });
178+
179+
assert.match(hint, /snapshot -i/);
180+
assert.match(hint, /selector/);
181+
assert.match(hint, /ref frame/);
182+
assert.doesNotMatch(hint, /wait '/);
183+
});

0 commit comments

Comments
 (0)