Skip to content

Commit 017113c

Browse files
committed
fix(daemon): record only the tree a mutation actually acted through (#3471)
`chooseMovingTargetTree` preferred the session snapshot whenever its node array differed from the tree the action resolved against, reasoning that a `--settle` or `--verify` capture must have published something newer. Array identity cannot establish that: a freshness capture and a `@ref` press's authorized frame reach `session.snapshot` the same way, so the marker could record a surface this action never looked at — the exact substitution its own docblock forbids, and a `stationary` verdict made against the wrong tree. The baseline is now the acted-through tree, always. A settled press simply owes its successor one re-observation instead of being answered from a tree of unknown provenance. The corroborated-failure path also dropped the tree. That sibling call reached the same finalizer without `preActionTree`, so a corroborated iOS tap retired the baseline a previous mutation had recorded and left the next press unscreened — a dispatch that touched the device behaving like one that never looked at a tree. It now passes the resolved target's own tree. Help and website docs now scope the guarantee to what the gate actually covers: ref and selector targets on the runtime path, dispatched (not "tapped") at the resting position, `movedToRect` optional, and a raw-coordinate press explicitly outside it. Refusing to narrow the WRITER to press/click: the ADR 0011 cells scope which commands the gate guards, while this marker asks whether a mutation just left an animating surface, and a `fill` that opened a keyboard or a `longpress` that started a drag answers that the same way a press does. The breadth is now pinned by a test rather than left implicit.
1 parent 862fce3 commit 017113c

6 files changed

Lines changed: 79 additions & 55 deletions

File tree

‎src/commands/interaction/metadata.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,9 @@ const FIND_ACTION_VALUES = [
5757

5858
const interactionCommandDescriptions = {
5959
click:
60-
"Activate a UI target by snapshot ref, selector, or coordinates. Prefer a ref or selector after a snapshot; use coordinates only when semantic targeting is unavailable. This can change app state. Success means the click was dispatched, not that it landed: use settle or verify to confirm the result without a follow-up snapshot, or wait for the expected screen. A target still moving shortly after the session's previous action is re-observed and tapped at its resting position; one that has not stopped refuses with target_still_moving rather than tapping where it was.",
60+
"Activate a UI target by snapshot ref, selector, or coordinates. Prefer a ref or selector after a snapshot; use coordinates only when semantic targeting is unavailable. This can change app state. Success means the click was dispatched, not that it landed: use settle or verify to confirm the result without a follow-up snapshot, or wait for the expected screen. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was.",
6161
press:
62-
"Short-press a UI target by snapshot ref, selector, or coordinates. Use longpress instead when the target requires a context-menu or hold gesture. Success means the press was dispatched, not that it landed: use settle or verify to confirm the result, or wait for the expected screen. A target still moving shortly after the session's previous action is re-observed and tapped at its resting position; one that has not stopped refuses with target_still_moving rather than tapping where it was.",
62+
"Short-press a UI target by snapshot ref, selector, or coordinates. Use longpress instead when the target requires a context-menu or hold gesture. Success means the press was dispatched, not that it landed: use settle or verify to confirm the result, or wait for the expected screen. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was.",
6363
fill: 'Replace text in a UI input selected by snapshot ref, selector, or coordinates. Pass an empty text to clear the field. Inspect unconfirmed evidence and assert the expected value or resulting screen before continuing. Prefer refs or selectors after snapshot; use recordAs to keep sensitive text out of a recorded replay while sending it to the live app.',
6464
longpress:
6565
'Hold a UI target by snapshot ref, selector, or coordinates to open a context menu or perform another hold gesture. Set durationMs when the default hold duration is unsuitable.',

‎src/daemon/__tests__/moving-target-baseline.test.ts‎

Lines changed: 32 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -92,28 +92,20 @@ test('a baseline from an earlier action is replaced rather than kept beside the
9292
assert.notEqual(second.tree.nodes, first.tree.nodes);
9393
});
9494

95-
test('a newer published observation beats the tree the action resolved against', () => {
96-
// The `--settle` and `--verify` case: the action published a tree AFTER its own resolution, and
97-
// that later capture is the better account of where the surface came to rest.
95+
test('the baseline is the tree the action acted through, whatever the session now holds', () => {
96+
// The session snapshot may have been replaced by a post-action capture (a `--settle` observation,
97+
// a `--verify` capture) — the better account of where the surface came to rest — but nothing here
98+
// can tell one from a freshness capture or a `@ref` press's authorized frame, because all three
99+
// arrive as a different node array. Adopting it on array identity would screen the next press
100+
// against positions this action never looked at, which is what this marker exists to prevent. So
101+
// the acted-through tree wins and the settled press simply owes its successor a re-observation.
98102
const session = makeSession('android');
99-
const settled = appTree('Settled');
100-
session.snapshot = settled;
103+
const actedThrough = appTree('Pre-action');
104+
session.snapshot = appTree('Newer, provenance unknown');
101105

102-
markPress(session, { nodes: appTree('Pre-action').nodes });
106+
markPress(session, { nodes: actedThrough.nodes });
103107

104-
assert.equal(movingTargetBaseline(session)?.tree.nodes, settled.nodes);
105-
});
106-
107-
test('the observation an action resolved against is kept when it is also the newest one', () => {
108-
// Identity, not content, decides: `surfaceScopedNodes` keeps the capture's own array, so a session
109-
// holding that same array has published nothing newer than the action's own resolution.
110-
const session = makeSession('android');
111-
const resolved = appTree();
112-
session.snapshot = resolved;
113-
114-
markPress(session, { nodes: resolved.nodes });
115-
116-
assert.equal(movingTargetBaseline(session)?.tree.nodes, resolved.nodes);
108+
assert.equal(movingTargetBaseline(session)?.tree.nodes, actedThrough.nodes);
117109
});
118110

119111
test('a baseline keeps the surface its tree came from', () => {
@@ -213,6 +205,27 @@ test('a gesture takes the surface over from a press, rather than joining it', ()
213205
assert.equal(movingTargetBaseline(session), undefined);
214206
});
215207

208+
test('the trigger is wider than the guard on purpose: any command that acted through a tree records it', () => {
209+
// The ADR 0011 `targetStability` cells scope which commands the GATE refuses to dispatch blind
210+
// (press, click). This marker answers a different question — did a mutation just leave a surface
211+
// that may still be animating — and a `longpress`, `hover`, or `fill` that resolved its target
212+
// through a captured tree can animate the surface exactly as a press does. Narrowing the writer to
213+
// press/click would un-guard the press that follows a `fill` which opened a keyboard, which is the
214+
// failure the issue describes with a different verb in front of it.
215+
const session = makeSession('android');
216+
const actedThrough = appTree();
217+
218+
markDeferredInteractionOutcome({
219+
session,
220+
command: 'fill',
221+
positionals: ['label="Field"', 'hello'],
222+
flags: undefined,
223+
preActionTree: { nodes: actedThrough.nodes },
224+
});
225+
226+
assert.equal(movingTargetBaseline(session)?.tree.nodes, actedThrough.nodes);
227+
});
228+
216229
test('an ineligible action marks nothing at all, leaving the record untouched', () => {
217230
// Eligibility stays per marker: `type` is neither gesture-shaped nor given a resolved tree here,
218231
// so a session that was idle stays idle rather than owing every later press a re-observation.

‎src/daemon/deferred-interaction-outcome.ts‎

Lines changed: 11 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -143,39 +143,30 @@ function markSurfaceStabilization(
143143
* tap, `back`, an app open) leave the session idle: the ordinary navigate-then-tap step pays the
144144
* gate nothing.
145145
*
146-
* When the action published a newer observation than the one it resolved against — a `--settle`
147-
* observation stores the settled tree, `--verify` captures after the action — that newer tree is the
148-
* better account of where the surface came to rest, so it becomes the baseline instead. The
149-
* comparison is by node-array identity: `surfaceScopedNodes` keeps the capture's own array, so a
150-
* session observation holding the same array is the same observation, and one holding a different
151-
* array is a later one.
146+
* The baseline is that tree and never the session's current snapshot. A `--settle` or `--verify`
147+
* press does publish a post-action observation, and it would be the better account of where the
148+
* surface came to rest — but nothing here can tell a post-action capture from a freshness capture or
149+
* a `@ref` press's authorized frame, since all three reach `session.snapshot` as a different node
150+
* array from the one the action resolved against. Inferring "newer" from array identity would
151+
* attribute to this action a surface it may never have looked at, which is the exact thing this
152+
* marker forbids. A settled press therefore leaves the next press owed a re-observation rather than
153+
* answered from a tree of unknown provenance: the cost lands on one extra pair of captures inside the
154+
* window, never on a rest judgement made against the wrong surface.
152155
*/
153156
function markMovingTargetBaseline(
154157
session: SessionState,
155158
preActionTree: SurfaceScopedNodes | undefined,
156159
): void {
157-
const tree = chooseMovingTargetTree(session.snapshot, preActionTree);
158-
if (!tree || tree.nodes.length === 0) {
160+
if (!preActionTree || preActionTree.nodes.length === 0) {
159161
retireMovingTargetBaseline(session);
160162
return;
161163
}
162164
session.deferredSurface = {
163165
kind: 'recheck',
164-
baseline: { finishedAt: Date.now(), tree },
166+
baseline: { finishedAt: Date.now(), tree: preActionTree },
165167
};
166168
}
167169

168-
function chooseMovingTargetTree(
169-
published: SnapshotState | undefined,
170-
preActionTree: SurfaceScopedNodes | undefined,
171-
): SurfaceScopedNodes | undefined {
172-
if (!preActionTree) return undefined;
173-
if (published && published.nodes !== preActionTree.nodes) {
174-
return movingTargetBaselineTree(published);
175-
}
176-
return preActionTree;
177-
}
178-
179170
/**
180171
* Retires the `recheck` variant only. Called from ADR 0014's device side-effect seam: a side effect
181172
* this session did not resolve a target through leaves a tree that predates a change nobody
@@ -189,19 +180,6 @@ export function retireMovingTargetBaseline(session: SessionState): void {
189180
session.deferredSurface = undefined;
190181
}
191182

192-
/** Narrows a capture to the shape a baseline keeps, so a stored tree never smuggles in a surface. */
193-
function movingTargetBaselineTree(
194-
snapshot: SnapshotState | undefined,
195-
): SurfaceScopedNodes | undefined {
196-
if (!snapshot) return undefined;
197-
return {
198-
nodes: snapshot.nodes,
199-
...(snapshot.iosSystemSurfaceBundleId
200-
? { surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
201-
: {}),
202-
};
203-
}
204-
205183
/**
206184
* The baseline the #3471 gate screens a freshly resolved rect against, or nothing when the session
207185
* is idle or owes the capture-side loop instead. The tree is carried to the command layer because

‎src/daemon/interaction/internal/__tests__/interaction-ios-tap-outcome.test.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import {
3333
} from '../../../__tests__/interaction-get-runtime-fixture.ts';
3434
import { captureSnapshotWithInteractor } from '../../../snapshot-interactor-capture.ts';
3535
import { corroborateIosTapFailure } from '../interaction-ios-tap-outcome.ts';
36+
import { movingTargetBaseline } from '../../../deferred-interaction-outcome.ts';
3637
import { mkdtempForTestSync } from '@agent-device/test-kit/tmp-dir';
3738
import { testCreateCommandSurface } from '../../../__tests__/command-surface-fixture.ts';
3839

@@ -134,6 +135,34 @@ test('a changed post-action capture corroborates an iOS tap reported as failed',
134135
expect(sessionStore.get(sessionName)?.actions).toHaveLength(1);
135136
});
136137

138+
test('a corroborated tap failure still records the tree it acted through (#3471)', async () => {
139+
// The corroboration is a dispatch that touched the device through a resolved tree, so it owes the
140+
// moving-target baseline like any other mutation: withholding it there would RETIRE the record a
141+
// previous action made and leave the next press unscreened by the gate.
142+
const sessionName = 'ios-tap-corroboration-baseline';
143+
const sessionStore = makeSessionStore();
144+
const session = makeIosSession(sessionName, {
145+
appBundleId: 'com.example.app',
146+
snapshot: snapshot(profileNodes),
147+
});
148+
sessionStore.publish(sessionName, session);
149+
legacyDispatchCapture.mockImplementation(async (_device, command) => {
150+
if (command === 'press') {
151+
throw new AppError(
152+
'XCTEST_RECORDED_FAILURE',
153+
'XCTest recorded a failure while executing tap; the action may not have been performed.',
154+
);
155+
}
156+
if (command === 'snapshot') return snapshotPayload(imageViewerNodes);
157+
return {};
158+
});
159+
160+
const response = await runClick(sessionStore, sessionName);
161+
162+
expect(response?.ok).toBe(true);
163+
expect(movingTargetBaseline(sessionStore.get(sessionName)!)).toBeDefined();
164+
});
165+
137166
test('an unchanged post-action capture keeps a failed iOS tap failed', async () => {
138167
const sessionName = 'ios-unchanged-tap-corroboration';
139168
const sessionStore = makeSessionStore();

‎src/daemon/interaction/internal/interaction-touch-runtime.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,10 @@ async function buildRuntimeIosCorroboratedResponse(params: {
191191
actionStartedAt: params.actionStartedAt,
192192
actionFinishedAt: Date.now(),
193193
androidFreshnessBaseline: params.androidFreshnessBaseline,
194+
// A corroborated failure still acted through a tree, and its own effect may still be animating.
195+
// Withholding it would retire the evidence a previous mutation recorded and leave the next press
196+
// unscreened, which is the opposite of what a dispatch that touched the device should do.
197+
...(resolvedTarget ? { preActionTree: resolvedTarget.preAction } : {}),
194198
});
195199
}
196200

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -422,7 +422,7 @@ agent-device gesture transform 200 420 80 -40 2 35 700 # combined pan, zoom, and
422422
423423
A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait <selector>`. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command).
424424
425-
A `click` or `press` that follows an action which animated the screen checks the target's position again before tapping, and taps where it came to rest. A target still moving when that check runs out refuses with `target_still_moving` and `dispatched: no`: nothing was tapped, so the point the command resolved is in `error.details.rect` and where the target was last seen is `error.details.movedToRect`. Run `wait <selector>` on the target and retry the same command, or add `--settle` so the action waits for the UI to go quiet. A refused `@ref` press needs a fresh `snapshot -i` and a selector, because the wait retires the ref.
425+
A `click` or `press` that names its target by `@ref` or selector, and follows an action which animated the screen, checks the target's position again before dispatching, and dispatches where it came to rest. A target still moving when that check runs out refuses with `target_still_moving` and `dispatched: no`: nothing was tapped, the point the command resolved is in `error.details.rect`, and where the target was last seen is in `error.details.movedToRect` when that last look measured one. Run `wait <selector>` on the target and retry the same command, or add `--settle` so the action waits for the UI to go quiet. A refused `@ref` press needs a fresh `snapshot -i` and a selector, because the wait retires the ref. A press at raw coordinates has no target to re-check.
426426
427427
### Text entry
428428

0 commit comments

Comments
 (0)