Skip to content

Commit 69a0b87

Browse files
claudethymikee
authored andcommitted
fix(find): resolve a mutating find's target once, not twice (#1654)
A mutating `find click`/`find fill` captured the screen, matched by locator under the `findAct` policy, promoted to a hittable ancestor, minted `@eN` off the node it chose — and then re-entered the interaction leaf by bare `@ref`, which looked that ref up AGAIN via resolveSnapshotForRef. The second lookup reads the SESSION frame tree, not the fresh capture find matched against, so the two could disagree: it could hand the action a different node than find picked, or refuse as unresolvable a ref find had observed a moment earlier. find now passes the node itself. `internal.findPreresolvedTarget` carries the resolved node and its tree over the in-process invoke hop, and the ref branch adopts them instead of resolving the ref a second time. What is NOT skipped: the guarantees. Occlusion, hittable-ancestor promotion, and the off-screen guard all still run, on that node, at the same symbols the ADR 0011 `runtime-ref` cells name. Only the LOOKUP is replaced. Recording, ref-frame effects, settle/observation, and deferred-outcome marking are untouched — they live in the dispatch wrapper, not in resolution, which is why the invoke hop is kept rather than bypassed the way find focus/type bypass it. The channel is a second field rather than widening `findResolvedTarget` because they answer different questions: that flag governs ref-frame admission and staleness (policy), this governs resolution (lookup), and focus/type set neither. It is `internal`-only, so it never crosses the wire and carrying live node references is sound. ADR 0011 re-check: the `runtime-ref` disclosure cell gains a second producer, adoptPreresolvedRefTarget, reporting `exact`. Truthful rather than borrowed — the ref is minted off the node handed over. It cannot report label-fallback, which is right: label recovery is a property of looking a stale ref up, and this path performs no lookup. Tests pin the behavior by tap coordinates, so they name which tree the leaf resolved against, plus a control proving the ordinary @ref path is unchanged and a case where the session tree cannot resolve the ref at all — that one can only pass if no second lookup runs. Verified revert-sensitive: removing the short-circuit fails 3 of the 4, and the control stays green. Out of scope, unchanged: the ADR 0011 `native-ref` path (web provider clickRef/fillRef only), where the ref is the provider's own element handle. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_0117wfrvC6MRDJUWdRBErNEF
1 parent 7b4e461 commit 69a0b87

9 files changed

Lines changed: 278 additions & 9 deletions

File tree

‎packages/contracts/src/facades/interaction.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ export type {
9090
LongPressCommandResponseData,
9191
LongPressCommandResult,
9292
PointTarget,
93+
PreresolvedInteractionTarget,
9394
PressCommandResponseData,
9495
PressCommandResult,
9596
RecordingTargetOverride,

‎packages/contracts/src/interaction-guarantees.ts‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,8 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
204204
},
205205
},
206206
'runtime-ref': {
207-
description: 'Session snapshot ref lookup, guarded coordinate tap.',
207+
description:
208+
'Session snapshot ref lookup, guarded coordinate tap. #1654: when the caller already resolved the node (a mutating `find`), the lookup is replaced by that node and every guarantee below is enforced against it — the guards are unchanged, only the lookup is skipped.',
208209
commands: ['press', 'click', 'fill', 'longpress'],
209210
guarantees: {
210211
...RUNTIME_TREE_SHARED_GUARANTEES,
@@ -219,6 +220,13 @@ export const INTERACTION_DISPATCH_PATHS: Record<InteractionPathId, InteractionPa
219220
},
220221
// ADR 0012 decision 2: tryResolveRefNode produces both outcomes — exact
221222
// for a resolved @ref, label-fallback for trailing-label recovery.
223+
//
224+
// #1654 second producer on this path: adoptPreresolvedRefTarget reports
225+
// `exact` for a pre-resolved target. Truthful rather than borrowed — the
226+
// producer mints the ref off the node it hands over, so the ref does name
227+
// that node exactly. It cannot report label-fallback, which is correct:
228+
// label recovery is a property of LOOKING a stale ref up, and a
229+
// pre-resolved target never performs that lookup.
222230
resolutionDisclosure: {
223231
kind: 'runtime',
224232
via: 'src/commands/interaction/runtime/resolution.ts#tryResolveRefNode',

‎packages/contracts/src/interaction.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,35 @@ export type RecordingTargetOverride = {
8686
refLabel?: string;
8787
};
8888

89+
/**
90+
* #1654: a target its CALLER already resolved, against a tree the caller
91+
* captured itself, handed to the interaction leaf so the leaf does not resolve
92+
* the same `@ref` a second time.
93+
*
94+
* The sole producer is a mutating `find` (`src/daemon/handlers/find.ts`): it
95+
* captures, matches by locator, promotes to a hittable ancestor, and mints
96+
* `@eN` off the node it chose — then re-enters the interaction leaf. Without
97+
* this channel the leaf looked `@eN` up again in the session frame tree, so a
98+
* tree that had since advanced could hand the action a DIFFERENT node than the
99+
* one find matched, or refuse a ref find had just observed.
100+
*
101+
* What this does NOT skip: the shared guards. Occlusion, hittable-ancestor
102+
* promotion, and the off-screen check still run, on this node, at the same
103+
* symbols the ADR 0011 `runtime-ref` cells name — the pre-resolution replaces
104+
* the LOOKUP, not the guarantees.
105+
*
106+
* In-process only, like `RecordingTargetOverride` above: it travels on the
107+
* daemon-only `internal` request channel (`toDaemonRequest` never copies that
108+
* off the wire), carries live node references, and is never serialized into a
109+
* response.
110+
*/
111+
export type PreresolvedInteractionTarget = {
112+
/** The node the caller resolved, after the caller's own promotion. */
113+
node: SnapshotNode;
114+
/** The tree `node` came from — the guards read its siblings for occlusion/viewport. */
115+
nodes: SnapshotNode[];
116+
};
117+
89118
export type ResolvedInteractionTarget =
90119
| {
91120
kind: 'point';

‎src/commands/interaction/runtime/interactions.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import type {
22
ClickButton,
33
FillCommandResult,
4+
PreresolvedInteractionTarget,
45
PressCommandResult,
56
RepeatedInput,
67
ResolvedTarget,
@@ -49,6 +50,8 @@ export type PressCommandOptions = CommandContext &
4950
button?: ClickButton;
5051
/** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */
5152
expectedResolvedTarget?: ExpectedResolvedTarget;
53+
/** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */
54+
preresolvedTarget?: PreresolvedInteractionTarget;
5255
} & PostActionObservationOptions;
5356

5457
export type ClickCommandOptions = PressCommandOptions;
@@ -63,6 +66,8 @@ export type FillCommandOptions = CommandContext & {
6366
allowNonHittableCoordinateFallback?: boolean;
6467
/** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */
6568
expectedResolvedTarget?: ExpectedResolvedTarget;
69+
/** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */
70+
preresolvedTarget?: PreresolvedInteractionTarget;
6671
} & PostActionObservationOptions;
6772

6873
export type TypeTextCommandOptions = CommandContext & {
@@ -103,6 +108,7 @@ export const fillCommand: RuntimeCommand<FillCommandOptions, FillCommandResult>
103108
promoteToHittableAncestor: false,
104109
captureEvidenceBaseline: observation.needsPreActionBaseline,
105110
expectedResolvedTarget: options.expectedResolvedTarget,
111+
preresolvedTarget: options.preresolvedTarget,
106112
});
107113
if (!runtime.backend.fill) {
108114
throw new AppError('UNSUPPORTED_OPERATION', 'fill is not supported by this backend');
@@ -192,6 +198,7 @@ async function tapCommand(
192198
promoteToHittableAncestor: true,
193199
captureEvidenceBaseline: observation.needsPreActionBaseline,
194200
expectedResolvedTarget: options.expectedResolvedTarget,
201+
preresolvedTarget: options.preresolvedTarget,
195202
});
196203
if (!runtime.backend.tap) {
197204
throw new AppError('UNSUPPORTED_OPERATION', 'tap is not supported by this backend');
@@ -246,6 +253,13 @@ async function maybeTapRefTarget(
246253
// ADR 0012 step 4: a guarded replay action needs the runtime resolution
247254
// path so the post-resolution identity guard actually runs.
248255
if (options.expectedResolvedTarget) return null;
256+
// #1654 scope note: `preresolvedTarget` deliberately does NOT divert this
257+
// path. It is the separate ADR 0011 `native-ref` dispatch path, wired only
258+
// to the web provider's clickRef/fillRef (no mobile backend implements
259+
// tapTarget/fillTarget), where the ref IS the provider's own element handle.
260+
// Forcing a coordinate tap there to save a resolution would trade a working
261+
// provider-native dispatch for a worse one. The double-hop #1654 removes is
262+
// the runtime tree path below.
249263
// ADR 0011 native-ref preflight: the shared occlusion/offscreen guards run
250264
// against the stored session snapshot node before the backend call (a
251265
// backend fast path can silently "succeed", so errors must be raised here).

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 49 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ import { truncateUtf8 } from '../../../utils/truncate-utf8.ts';
3636
import type {
3737
InteractionTarget,
3838
PointTarget,
39+
PreresolvedInteractionTarget,
3940
RecordingTargetOverride,
4041
ResolutionDiagnosticEntry,
4142
ResolutionDisclosure,
@@ -147,6 +148,12 @@ type ResolveInteractionTargetParams = {
147148
expectedResolvedTarget?: ExpectedResolvedTarget;
148149
/** Identifies one endpoint when a multi-target replay guard refuses. */
149150
replayTargetRole?: 'source' | 'destination';
151+
/**
152+
* #1654: the caller already resolved this `@ref` against its own capture, so
153+
* the ref branch adopts that node instead of looking the ref up again. Ref
154+
* targets only — a selector target has nothing pre-resolved to adopt.
155+
*/
156+
preresolvedTarget?: PreresolvedInteractionTarget;
150157
};
151158

152159
export async function resolveInteractionTarget(
@@ -225,17 +232,53 @@ async function tryCaptureEvidenceBaseline(
225232
}
226233
}
227234

235+
/** The node a ref target acts on, plus the tree the shared guards read it against. */
236+
type RefResolution = { nodes: SnapshotState['nodes']; resolved: ResolvedRefNode };
237+
238+
/**
239+
* #1654: adopt the node the caller already resolved instead of resolving the
240+
* same `@ref` a second time. This replaces the LOOKUP only — every guard below
241+
* still runs, against the caller's tree, at the symbols the ADR 0011
242+
* `runtime-ref` cells name.
243+
*
244+
* `exact` is the truthful disclosure here rather than a borrowed default: the
245+
* producer mints `@eN` off the very node it hands over, so the ref does name
246+
* that node exactly — the same provenance `tryResolveRefNode` reports when a
247+
* ref resolves without label recovery.
248+
*/
249+
function adoptPreresolvedRefTarget(
250+
target: Extract<InteractionTarget, { kind: 'ref' }>,
251+
preresolved: PreresolvedInteractionTarget,
252+
): RefResolution {
253+
const ref = normalizeRef(target.ref);
254+
if (!ref) throw new AppError('INVALID_ARGS', `Invalid ref: ${target.ref}`);
255+
return {
256+
nodes: preresolved.nodes,
257+
resolved: { ref, node: preresolved.node, resolution: EXACT_REF_RESOLUTION },
258+
};
259+
}
260+
261+
async function readRefResolution(
262+
runtime: AgentDeviceRuntime,
263+
options: CommandContext,
264+
target: Extract<InteractionTarget, { kind: 'ref' }>,
265+
): Promise<RefResolution> {
266+
const capture = await resolveSnapshotForRef(runtime, options, target);
267+
return { nodes: capture.snapshot.nodes, resolved: capture.resolved };
268+
}
269+
228270
async function resolveRefInteractionTarget(
229271
runtime: AgentDeviceRuntime,
230272
options: CommandContext,
231273
target: Extract<InteractionTarget, { kind: 'ref' }>,
232274
params: ResolveInteractionTargetParams,
233275
): Promise<ResolvedInteractionTarget> {
234-
const capture = await resolveSnapshotForRef(runtime, options, target);
235-
const resolved = capture.resolved;
236-
assertReplayTargetResolution(resolved.node, capture.snapshot.nodes, params);
276+
const { nodes, resolved } = params.preresolvedTarget
277+
? adoptPreresolvedRefTarget(target, params.preresolvedTarget)
278+
: await readRefResolution(runtime, options, target);
279+
assertReplayTargetResolution(resolved.node, nodes, params);
237280
const node = params.promoteToHittableAncestor
238-
? resolveActionableNodeOrThrow(capture.snapshot.nodes, resolved.node, {
281+
? resolveActionableNodeOrThrow(nodes, resolved.node, {
239282
action: params.action,
240283
label: `Ref ${target.ref}`,
241284
})
@@ -246,7 +289,7 @@ async function resolveRefInteractionTarget(
246289
runtime,
247290
options,
248291
node,
249-
capture.snapshot.nodes,
292+
nodes,
250293
target.ref,
251294
params.action,
252295
);
@@ -258,7 +301,7 @@ async function resolveRefInteractionTarget(
258301
...describeResolvedInteractionNode(
259302
runtime,
260303
visibleNode,
261-
capture.snapshot.nodes,
304+
nodes,
262305
params.action,
263306
resolved.resolution,
264307
),

‎src/daemon/handlers/__tests__/interaction.test.ts‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3737,6 +3737,138 @@ test("ADR 0014 blocker-2: a mutating find's internal fill from an expired frame
37373737
}
37383738
});
37393739

3740+
// #1654: the tree a mutating `find` matched against, deliberately NOT the tree
3741+
// stored on the session. `@e2` names the "Continue" button in both trees, but
3742+
// at a distinctive point here and at (60, 40) in the session frame tree — so
3743+
// the tap coordinates say which tree the leaf actually resolved against.
3744+
function makeFindPreresolvedTree() {
3745+
const nodes = attachRefs([
3746+
{ index: 0, type: 'Application', rect: { x: 0, y: 0, width: 390, height: 844 } },
3747+
{
3748+
index: 1,
3749+
parentIndex: 0,
3750+
type: 'XCUIElementTypeButton',
3751+
label: 'Continue',
3752+
rect: { x: 300, y: 500, width: 20, height: 20 },
3753+
enabled: true,
3754+
hittable: true,
3755+
},
3756+
] as never);
3757+
return { nodes, node: nodes[1] as NonNullable<(typeof nodes)[number]> };
3758+
}
3759+
3760+
async function runFindInternalClick(
3761+
sessionStore: SessionStore,
3762+
sessionName: string,
3763+
internal: Record<string, unknown>,
3764+
) {
3765+
return await handleInteractionCommands({
3766+
req: {
3767+
token: 't',
3768+
session: sessionName,
3769+
command: 'click',
3770+
positionals: ['@e2'],
3771+
flags: { noRecord: true },
3772+
internal,
3773+
},
3774+
sessionName,
3775+
sessionStore,
3776+
contextFromFlags,
3777+
});
3778+
}
3779+
3780+
function readPressPoint(): string[] | undefined {
3781+
const call = mockDispatch.mock.calls.find((entry) => entry[1] === 'press');
3782+
return call?.[2] as string[] | undefined;
3783+
}
3784+
3785+
test('#1654: a mutating find click acts on the node find resolved, not a re-resolved @ref', async () => {
3786+
const sessionStore = makeSessionStore();
3787+
const sessionName = 'find-preresolved-click';
3788+
sessionStore.set(sessionName, makeStaleRefSession(sessionName));
3789+
mockDispatch.mockResolvedValue({});
3790+
const preresolved = makeFindPreresolvedTree();
3791+
3792+
const response = await runFindInternalClick(sessionStore, sessionName, {
3793+
findResolvedTarget: true,
3794+
findPreresolvedTarget: { node: preresolved.node, nodes: preresolved.nodes },
3795+
});
3796+
3797+
expect(response?.ok).toBe(true);
3798+
// find's node — (300,500,20,20) → (310, 510). A second resolution would read
3799+
// @e2 out of the SESSION frame tree instead and tap (60, 40).
3800+
expect(readPressPoint()).toEqual(['310', '510']);
3801+
});
3802+
3803+
test('#1654 control: without the pre-resolved node the same click still resolves @ref from the session tree', async () => {
3804+
const sessionStore = makeSessionStore();
3805+
const sessionName = 'find-preresolved-control';
3806+
sessionStore.set(sessionName, makeStaleRefSession(sessionName));
3807+
mockDispatch.mockResolvedValue({});
3808+
3809+
const response = await runFindInternalClick(sessionStore, sessionName, {
3810+
findResolvedTarget: true,
3811+
});
3812+
3813+
// The ordinary ref path is untouched: @e1 is "Continue" at (10,20,100,40).
3814+
expect(response?.ok).toBe(true);
3815+
expect(readPressPoint()).toEqual(['60', '40']);
3816+
});
3817+
3818+
test('#1654: the leaf performs no ref lookup at all — a session that could not resolve @e2 still acts', async () => {
3819+
// The decisive shape: this session has NO stored snapshot, so any second
3820+
// resolution necessarily fails ("Ref @e2 not found or has no bounds"). The
3821+
// action succeeding proves the lookup did not run, rather than merely
3822+
// agreeing with it. It is also the real-world win — find captured this node
3823+
// a moment ago, so refusing it as unresolvable was never right.
3824+
const sessionStore = makeSessionStore();
3825+
const sessionName = 'find-preresolved-no-session-tree';
3826+
const session = makeSession(sessionName);
3827+
session.snapshot = undefined;
3828+
sessionStore.set(sessionName, session);
3829+
mockDispatch.mockResolvedValue({});
3830+
const preresolved = makeFindPreresolvedTree();
3831+
3832+
const withoutPreresolution = await runFindInternalClick(sessionStore, sessionName, {
3833+
findResolvedTarget: true,
3834+
});
3835+
expect(withoutPreresolution?.ok).toBe(false);
3836+
3837+
const response = await runFindInternalClick(sessionStore, sessionName, {
3838+
findResolvedTarget: true,
3839+
findPreresolvedTarget: { node: preresolved.node, nodes: preresolved.nodes },
3840+
});
3841+
3842+
expect(response?.ok).toBe(true);
3843+
expect(readPressPoint()).toEqual(['310', '510']);
3844+
});
3845+
3846+
test('#1654: the shared guards still run on the pre-resolved node', async () => {
3847+
// The pre-resolution replaces the lookup, not the guarantees: an occluded
3848+
// node is still refused, at the same ADR 0011 `runtime-ref` occlusion cell.
3849+
const sessionStore = makeSessionStore();
3850+
const sessionName = 'find-preresolved-occluded';
3851+
sessionStore.set(sessionName, makeStaleRefSession(sessionName));
3852+
mockDispatch.mockResolvedValue({});
3853+
const preresolved = makeFindPreresolvedTree();
3854+
const blocked = {
3855+
...preresolved.node,
3856+
interactionBlocked: { by: 'overlay', reason: 'covered' as const },
3857+
};
3858+
3859+
const response = await runFindInternalClick(sessionStore, sessionName, {
3860+
findResolvedTarget: true,
3861+
findPreresolvedTarget: { node: blocked, nodes: [preresolved.nodes[0], blocked] },
3862+
});
3863+
3864+
expect(response?.ok).toBe(false);
3865+
if (response && !response.ok) {
3866+
expect(response.error.code).toBe('COMMAND_FAILED');
3867+
expect(response.error.message).toContain('covered by another visible element');
3868+
}
3869+
expect(readPressPoint()).toBeUndefined();
3870+
});
3871+
37403872
test('get text with a pinned stale ref gets the precise warning', async () => {
37413873
const sessionStore = makeSessionStore();
37423874
const sessionName = 'pinned-get-text';

0 commit comments

Comments
 (0)