Skip to content

Commit 19089ee

Browse files
committed
fix(ref): let the frame model report which tree a ref's node was read from (#3471)
`measureRefObservation` re-derived the adoption rule that `reconcileFreshObservation` had already applied: it looked the ref up in the capture again and required the dispatched node to match it on rect and local identity. Two implementations of one ADR 0014 step 5 decision, so the day reconciliation gains or drops a rule, the gate can call a look "measured" on an adoption the rule no longer grants — or refuse one it does grant. `reconcileFreshObservation` now states what it chose, on every branch: `adoptedFrom: 'observation' | 'authorized-frame'`. The value rides the ref's `ResolvedInteractionTarget`, which is where a caller reads the node, rect and `preAction` it dispatched, so the provenance travels with the evidence it describes rather than being inferred from it afterwards. The gate reads that field: a look measured the target exactly when the resolution says it read the dispatched node out of the observation, and the look's surface now comes from that same capture through `surfaceScopedNodes` — the way the selector branch reads its resolution's `preAction` — instead of reaching for `snapshot.iosSystemSurfaceBundleId` directly. The two refusal tests are the ones that pay for this, and they are the reason rect agreement is still not enough: a different control that took over the ref at the frame's own coordinates produces two agreeing rects, and only the reported provenance says the dispatch read them from the authorized frame. Verified by mutating the reported value on each branch: reporting 'authorized-frame' where the observation was adopted reddens `the @ref path takes its own observations, because its resolution spends none`; reporting 'observation' on either keep-the-frame branch reddens `a ref the second observation cannot re-measure is refused, never self-agreed` and `a replacement that took the ref at the frame coordinates is refused, not agreed`.
1 parent 3d8e3ff commit 19089ee

3 files changed

Lines changed: 53 additions & 33 deletions

File tree

‎packages/contracts/src/interaction.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,9 @@ export type SurfaceScopedNodes = {
150150
surfaceBundleId?: string;
151151
};
152152

153+
/** Which tree a resolved `@ref`'s node was read from (ADR 0014 step 5). */
154+
export type RefObservationSource = 'observation' | 'authorized-frame';
155+
153156
export type ResolvedInteractionTarget =
154157
| {
155158
kind: 'point';
@@ -168,6 +171,16 @@ export type ResolvedInteractionTarget =
168171
resolution?: ResolutionDisclosure;
169172
recordingTarget?: RecordingTargetOverride;
170173
preAction?: SurfaceScopedNodes;
174+
/**
175+
* ADR 0014 step 5: which tree the dispatched node was read from, reported by the
176+
* reconciliation that chose it. `'observation'` means the session's newest capture, so the
177+
* node's rect is a position that capture measured; `'authorized-frame'` means the older tree
178+
* the ref was authorized against, whose rect says nothing about where the target sits now.
179+
* Reported rather than re-derived downstream from node or rect identity, so no caller can
180+
* disagree with the rule after the rule has decided. Absent on a node a caller carried in
181+
* (#1654), which no reconciliation chose.
182+
*/
183+
adoptedFrom?: RefObservationSource;
171184
}
172185
| {
173186
kind: 'selector';

‎src/commands/interaction/runtime/moving-target-gate.ts‎

Lines changed: 15 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,11 @@ import {
66
} from '@agent-device/contracts/moving-target-gate';
77
import { INTERACTION_ERROR_REASONS } from '@agent-device/selectors/interaction-error';
88
import type {
9+
InteractionTarget,
910
ResolvedInteractionTarget,
1011
SurfaceScopedNodes,
1112
} from '@agent-device/contracts/interaction';
12-
import type { Rect, SnapshotNode, SnapshotState } from '@agent-device/kernel/snapshot';
13+
import type { Rect, SnapshotNode } from '@agent-device/kernel/snapshot';
1314
import { areCapturedRectsAtSamePosition } from '@agent-device/kernel/rect-center';
1415
import { AppError, discloseDispatch } from '@agent-device/kernel/errors';
1516
import { shellQuoteIfNeeded } from '@agent-device/kernel/device-shell';
@@ -18,15 +19,11 @@ import type {
1819
CommandContext,
1920
CommandSessionRecord,
2021
} from '@agent-device/contracts/runtime-contract';
21-
import {
22-
localIdentitiesEqual,
23-
nodeRectByIdentity,
24-
readNodeLocalIdentity,
25-
} from '@agent-device/ad-script';
22+
import { nodeRectByIdentity } from '@agent-device/ad-script';
2623
import { now, sleep } from '../../runtime-common.ts';
2724
import { captureInteractionSnapshot } from './interaction-snapshot-capture.ts';
28-
import { tryResolveRefNode } from './ref-target-resolution.ts';
29-
import { resolveInteractionTarget, type InteractionTarget } from './resolution.ts';
25+
import { surfaceScopedNodes } from './post-action-surface.ts';
26+
import { resolveInteractionTarget } from './resolution.ts';
3027
import type { ResolveInteractionTargetParams } from './interaction-resolution-request.ts';
3128

3229
/**
@@ -233,38 +230,30 @@ async function reObserveTarget(
233230
}
234231
const { snapshot } = await captureInteractionSnapshot(runtime, options, true);
235232
const resolved = await resolveInteractionTarget(runtime, options, params);
236-
return { resolved, ...measureRefObservation(options.target, snapshot, resolved) };
233+
return { resolved, ...measureRefObservation(surfaceScopedNodes(snapshot), resolved) };
237234
}
238235

239236
/**
240-
* Whether this look measured the target, stated over the two values that survive the session store's
241-
* clone.
237+
* Whether this look measured the target, stated over the provenance the resolution itself reports.
242238
*
243239
* Rect agreement alone is not enough, and this is the hole that makes it so: ADR 0014 keeps the
244240
* authorized frame node when the captured node's identity differs, so a DIFFERENT control that took
245241
* over the ref at the frame's own coordinates would produce two agreeing "observations" while the
246-
* dispatch went to the stale point. The look measured something only when the node it dispatched is
247-
* the node this capture names — same position and same local identity.
242+
* dispatch went to the stale point. The look measured something only when that reconciliation
243+
* REPORTED reading the dispatched node out of the observation — the one account of what got
244+
* dispatched that cannot drift from the rule it applied, whatever the rule becomes. The surface
245+
* comes from the same capture through `surfaceScopedNodes`, the way the selector branch reads its
246+
* resolution's `preAction`.
248247
*/
249248
function measureRefObservation(
250-
target: Extract<InteractionTarget, { kind: 'ref' }>,
251-
snapshot: SnapshotState,
249+
capture: SurfaceScopedNodes,
252250
resolved: ResolvedInteractionTarget,
253251
): MovingTargetMeasurement {
254252
const rect = resolvedRect(resolved);
255-
const dispatchedNode = 'node' in resolved ? resolved.node : undefined;
256-
const inCapture = tryResolveRefNode(snapshot.nodes, target.ref, {
257-
fallbackLabel: target.fallbackLabel ?? '',
258-
});
259-
const captured = inCapture.kind === 'resolved' ? inCapture.resolved.node : undefined;
260253
const measured =
261-
rect !== undefined &&
262-
dispatchedNode !== undefined &&
263-
captured !== undefined &&
264-
areCapturedRectsAtSamePosition(captured.rect, rect) &&
265-
localIdentitiesEqual(readNodeLocalIdentity(captured), readNodeLocalIdentity(dispatchedNode));
254+
rect !== undefined && 'adoptedFrom' in resolved && resolved.adoptedFrom === 'observation';
266255
return measured
267-
? { measured: true, rect, surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
256+
? { measured: true, rect, surfaceBundleId: capture.surfaceBundleId }
268257
: { measured: false, rect: undefined };
269258
}
270259

‎src/commands/interaction/runtime/ref-target-resolution.ts‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import { surfaceScopedNodes } from './post-action-surface.ts';
1919
import type {
2020
InteractionTarget,
2121
PreresolvedInteractionTarget,
22+
RefObservationSource,
2223
ResolvedInteractionTarget,
2324
SurfaceScopedNodes,
2425
} from '@agent-device/contracts/interaction';
@@ -43,6 +44,11 @@ type RefResolution = {
4344
resolved: ResolvedRefNode;
4445
/** The keyboard band the capture of `tree.nodes` measured, when it measured one (#2660). */
4546
keyboard?: SnapshotKeyboardBandFact;
47+
/**
48+
* Which tree `resolved.node` was read from, as the reconciliation decided it (ADR 0014 step 5).
49+
* Absent for a caller-carried node (#1654), which no reconciliation of this module chose.
50+
*/
51+
adoptedFrom?: RefObservationSource;
4652
};
4753

4854
/**
@@ -90,6 +96,7 @@ async function readRefResolution(
9096
return {
9197
tree: surfaceScopedNodes(capture.snapshot),
9298
...(capture.snapshot.keyboard ? { keyboard: capture.snapshot.keyboard } : {}),
99+
...(capture.adoptedFrom ? { adoptedFrom: capture.adoptedFrom } : {}),
93100
resolved: capture.resolved,
94101
};
95102
}
@@ -100,7 +107,7 @@ export async function resolveRefInteractionTarget(
100107
target: Extract<InteractionTarget, { kind: 'ref' }>,
101108
params: ResolveInteractionTargetParams,
102109
): Promise<ResolvedInteractionTarget> {
103-
const { tree, keyboard, resolved } = params.preresolvedTarget
110+
const { tree, keyboard, resolved, adoptedFrom } = params.preresolvedTarget
104111
? adoptPreresolvedRefTarget(target, params.preresolvedTarget)
105112
: await readRefResolution(runtime, options, target);
106113
const nodes = tree.nodes;
@@ -128,6 +135,7 @@ export async function resolveRefInteractionTarget(
128135
kind: 'ref',
129136
point,
130137
target: { kind: 'ref', ref: `@${resolved.ref}` },
138+
...(adoptedFrom ? { adoptedFrom } : {}),
131139
...describeResolvedInteractionNode(
132140
runtime,
133141
visibleNode,
@@ -138,11 +146,18 @@ export async function resolveRefInteractionTarget(
138146
};
139147
}
140148

149+
/** What the frame model decided for one `@ref`: the node to act on, the tree it came from, and why. */
150+
type RefObservationReconciliation = InteractionSnapshot & {
151+
resolved: ResolvedRefNode;
152+
/** Which of the candidate trees `resolved.node` was read from: what the rule chose. */
153+
adoptedFrom: RefObservationSource;
154+
};
155+
141156
async function resolveSnapshotForRef(
142157
runtime: AgentDeviceRuntime,
143158
options: CommandContext,
144159
target: Extract<InteractionTarget, { kind: 'ref' }>,
145-
): Promise<InteractionSnapshot & { resolved: ResolvedRefNode }> {
160+
): Promise<RefObservationReconciliation> {
146161
const { session, snapshot: frameTree } = await requireSnapshotSession(runtime, options.session);
147162

148163
const fallbackLabel = target.fallbackLabel ?? '';
@@ -180,11 +195,14 @@ function reconcileFreshObservation(params: {
180195
target: Extract<InteractionTarget, { kind: 'ref' }>;
181196
fallbackLabel: string;
182197
authorized: ResolvedRefNode;
183-
}): InteractionSnapshot & { resolved: ResolvedRefNode } {
198+
}): RefObservationReconciliation {
184199
const { session, frameTree, target, fallbackLabel, authorized } = params;
185200
const observation = session.snapshot;
186-
if (!observation || observation === frameTree) {
187-
return { snapshot: frameTree, resolved: authorized };
201+
if (!observation) {
202+
return { snapshot: frameTree, resolved: authorized, adoptedFrom: 'authorized-frame' };
203+
}
204+
if (observation === frameTree) {
205+
return { snapshot: frameTree, resolved: authorized, adoptedFrom: 'observation' };
188206
}
189207
const observed = tryResolveRefNode(observation.nodes, target.ref, { fallbackLabel });
190208
if (
@@ -194,9 +212,9 @@ function reconcileFreshObservation(params: {
194212
readNodeLocalIdentity(observed.resolved.node),
195213
)
196214
) {
197-
return { snapshot: observation, resolved: observed.resolved };
215+
return { snapshot: observation, resolved: observed.resolved, adoptedFrom: 'observation' };
198216
}
199-
return { snapshot: frameTree, resolved: authorized };
217+
return { snapshot: frameTree, resolved: authorized, adoptedFrom: 'authorized-frame' };
200218
}
201219

202220
/** The runtime-ref resolver: `exact` for a resolved `@ref`, `label-fallback` for trailing-label recovery. */

0 commit comments

Comments
 (0)