Skip to content

Commit c45f2d6

Browse files
committed
fix(replay): carry the press retarget on a recording-only side channel; harden the guard (#1280 re-review)
Maintainer re-review corrections, four findings: P1a (side channel): the runtime response is now entirely container-based — node, selectorChain, refLabel, point, resolution disclosure, hittability all describe the dispatched container, restoring the response-identity contract. The retarget travels as an optional recordingTarget {node, selectorChain, refLabel} on the runtime result (contracts/interaction.ts), consumed only at the recording boundary (interaction-touch-response.ts): the recorded action entry — the .ad writer's result.selectorChain source — takes the descendant chain/ref-label and recordedTargetCapture feeds the descendant node to computeTargetEvidence, while both wire payloads keep container materials. Daemon-route regression proves response container-based + recorded entry, target-v1 evidence, and the physically written .ad line descendant-based. P1b (fill): removed from retarget scope — a fill chain carries editable=true constraints a label descendant can never satisfy, saving an unreplayable script. click/press only; replay test proves the recorded fill chain on an identity-empty editable container still resolves uniquely. P2a (duplicate container ids): the identity-empty predicate now evaluates from the DEMOTED identity view — dropped the extractNodeText probe whose raw-identifier fallback resurrected an id that had been demoted for non-uniqueness, which made duplicated-container-id rows skip the retarget they need most. Fixture proves retarget fires; unique-id contrast unchanged. P2b (guard): replaced the private role-fragment list with the canonical interactive classification — isSemanticTouchTarget (exported from core/interaction-targeting.ts, the same policy hittable-ancestor promotion uses) plus the hittable flag; the module moves to src/core/press-retarget.ts since selectors -> core would be a layering back-edge. Added the geometric containment condition: the selected descendant's rect center must lie inside the container's rect (missing rects fail closed) — the replay tap point must be provably within the original activation region. Tests: nested Cell (role the old list missed) blocks; out-of-bounds descendant blocks; rect-less container blocks. ADR 0012 decision-3 amendment rewritten to the side-channel design, click/press-only scope, demoted-view rule, and both guard halves. The daemon regression runs on the iOS runtime path (direct-iOS is recording-gated) so the unit lane spends no real wall-clock on Android adb dialog probes.
1 parent 49aaadf commit c45f2d6

12 files changed

Lines changed: 567 additions & 183 deletions

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 31 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,13 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig
2424
(a shared Android framework resource id such as `android:id/title`, or a reused RN `FlatList` `testID`)
2525
falls back to role+label in both `computeTargetEvidence`'s `target-v1` tuple and
2626
`buildSelectorChainForNode`'s chain (#1269).
27-
- Decision 3 amendment, record-time press retarget to a labeled descendant — a press/click/fill whose
28-
resolved winner is an identity-empty container (no surviving id, no label, no value/text) is recorded
29-
against its first labeled descendant instead, so both writers key off a node with selective identity; a
30-
container whose subtree holds another interactive/hittable node is recorded unchanged (#1280).
27+
- Decision 3 amendment, record-time press retarget to a labeled descendant — a click/press whose
28+
resolved winner is an identity-empty container (no surviving id, no label, no value, from the demoted
29+
identity view) is recorded against its first labeled descendant via a recording-only side channel, so
30+
both writers key off a node with selective identity while the live response stays container-based
31+
end-to-end; a container whose subtree holds a competing interactive node (canonical classification or
32+
hittable flag), or whose selected descendant's center falls outside the container rect, is recorded
33+
unchanged; `fill` is excluded (#1280).
3134

3235
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
3336
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -333,25 +336,30 @@ A recorded `id` never matches a node without that id.
333336
> case, but the rule is capture-time uniqueness, not an `android:id/*` namespace heuristic: a reused RN
334337
> `FlatList` `testID` hits the same demotion on iOS.
335338
336-
> **Amendment (#1280).** A press/click/fill whose resolved winner is an *identity-empty* container — no
337-
> id survives #1269's demotion (absent, or demoted for being non-unique), no label, and no value/text
338-
> under the same normalization `buildSelectorChainForNode` uses — is recorded against its first labeled
339-
> descendant instead of the container itself, so the recorded selector chain and the `target-v1` tuple
340-
> both carry a selective role+label identity rather than a bare, tree-wide-shared role. The measured case
341-
> is Android: a list row's clickable node is a label-less `role="linearlayout"` container whose visible
342-
> title lives on a child `TextView`. "First labeled descendant" means the first node in document order
343-
> (this section's canonical total order) within the container's whole subtree whose normalized label is
344-
> non-empty. The substitution is guarded fail-closed: it fires only when that subtree contains **no other**
345-
> interactive/hittable node besides the container itself — a row with a trailing `Switch` or `Checkbox`
346-
> must not retarget, because a tap at the labeled descendant's center and a tap at the container's center
347-
> could land on different controls once the descendant, not the container, is what replay taps. A
348-
> container the guard blocks, or for which no descendant carries a label, is recorded exactly as before
349-
> this amendment. This is a **recording-time** substitution only — `resolveSelectorChain` and live
350-
> press/fill dispatch are unchanged; the action still taps the point resolved against the original
351-
> container. Only the recording-coupled fields retarget (the selector chain, the `target-v1` evidence
352-
> source, and the recorded ref-label); response-semantic disclosure — the `targetHittable`/hint
353-
> annotation — keeps describing the container actually dispatched, so a hittable row whose title
354-
> `TextView` is non-hittable never reports a false `targetHittable: false`. The rule is
339+
> **Amendment (#1280).** A **click/press** whose resolved winner is an *identity-empty* container — no
340+
> id survives #1269's demotion (absent, or demoted for being non-unique), no label, and no value, all
341+
> evaluated from the demoted identity view (a raw identifier that did not survive demotion never counts
342+
> as identity or text) — is **recorded** against its first labeled descendant instead of the container
343+
> itself, so the recorded selector chain and the `target-v1` tuple both carry a selective role+label
344+
> identity rather than a bare, tree-wide-shared role. The measured case is Android: a list row's
345+
> clickable node is a label-less `role="linearlayout"` container whose visible title lives on a child
346+
> `TextView`. "First labeled descendant" means the first node in document order (this section's
347+
> canonical total order) within the container's whole subtree whose normalized label is non-empty.
348+
> `fill` is deliberately excluded: its recorded chain carries `editable=true` constraints a label
349+
> descendant cannot satisfy, which would save an unreplayable script. The substitution is guarded
350+
> fail-closed, twice over: (i) it fires only when the subtree contains **no competing interactive node**
351+
> — none flagged `hittable`, and none whose role the canonical interactive-role classification
352+
> (`isSemanticTouchTarget`, the same policy hittable-ancestor promotion uses) names as an independently
353+
> tappable control — a row with a trailing `Switch` or `Checkbox` must not retarget, because a tap at
354+
> the labeled descendant's center and a tap at the container's center could land on different controls
355+
> once the descendant, not the container, is what replay taps; and (ii) the selected descendant's rect
356+
> center must lie **inside the container's rect** — the replay tap point must be provably within the
357+
> original activation region (missing rects fail closed). A container either guard blocks, or for which
358+
> no descendant carries a label, is recorded exactly as before this amendment. This is a
359+
> **recording-only side channel**: the live response — resolved node, selector chain, ref-label,
360+
> tap point, resolution disclosure, and `targetHittable`/hint — describes the dispatched container
361+
> end-to-end, and `resolveSelectorChain`/dispatch are unchanged; only the recorded action entry (the
362+
> `.ad` writer's source) and its `target-v1` evidence consume the retargeted descendant. The rule is
355363
> platform-agnostic: an RN `FlatList` row (`Cell`) that is label-less with a labeled `Text` child hits
356364
> the same substitution on iOS.
357365

‎src/commands/interaction/runtime/resolution.test.ts‎

Lines changed: 55 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { BackendSnapshotOptions } from '../../../backend.ts';
44
import { ref, selector } from './selector-read.ts';
55
import { resolveActionableTouchResolution } from '../../../core/interaction-targeting.ts';
66
import { tryResolveRefNode } from './resolution.ts';
7+
import { parseSelectorChain, resolveSelectorChain } from '../../../selectors/index.ts';
78
import { makeSnapshotState } from '../../../__tests__/test-utils/index.ts';
89
import type { Point } from '../../../kernel/snapshot.ts';
910
import {
@@ -231,15 +232,10 @@ test('runtime press omits targetHittable and hint when the resolved node is hitt
231232
assert.equal(result.hint, undefined);
232233
});
233234

234-
test('runtime press #1280 retarget: recording fields follow the labeled descendant, response hittability stays on the dispatched container', async () => {
235-
// The #1280 measured shape: a hittable, identity-empty LinearLayout row
236-
// container whose title lives on a NON-hittable TextView child. The tap
237-
// and the response-semantic fields (targetHittable/hint) must describe
238-
// the container actually dispatched — never a false `targetHittable:
239-
// false` from the descendant — while the recording-coupled fields
240-
// (node-as-evidence-source, selectorChain, refLabel) follow the
241-
// retargeted descendant.
242-
const snapshot = makeSnapshotState([
235+
// The #1280 measured shape: a hittable, identity-empty LinearLayout row
236+
// container whose title lives on a NON-hittable TextView child.
237+
function identityEmptyRowSnapshot(containerType = 'LinearLayout') {
238+
return makeSnapshotState([
243239
{
244240
index: 0,
245241
depth: 0,
@@ -251,7 +247,7 @@ test('runtime press #1280 retarget: recording fields follow the labeled descenda
251247
index: 1,
252248
depth: 1,
253249
parentIndex: 0,
254-
type: 'LinearLayout',
250+
type: containerType,
255251
rect: { x: 0, y: 100, width: 300, height: 48 },
256252
hittable: true,
257253
},
@@ -265,8 +261,11 @@ test('runtime press #1280 retarget: recording fields follow the labeled descenda
265261
hittable: false,
266262
},
267263
]);
264+
}
265+
266+
test('runtime press #1280 retarget: the response is entirely container-based; the descendant rides only the recordingTarget side channel', async () => {
268267
const calls: Point[] = [];
269-
const device = createInteractionDevice(snapshot, {
268+
const device = createInteractionDevice(identityEmptyRowSnapshot(), {
270269
platform: 'android',
271270
tap: async (_context, point) => {
272271
calls.push(point);
@@ -280,17 +279,54 @@ test('runtime press #1280 retarget: recording fields follow the labeled descenda
280279
// Dispatch: the tap lands at the CONTAINER's center.
281280
assert.deepEqual(calls, [{ x: 150, y: 124 }]);
282281
assert.equal(result.kind, 'selector');
283-
// Recording-coupled fields: the retargeted descendant.
284-
assert.equal(result.node?.label, 'Connected devices');
285-
assert.deepEqual(result.selectorChain, [
282+
// The whole runtime response describes the dispatched container — node,
283+
// chain, hittability. The descendant's `hittable: false` must not leak.
284+
assert.equal(result.node?.type, 'LinearLayout');
285+
assert.deepEqual(result.selectorChain, ['role="linearlayout"']);
286+
assert.equal(result.targetHittable, undefined);
287+
assert.equal(result.hint, undefined);
288+
// The retarget rides ONLY on the recording-only side channel.
289+
assert.equal(result.recordingTarget?.node.label, 'Connected devices');
290+
assert.deepEqual(result.recordingTarget?.selectorChain, [
286291
'role="textview" label="Connected devices"',
287292
'label="Connected devices"',
288293
]);
289-
assert.equal(result.refLabel, 'Connected devices');
290-
// Response-semantic fields: the container is hittable, so no
291-
// targetHittable/hint — the descendant's `hittable: false` must not leak.
292-
assert.equal(result.targetHittable, undefined);
293-
assert.equal(result.hint, undefined);
294+
assert.equal(result.recordingTarget?.refLabel, 'Connected devices');
295+
});
296+
297+
test('runtime fill #1280: fill is excluded from retargeting — the chain stays on the editable container and resolves for replay', async () => {
298+
// An identity-empty EDITABLE container (no id/label/value) with a labeled
299+
// non-editable TextView child. Retargeting a fill would record a chain
300+
// whose `editable=true` constraint the label descendant can never satisfy
301+
// — an unreplayable script — so fill must record as before, no retarget.
302+
const snapshot = identityEmptyRowSnapshot('EditText');
303+
const calls: Array<{ point: Point; text: string }> = [];
304+
const device = createInteractionDevice(snapshot, {
305+
platform: 'android',
306+
fill: async (_context, point, text) => {
307+
calls.push({ point, text });
308+
},
309+
});
310+
311+
const result = await device.interactions.fill(selector('role=edittext'), 'hello', {
312+
session: 'default',
313+
});
314+
315+
assert.equal(result.kind, 'selector');
316+
assert.deepEqual(calls, [{ point: { x: 150, y: 124 }, text: 'hello' }]);
317+
// No side channel: fill never retargets.
318+
assert.equal(result.recordingTarget, undefined);
319+
// The recorded chain belongs to the container and carries the editable
320+
// constraint...
321+
assert.deepEqual(result.selectorChain, ['role="edittext" editable=true']);
322+
// ...and it resolves back to the editable container on the record-time
323+
// tree — the saved script stays replayable.
324+
const resolved = resolveSelectorChain(
325+
snapshot.nodes,
326+
parseSelectorChain(result.selectorChain!.join(' || ')),
327+
{ platform: 'android', requireRect: true, requireUnique: true },
328+
);
329+
assert.equal(resolved?.node.type, 'EditText');
294330
});
295331

296332
test('runtime fill surfaces targetHittable and a hint for a non-hittable selector match (Maps pin case, #1037)', async () => {

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 41 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ import {
1616
type SelectorResolution,
1717
} from '../../../selectors/index.ts';
1818
import { buildSelectorChainForNode } from '../../../selectors/build.ts';
19-
import { resolvePressRecordingTarget } from '../../../selectors/press-retarget.ts';
19+
import { resolvePressRecordingTarget } from '../../../core/press-retarget.ts';
2020
import {
2121
findNodeByLabel,
2222
normalizeType,
@@ -32,6 +32,7 @@ import { truncateUtf8 } from '../../../utils/truncate-utf8.ts';
3232
import type {
3333
InteractionTarget,
3434
PointTarget,
35+
RecordingTargetOverride,
3536
ResolutionDiagnosticEntry,
3637
ResolutionDisclosure,
3738
ResolvedInteractionTarget,
@@ -383,15 +384,10 @@ function buildResolutionDiagnosticEntry(
383384
};
384385
}
385386

386-
// #1280: press/click/fill are the actions whose recorded evidence can
387-
// substitute an identity-empty container for its labeled descendant (a
388-
// list-row press is the measured shape); reads (get/find) already have
389-
// their own #1269 id-demotion treatment and go through selector-read.ts,
390-
// not this function.
391-
const PRESS_RETARGET_ACTIONS = new Set<InteractionAction>(['click', 'press', 'fill']);
392-
393387
// Shared tail of a resolved ref/selector interaction target: the node itself
394-
// plus everything derived from it for the response.
388+
// plus everything derived from it for the response. Every response field
389+
// describes the DISPATCHED node — the #1280 retarget rides only on the
390+
// `recordingTarget` side channel below.
395391
function describeResolvedInteractionNode(
396392
runtime: AgentDeviceRuntime,
397393
node: SnapshotNode,
@@ -406,33 +402,50 @@ function describeResolvedInteractionNode(
406402
hint?: string;
407403
preActionNodes: SnapshotState['nodes'];
408404
resolution: ResolutionDisclosure;
405+
recordingTarget?: RecordingTargetOverride;
409406
} {
410-
// #1280: the ONE substitution point feeding both writers — the returned
411-
// `node` is what `computeTargetEvidence` records evidence for
412-
// (`recordedTargetCapture`, interaction-touch-response.ts), and
413-
// `selectorChain` below is built from the same `recordedNode`, so the two
414-
// never half-retarget. The live tap point was already resolved against
415-
// the ORIGINAL `node` before this function runs (see callers) — this is
416-
// recording-time only. The fields split by what they are FOR:
417-
// recording-coupled fields (they become the .ad step: `node` as evidence
418-
// source, `selectorChain`, `refLabel`) follow the retargeted node;
419-
// response-semantic fields (`describeNonHittableTarget`'s targetHittable +
420-
// hint) describe the node actually dispatched and stay on the original —
421-
// a hittable container's press must never report its non-hittable title
422-
// descendant as the tap target.
423-
const recordedNode = PRESS_RETARGET_ACTIONS.has(action)
424-
? resolvePressRecordingTarget(node, nodes)
425-
: node;
426407
return {
427-
node: recordedNode,
428-
selectorChain: buildSelectorChainForNode(recordedNode, runtime.backend.platform, {
408+
node,
409+
selectorChain: buildSelectorChainForNode(node, runtime.backend.platform, {
429410
action: action === 'fill' ? 'fill' : 'click',
430411
nodes,
431412
}),
432-
refLabel: resolveRefLabel(recordedNode, nodes),
413+
refLabel: resolveRefLabel(node, nodes),
433414
...describeNonHittableTarget(node, action),
434415
preActionNodes: nodes,
435416
resolution,
417+
...pressRecordingTargetOverride(runtime, node, nodes, action),
418+
};
419+
}
420+
421+
/**
422+
* #1280 (ADR 0012 decision 3 amendment): the recording-only side channel.
423+
* When a click/press resolves to an identity-empty container, the RECORDED
424+
* step retargets to its first labeled descendant — node, chain, and
425+
* ref-label computed together here so the recorded action entry and its
426+
* `target-v1` evidence can never half-retarget. The response payloads never
427+
* consume this (see `interaction-touch-response.ts`). `fill` is deliberately
428+
* excluded: its chain carries `editable=true` constraints a label descendant
429+
* cannot satisfy, which would record an unreplayable selector.
430+
*/
431+
function pressRecordingTargetOverride(
432+
runtime: AgentDeviceRuntime,
433+
node: SnapshotNode,
434+
nodes: SnapshotState['nodes'],
435+
action: InteractionAction,
436+
): { recordingTarget?: RecordingTargetOverride } {
437+
if (action !== 'click' && action !== 'press') return {};
438+
const recordingNode = resolvePressRecordingTarget(node, nodes);
439+
if (recordingNode === node) return {};
440+
return {
441+
recordingTarget: {
442+
node: recordingNode,
443+
selectorChain: buildSelectorChainForNode(recordingNode, runtime.backend.platform, {
444+
action: 'click',
445+
nodes,
446+
}),
447+
refLabel: resolveRefLabel(recordingNode, nodes),
448+
},
436449
};
437450
}
438451

‎src/contracts/interaction.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,21 @@ export type ResolutionDisclosure =
6969
| { source: 'ref'; phase: 'pre-action'; kind: 'label-fallback' }
7070
| { source: 'direct-ios'; kind: 'not-observed' };
7171

72+
/**
73+
* #1280 (ADR 0012 decision 3 amendment): recording-only side channel — the
74+
* labeled descendant an identity-empty press container was retargeted to.
75+
* Consumed exclusively at the recording boundary
76+
* (`interaction-touch-response.ts`): the recorded action entry and its
77+
* `target-v1` evidence key off this node/chain/ref-label, while every
78+
* response payload keeps describing the dispatched container. Never
79+
* serialized into a response.
80+
*/
81+
export type RecordingTargetOverride = {
82+
node: SnapshotNode;
83+
selectorChain: string[];
84+
refLabel?: string;
85+
};
86+
7287
export type ResolvedInteractionTarget =
7388
| {
7489
kind: 'point';
@@ -86,6 +101,7 @@ export type ResolvedInteractionTarget =
86101
hint?: string;
87102
preActionNodes?: SnapshotNode[];
88103
resolution?: ResolutionDisclosure;
104+
recordingTarget?: RecordingTargetOverride;
89105
}
90106
| {
91107
kind: 'selector';
@@ -98,6 +114,7 @@ export type ResolvedInteractionTarget =
98114
hint?: string;
99115
preActionNodes?: SnapshotNode[];
100116
resolution?: ResolutionDisclosure;
117+
recordingTarget?: RecordingTargetOverride;
101118
};
102119

103120
/**

‎src/core/interaction-targeting.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,14 @@ function findPreferredActionableDescendant(
104104
return current === node ? null : current;
105105
}
106106

107-
function isSemanticTouchTarget(node: SnapshotNode): boolean {
107+
/**
108+
* THE canonical interactive-role classification for touch: a node whose
109+
* type/role/subrole names a control that independently receives taps. Shared
110+
* by the hittable-ancestor promotion above and #1280's press-retarget
111+
* competing-descendant guard (`press-retarget.ts`) — one list, never a
112+
* parallel copy.
113+
*/
114+
export function isSemanticTouchTarget(node: SnapshotNode): boolean {
108115
const roles = [node.type, node.role, node.subrole].map((value) => normalizeType(value ?? ''));
109116
return roles.some(isSemanticTouchRole);
110117
}

0 commit comments

Comments
 (0)