Skip to content

Commit 1fdbf80

Browse files
authored
fix(replay): retarget identity-empty press containers to their labeled descendant (#1280) (#1286)
* refactor(replay): share the id-demotion predicate via target-identity-node Extract session-target-evidence.ts's demoteNonUniqueId into a shared demoteNonUniqueLocalIdentity (target-identity-node.ts), and export build.ts's normalizeSelectorText. Both become shared building blocks a third call site (#1280's press-retarget identity-empty check) reuses instead of re-deriving the id-demotion rule and value/text normalization a third way. No behavior change. * fix(replay): retarget identity-empty press containers to their labeled descendant (#1280) Android list-row presses target a clickable container (role="linearlayout") with no id, no label, no value/text — its title lives on a labeled descendant (the android:id/title TextView, whose own id #1272 already demotes for being non-unique). The container's identity is role-only and shared by every row, so replay disambiguates positionally and mis-binds under reorder (measured matchCount 12, 20/20 identity-mismatch). Retarget at record time: when a press/click/fill resolves to an identity-empty container (rule 1), substitute its first labeled descendant in document order (rule 2), but only when the container's subtree has no other interactive/hittable node (rule 3, fail-closed — a trailing Switch/Checkbox must not retarget, since a tap at the descendant's center vs the container's could land on different controls). Guard-blocked or label-less subtrees record exactly as today. Implemented once at the single recording choke point (describeResolvedInteractionNode, resolution.ts): the returned node feeds BOTH buildSelectorChainForNode's chain and (downstream, via recordedTargetCapture) computeTargetEvidence, so the two writers can never half-retarget. Recording-time only — resolveSelectorChain and live press/fill dispatch are unchanged; the tap point is already fixed against the original container before this substitution runs. Adds an ADR 0012 decision 3 amendment (mirroring #1269's), a press-retarget unit/guard/cross-invariant suite (including an RN FlatList iOS parity fixture), and a reorder+insert e2e proving the retargeted recording rebinds by role+label where the un-retargeted container recording refuses. * fix(replay): keep response hittability on the dispatched container, not the retargeted descendant Review blocker on #1286 (flag 1 adjudicated): describeResolvedInteractionNode was computing describeNonHittableTarget from the retargeted descendant, so every retargeted press on a non-hittable title TextView would emit a false `targetHittable: false` + misleading hint on the exact happy path the fix serves — a live-response regression violating the design's recording-time-only rule. Split the fields by what they are FOR: recording-coupled fields (node as evidence source, selectorChain, refLabel — they become the .ad step) keep following the retargeted descendant; the response-semantic describeNonHittableTarget (targetHittable + hint) reverts to the original node, describing what was actually dispatched. Documented in the function comment and the ADR amendment; new load-bearing test (fails against the pre-fix line): a hittable container with a non-hittable labeled child presses with no targetHittable/hint while chain/evidence/refLabel belong to the descendant. * fix(replay): carry the press retarget on a recording-only side channel; harden the guard (#1280 re-review) Maintainer re-review corrections, four findings: P1a (side channel): the runtime response is now entirely container-based — node, selectorChain, refLabel, point, resolution disclosure, hittability all describe the dispatched container, restoring the response-identity contract. The retarget travels as an optional recordingTarget {node, selectorChain, refLabel} on the runtime result (contracts/interaction.ts), consumed only at the recording boundary (interaction-touch-response.ts): the recorded action entry — the .ad writer's result.selectorChain source — takes the descendant chain/ref-label and recordedTargetCapture feeds the descendant node to computeTargetEvidence, while both wire payloads keep container materials. Daemon-route regression proves response container-based + recorded entry, target-v1 evidence, and the physically written .ad line descendant-based. P1b (fill): removed from retarget scope — a fill chain carries editable=true constraints a label descendant can never satisfy, saving an unreplayable script. click/press only; replay test proves the recorded fill chain on an identity-empty editable container still resolves uniquely. P2a (duplicate container ids): the identity-empty predicate now evaluates from the DEMOTED identity view — dropped the extractNodeText probe whose raw-identifier fallback resurrected an id that had been demoted for non-uniqueness, which made duplicated-container-id rows skip the retarget they need most. Fixture proves retarget fires; unique-id contrast unchanged. P2b (guard): replaced the private role-fragment list with the canonical interactive classification — isSemanticTouchTarget (exported from core/interaction-targeting.ts, the same policy hittable-ancestor promotion uses) plus the hittable flag; the module moves to src/core/press-retarget.ts since selectors -> core would be a layering back-edge. Added the geometric containment condition: the selected descendant's rect center must lie inside the container's rect (missing rects fail closed) — the replay tap point must be provably within the original activation region. Tests: nested Cell (role the old list missed) blocks; out-of-bounds descendant blocks; rect-less container blocks. ADR 0012 decision-3 amendment rewritten to the side-channel design, click/press-only scope, demoted-view rule, and both guard halves. The daemon regression runs on the iOS runtime path (direct-iOS is recording-gated) so the unit lane spends no real wall-clock on Android adb dialog probes.
1 parent 991e723 commit 1fdbf80

13 files changed

Lines changed: 1047 additions & 16 deletions

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,13 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig
2424
(a shared Android framework resource id such as `android:id/title`, or a reused RN `FlatList` `testID`)
2525
falls back to role+label in both `computeTargetEvidence`'s `target-v1` tuple and
2626
`buildSelectorChainForNode`'s chain (#1269).
27+
- Decision 3 amendment, record-time press retarget to a labeled descendant — a click/press whose
28+
resolved winner is an identity-empty container (no surviving id, no label, no value, from the demoted
29+
identity view) is recorded against its first labeled descendant via a recording-only side channel, so
30+
both writers key off a node with selective identity while the live response stays container-based
31+
end-to-end; a container whose subtree holds a competing interactive node (canonical classification or
32+
hittable flag), or whose selected descendant's center falls outside the container rect, is recorded
33+
unchanged; `fill` is excluded (#1280).
2734

2835
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
2936
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -329,6 +336,33 @@ A recorded `id` never matches a node without that id.
329336
> case, but the rule is capture-time uniqueness, not an `android:id/*` namespace heuristic: a reused RN
330337
> `FlatList` `testID` hits the same demotion on iOS.
331338
339+
> **Amendment (#1280).** A **click/press** whose resolved winner is an *identity-empty* container — no
340+
> id survives #1269's demotion (absent, or demoted for being non-unique), no label, and no value, all
341+
> evaluated from the demoted identity view (a raw identifier that did not survive demotion never counts
342+
> as identity or text) — is **recorded** against its first labeled descendant instead of the container
343+
> itself, so the recorded selector chain and the `target-v1` tuple both carry a selective role+label
344+
> identity rather than a bare, tree-wide-shared role. The measured case is Android: a list row's
345+
> clickable node is a label-less `role="linearlayout"` container whose visible title lives on a child
346+
> `TextView`. "First labeled descendant" means the first node in document order (this section's
347+
> canonical total order) within the container's whole subtree whose normalized label is non-empty.
348+
> `fill` is deliberately excluded: its recorded chain carries `editable=true` constraints a label
349+
> descendant cannot satisfy, which would save an unreplayable script. The substitution is guarded
350+
> fail-closed, twice over: (i) it fires only when the subtree contains **no competing interactive node**
351+
> — none flagged `hittable`, and none whose role the canonical interactive-role classification
352+
> (`isSemanticTouchTarget`, the same policy hittable-ancestor promotion uses) names as an independently
353+
> tappable control — a row with a trailing `Switch` or `Checkbox` must not retarget, because a tap at
354+
> the labeled descendant's center and a tap at the container's center could land on different controls
355+
> once the descendant, not the container, is what replay taps; and (ii) the selected descendant's rect
356+
> center must lie **inside the container's rect** — the replay tap point must be provably within the
357+
> original activation region (missing rects fail closed). A container either guard blocks, or for which
358+
> no descendant carries a label, is recorded exactly as before this amendment. This is a
359+
> **recording-only side channel**: the live response — resolved node, selector chain, ref-label,
360+
> tap point, resolution disclosure, and `targetHittable`/hint — describes the dispatched container
361+
> end-to-end, and `resolveSelectorChain`/dispatch are unchanged; only the recorded action entry (the
362+
> `.ad` writer's source) and its `target-v1` evidence consume the retargeted descendant. The rule is
363+
> platform-agnostic: an RN `FlatList` row (`Cell`) that is label-less with a labeled `Text` child hits
364+
> the same substitution on iOS.
365+
332366
**Ancestry.** The chain is the nearest **K = 8** ancestors of the target, ordered **leaf→root** (nearest
333367
ancestor first), each entry `{ role, label? }` under the same normalization (`role` may be the empty
334368
string when the node has no type; `label` is omitted when empty). Truncation drops entries from the

‎src/commands/interaction/runtime/resolution.test.ts‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { BackendSnapshotOptions } from '../../../backend.ts';
44
import { ref, selector } from './selector-read.ts';
55
import { resolveActionableTouchResolution } from '../../../core/interaction-targeting.ts';
66
import { tryResolveRefNode } from './resolution.ts';
7+
import { parseSelectorChain, resolveSelectorChain } from '../../../selectors/index.ts';
78
import { makeSnapshotState } from '../../../__tests__/test-utils/index.ts';
89
import type { Point } from '../../../kernel/snapshot.ts';
910
import {
@@ -231,6 +232,103 @@ test('runtime press omits targetHittable and hint when the resolved node is hitt
231232
assert.equal(result.hint, undefined);
232233
});
233234

235+
// The #1280 measured shape: a hittable, identity-empty LinearLayout row
236+
// container whose title lives on a NON-hittable TextView child.
237+
function identityEmptyRowSnapshot(containerType = 'LinearLayout') {
238+
return makeSnapshotState([
239+
{
240+
index: 0,
241+
depth: 0,
242+
type: 'FrameLayout',
243+
rect: { x: 0, y: 0, width: 400, height: 800 },
244+
hittable: true,
245+
},
246+
{
247+
index: 1,
248+
depth: 1,
249+
parentIndex: 0,
250+
type: containerType,
251+
rect: { x: 0, y: 100, width: 300, height: 48 },
252+
hittable: true,
253+
},
254+
{
255+
index: 2,
256+
depth: 2,
257+
parentIndex: 1,
258+
type: 'TextView',
259+
label: 'Connected devices',
260+
rect: { x: 0, y: 100, width: 300, height: 48 },
261+
hittable: false,
262+
},
263+
]);
264+
}
265+
266+
test('runtime press #1280 retarget: the response is entirely container-based; the descendant rides only the recordingTarget side channel', async () => {
267+
const calls: Point[] = [];
268+
const device = createInteractionDevice(identityEmptyRowSnapshot(), {
269+
platform: 'android',
270+
tap: async (_context, point) => {
271+
calls.push(point);
272+
},
273+
});
274+
275+
const result = await device.interactions.press(selector('role=linearlayout'), {
276+
session: 'default',
277+
});
278+
279+
// Dispatch: the tap lands at the CONTAINER's center.
280+
assert.deepEqual(calls, [{ x: 150, y: 124 }]);
281+
assert.equal(result.kind, 'selector');
282+
// The whole runtime response describes the dispatched container — node,
283+
// chain, hittability. The descendant's `hittable: false` must not leak.
284+
assert.equal(result.node?.type, 'LinearLayout');
285+
assert.deepEqual(result.selectorChain, ['role="linearlayout"']);
286+
assert.equal(result.targetHittable, undefined);
287+
assert.equal(result.hint, undefined);
288+
// The retarget rides ONLY on the recording-only side channel.
289+
assert.equal(result.recordingTarget?.node.label, 'Connected devices');
290+
assert.deepEqual(result.recordingTarget?.selectorChain, [
291+
'role="textview" label="Connected devices"',
292+
'label="Connected devices"',
293+
]);
294+
assert.equal(result.recordingTarget?.refLabel, 'Connected devices');
295+
});
296+
297+
test('runtime fill #1280: fill is excluded from retargeting — the chain stays on the editable container and resolves for replay', async () => {
298+
// An identity-empty EDITABLE container (no id/label/value) with a labeled
299+
// non-editable TextView child. Retargeting a fill would record a chain
300+
// whose `editable=true` constraint the label descendant can never satisfy
301+
// — an unreplayable script — so fill must record as before, no retarget.
302+
const snapshot = identityEmptyRowSnapshot('EditText');
303+
const calls: Array<{ point: Point; text: string }> = [];
304+
const device = createInteractionDevice(snapshot, {
305+
platform: 'android',
306+
fill: async (_context, point, text) => {
307+
calls.push({ point, text });
308+
},
309+
});
310+
311+
const result = await device.interactions.fill(selector('role=edittext'), 'hello', {
312+
session: 'default',
313+
});
314+
315+
assert.equal(result.kind, 'selector');
316+
assert.deepEqual(calls, [{ point: { x: 150, y: 124 }, text: 'hello' }]);
317+
// No side channel: fill never retargets.
318+
assert.equal(result.recordingTarget, undefined);
319+
// The recorded chain belongs to the container and carries the editable
320+
// constraint...
321+
assert.deepEqual(result.selectorChain, ['role="edittext" editable=true']);
322+
// ...and it resolves back to the editable container on the record-time
323+
// tree — the saved script stays replayable.
324+
const resolved = resolveSelectorChain(
325+
snapshot.nodes,
326+
parseSelectorChain(result.selectorChain!.join(' || ')),
327+
{ platform: 'android', requireRect: true, requireUnique: true },
328+
);
329+
assert.equal(resolved?.node.type, 'EditText');
330+
});
331+
234332
test('runtime fill surfaces targetHittable and a hint for a non-hittable selector match (Maps pin case, #1037)', async () => {
235333
const calls: Array<{ point: Point; text: string }> = [];
236334
const device = createInteractionDevice(mapPinAnnotationSnapshot(), {

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
type SelectorResolution,
1717
} from '../../../selectors/index.ts';
1818
import { buildSelectorChainForNode } from '../../../selectors/build.ts';
19+
import { resolvePressRecordingTarget } from '../../../core/press-retarget.ts';
1920
import {
2021
findNodeByLabel,
2122
normalizeType,
@@ -31,6 +32,7 @@ import { truncateUtf8 } from '../../../utils/truncate-utf8.ts';
3132
import type {
3233
InteractionTarget,
3334
PointTarget,
35+
RecordingTargetOverride,
3436
ResolutionDiagnosticEntry,
3537
ResolutionDisclosure,
3638
ResolvedInteractionTarget,
@@ -383,7 +385,9 @@ function buildResolutionDiagnosticEntry(
383385
}
384386

385387
// Shared tail of a resolved ref/selector interaction target: the node itself
386-
// plus everything derived from it for the response.
388+
// plus everything derived from it for the response. Every response field
389+
// describes the DISPATCHED node — the #1280 retarget rides only on the
390+
// `recordingTarget` side channel below.
387391
function describeResolvedInteractionNode(
388392
runtime: AgentDeviceRuntime,
389393
node: SnapshotNode,
@@ -398,6 +402,7 @@ function describeResolvedInteractionNode(
398402
hint?: string;
399403
preActionNodes: SnapshotState['nodes'];
400404
resolution: ResolutionDisclosure;
405+
recordingTarget?: RecordingTargetOverride;
401406
} {
402407
return {
403408
node,
@@ -409,6 +414,38 @@ function describeResolvedInteractionNode(
409414
...describeNonHittableTarget(node, action),
410415
preActionNodes: nodes,
411416
resolution,
417+
...pressRecordingTargetOverride(runtime, node, nodes, action),
418+
};
419+
}
420+
421+
/**
422+
* #1280 (ADR 0012 decision 3 amendment): the recording-only side channel.
423+
* When a click/press resolves to an identity-empty container, the RECORDED
424+
* step retargets to its first labeled descendant — node, chain, and
425+
* ref-label computed together here so the recorded action entry and its
426+
* `target-v1` evidence can never half-retarget. The response payloads never
427+
* consume this (see `interaction-touch-response.ts`). `fill` is deliberately
428+
* excluded: its chain carries `editable=true` constraints a label descendant
429+
* cannot satisfy, which would record an unreplayable selector.
430+
*/
431+
function pressRecordingTargetOverride(
432+
runtime: AgentDeviceRuntime,
433+
node: SnapshotNode,
434+
nodes: SnapshotState['nodes'],
435+
action: InteractionAction,
436+
): { recordingTarget?: RecordingTargetOverride } {
437+
if (action !== 'click' && action !== 'press') return {};
438+
const recordingNode = resolvePressRecordingTarget(node, nodes);
439+
if (recordingNode === node) return {};
440+
return {
441+
recordingTarget: {
442+
node: recordingNode,
443+
selectorChain: buildSelectorChainForNode(recordingNode, runtime.backend.platform, {
444+
action: 'click',
445+
nodes,
446+
}),
447+
refLabel: resolveRefLabel(recordingNode, nodes),
448+
},
412449
};
413450
}
414451

‎src/contracts/interaction.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,21 @@ export type ResolutionDisclosure =
6969
| { source: 'ref'; phase: 'pre-action'; kind: 'label-fallback' }
7070
| { source: 'direct-ios'; kind: 'not-observed' };
7171

72+
/**
73+
* #1280 (ADR 0012 decision 3 amendment): recording-only side channel — the
74+
* labeled descendant an identity-empty press container was retargeted to.
75+
* Consumed exclusively at the recording boundary
76+
* (`interaction-touch-response.ts`): the recorded action entry and its
77+
* `target-v1` evidence key off this node/chain/ref-label, while every
78+
* response payload keeps describing the dispatched container. Never
79+
* serialized into a response.
80+
*/
81+
export type RecordingTargetOverride = {
82+
node: SnapshotNode;
83+
selectorChain: string[];
84+
refLabel?: string;
85+
};
86+
7287
export type ResolvedInteractionTarget =
7388
| {
7489
kind: 'point';
@@ -86,6 +101,7 @@ export type ResolvedInteractionTarget =
86101
hint?: string;
87102
preActionNodes?: SnapshotNode[];
88103
resolution?: ResolutionDisclosure;
104+
recordingTarget?: RecordingTargetOverride;
89105
}
90106
| {
91107
kind: 'selector';
@@ -98,6 +114,7 @@ export type ResolvedInteractionTarget =
98114
hint?: string;
99115
preActionNodes?: SnapshotNode[];
100116
resolution?: ResolutionDisclosure;
117+
recordingTarget?: RecordingTargetOverride;
101118
};
102119

103120
/**

‎src/core/interaction-targeting.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,14 @@ function findPreferredActionableDescendant(
104104
return current === node ? null : current;
105105
}
106106

107-
function isSemanticTouchTarget(node: SnapshotNode): boolean {
107+
/**
108+
* THE canonical interactive-role classification for touch: a node whose
109+
* type/role/subrole names a control that independently receives taps. Shared
110+
* by the hittable-ancestor promotion above and #1280's press-retarget
111+
* competing-descendant guard (`press-retarget.ts`) — one list, never a
112+
* parallel copy.
113+
*/
114+
export function isSemanticTouchTarget(node: SnapshotNode): boolean {
108115
const roles = [node.type, node.role, node.subrole].map((value) => normalizeType(value ?? ''));
109116
return roles.some(isSemanticTouchRole);
110117
}

0 commit comments

Comments
 (0)