Skip to content

Commit a0ccd09

Browse files
committed
fix(replay): retarget identity-empty press containers to their labeled descendant (#1280)
Android list-row presses target a clickable container (role="linearlayout") with no id, no label, no value/text — its title lives on a labeled descendant (the android:id/title TextView, whose own id #1272 already demotes for being non-unique). The container's identity is role-only and shared by every row, so replay disambiguates positionally and mis-binds under reorder (measured matchCount 12, 20/20 identity-mismatch). Retarget at record time: when a press/click/fill resolves to an identity-empty container (rule 1), substitute its first labeled descendant in document order (rule 2), but only when the container's subtree has no other interactive/hittable node (rule 3, fail-closed — a trailing Switch/Checkbox must not retarget, since a tap at the descendant's center vs the container's could land on different controls). Guard-blocked or label-less subtrees record exactly as today. Implemented once at the single recording choke point (describeResolvedInteractionNode, resolution.ts): the returned node feeds BOTH buildSelectorChainForNode's chain and (downstream, via recordedTargetCapture) computeTargetEvidence, so the two writers can never half-retarget. Recording-time only — resolveSelectorChain and live press/fill dispatch are unchanged; the tap point is already fixed against the original container before this substitution runs. Adds an ADR 0012 decision 3 amendment (mirroring #1269's), a press-retarget unit/guard/cross-invariant suite (including an RN FlatList iOS parity fixture), and a reorder+insert e2e proving the retargeted recording rebinds by role+label where the un-retargeted container recording refuses.
1 parent f774738 commit a0ccd09

5 files changed

Lines changed: 560 additions & 4 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig
2424
(a shared Android framework resource id such as `android:id/title`, or a reused RN `FlatList` `testID`)
2525
falls back to role+label in both `computeTargetEvidence`'s `target-v1` tuple and
2626
`buildSelectorChainForNode`'s chain (#1269).
27+
- Decision 3 amendment, record-time press retarget to a labeled descendant — a press/click/fill whose
28+
resolved winner is an identity-empty container (no surviving id, no label, no value/text) is recorded
29+
against its first labeled descendant instead, so both writers key off a node with selective identity; a
30+
container whose subtree holds another interactive/hittable node is recorded unchanged (#1280).
2731

2832
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
2933
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -329,6 +333,25 @@ A recorded `id` never matches a node without that id.
329333
> case, but the rule is capture-time uniqueness, not an `android:id/*` namespace heuristic: a reused RN
330334
> `FlatList` `testID` hits the same demotion on iOS.
331335
336+
> **Amendment (#1280).** A press/click/fill whose resolved winner is an *identity-empty* container — no
337+
> id survives #1269's demotion (absent, or demoted for being non-unique), no label, and no value/text
338+
> under the same normalization `buildSelectorChainForNode` uses — is recorded against its first labeled
339+
> descendant instead of the container itself, so the recorded selector chain and the `target-v1` tuple
340+
> both carry a selective role+label identity rather than a bare, tree-wide-shared role. The measured case
341+
> is Android: a list row's clickable node is a label-less `role="linearlayout"` container whose visible
342+
> title lives on a child `TextView`. "First labeled descendant" means the first node in document order
343+
> (this section's canonical total order) within the container's whole subtree whose normalized label is
344+
> non-empty. The substitution is guarded fail-closed: it fires only when that subtree contains **no other**
345+
> interactive/hittable node besides the container itself — a row with a trailing `Switch` or `Checkbox`
346+
> must not retarget, because a tap at the labeled descendant's center and a tap at the container's center
347+
> could land on different controls once the descendant, not the container, is what replay taps. A
348+
> container the guard blocks, or for which no descendant carries a label, is recorded exactly as before
349+
> this amendment. This is a **recording-time** substitution only — `resolveSelectorChain` and live
350+
> press/fill dispatch are unchanged; the action still taps the point resolved against the original
351+
> container, and only what gets *written* to session history / the `.ad` script retargets to the
352+
> descendant. The rule is platform-agnostic: an RN `FlatList` row (`Cell`) that is label-less with a
353+
> labeled `Text` child hits the same substitution on iOS.
354+
332355
**Ancestry.** The chain is the nearest **K = 8** ancestors of the target, ordered **leaf→root** (nearest
333356
ancestor first), each entry `{ role, label? }` under the same normalization (`role` may be the empty
334357
string when the node has no type; `label` is omitted when empty). Truncation drops entries from the

‎src/commands/interaction/runtime/resolution.ts‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
type SelectorResolution,
1717
} from '../../../selectors/index.ts';
1818
import { buildSelectorChainForNode } from '../../../selectors/build.ts';
19+
import { resolvePressRecordingTarget } from '../../../selectors/press-retarget.ts';
1920
import {
2021
findNodeByLabel,
2122
normalizeType,
@@ -382,6 +383,13 @@ function buildResolutionDiagnosticEntry(
382383
};
383384
}
384385

386+
// #1280: press/click/fill are the actions whose recorded evidence can
387+
// substitute an identity-empty container for its labeled descendant (a
388+
// list-row press is the measured shape); reads (get/find) already have
389+
// their own #1269 id-demotion treatment and go through selector-read.ts,
390+
// not this function.
391+
const PRESS_RETARGET_ACTIONS = new Set<InteractionAction>(['click', 'press', 'fill']);
392+
385393
// Shared tail of a resolved ref/selector interaction target: the node itself
386394
// plus everything derived from it for the response.
387395
function describeResolvedInteractionNode(
@@ -399,14 +407,24 @@ function describeResolvedInteractionNode(
399407
preActionNodes: SnapshotState['nodes'];
400408
resolution: ResolutionDisclosure;
401409
} {
410+
// #1280: the ONE substitution point feeding both writers — the returned
411+
// `node` is what `computeTargetEvidence` records evidence for
412+
// (`recordedTargetCapture`, interaction-touch-response.ts), and
413+
// `selectorChain` below is built from the same `recordedNode`, so the two
414+
// never half-retarget. The live tap point was already resolved against
415+
// the ORIGINAL `node` before this function runs (see callers) — this is
416+
// recording-time only.
417+
const recordedNode = PRESS_RETARGET_ACTIONS.has(action)
418+
? resolvePressRecordingTarget(node, nodes)
419+
: node;
402420
return {
403-
node,
404-
selectorChain: buildSelectorChainForNode(node, runtime.backend.platform, {
421+
node: recordedNode,
422+
selectorChain: buildSelectorChainForNode(recordedNode, runtime.backend.platform, {
405423
action: action === 'fill' ? 'fill' : 'click',
406424
nodes,
407425
}),
408-
refLabel: resolveRefLabel(node, nodes),
409-
...describeNonHittableTarget(node, action),
426+
refLabel: resolveRefLabel(recordedNode, nodes),
427+
...describeNonHittableTarget(recordedNode, action),
410428
preActionNodes: nodes,
411429
resolution,
412430
};

‎src/daemon/handlers/__tests__/session-replay-target-classification.test.ts‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import type { TargetAnnotationV1 } from '../../../replay/target-identity.ts';
55
import { computeTargetEvidence } from '../../session-target-evidence.ts';
66
import { buildSelectorChainForNode } from '../../../selectors/build.ts';
77
import { parseSelectorChain, resolveSelectorChain } from '../../../selectors/index.ts';
8+
import { resolvePressRecordingTarget } from '../../../selectors/press-retarget.ts';
89
import { classifyReplayTarget } from '../session-replay-target-classification.ts';
910
import {
1011
bottomTabsRealCaptureFixture,
@@ -561,3 +562,84 @@ test('#1269 e2e: a demoted shared-id row rebinds by role+label after the shared-
561562
});
562563

563564
// ---------------------------------------------------------------------------
565+
// #1280 e2e: a PRESS on a row's clickable CONTAINER (the LinearLayout
566+
// wrapper, not its title TextView) is identity-empty — no id, no label — so
567+
// record-time retarget (`resolvePressRecordingTarget`) substitutes the
568+
// labeled title descendant before either writer runs. Reorder the shared-id
569+
// rows and insert a new one on replay: the retargeted recording rebinds the
570+
// descendant by its (#1272-demoted) role+label. Contrast: recording the
571+
// container itself — today's behavior, no retarget — produces a role-only
572+
// selector shared by every row's wrapper, which refuses to bind uniquely
573+
// under the very same reorder.
574+
// ---------------------------------------------------------------------------
575+
576+
test('#1280 e2e: a retargeted press on a row container rebinds its labeled descendant after reorder + insert; the un-retargeted container recording refuses', () => {
577+
const ANDROID = 'android' as const;
578+
const recordNodes = androidSharedIdListFixture([
579+
{ id: 'android:id/title', label: 'Network & internet', y: 100 },
580+
{ id: 'android:id/title', label: 'Connected devices', y: 148 },
581+
{ id: 'android:id/title', label: 'Apps', y: 196 },
582+
]);
583+
const recordedTitle = recordNodes.find((node) => node.label === 'Connected devices')!;
584+
const container = recordNodes.find((node) => node.index === recordedTitle.parentIndex)!;
585+
assert.equal(container.type, 'LinearLayout');
586+
assert.equal(container.identifier, undefined);
587+
assert.equal(container.label, undefined);
588+
589+
// What #1280 changes: the recorded step is written against the
590+
// retargeted descendant, not the label-less container the press hit.
591+
const recordedNode = resolvePressRecordingTarget(container, recordNodes);
592+
assert.equal(recordedNode.ref, recordedTitle.ref);
593+
const recorded = computeTargetEvidence({ node: recordedNode, preActionNodes: recordNodes })!;
594+
assert.equal(recorded.id, undefined, 'the shared android:id/title stays demoted per #1272');
595+
assert.equal(recorded.role, 'textview');
596+
assert.equal(recorded.label, 'Connected devices');
597+
const chain = buildSelectorChainForNode(recordedNode, ANDROID, {
598+
action: 'click',
599+
nodes: recordNodes,
600+
});
601+
const token = chain.join(' || '); // the recorded selector positional the replay loop re-resolves
602+
603+
// Replay-time tree: a new conditional row appears at the top and the
604+
// shared-id rows are in a DIFFERENT document/viewport order.
605+
const replayNodes = androidSharedIdListFixture([
606+
{ id: 'android:id/title', label: 'Wi-Fi', y: 100 },
607+
{ id: 'android:id/title', label: 'Apps', y: 148 },
608+
{ id: 'android:id/title', label: 'Connected devices', y: 196 },
609+
{ id: 'android:id/title', label: 'Network & internet', y: 244 },
610+
]);
611+
const expected = replayNodes.find((node) => node.label === 'Connected devices')!;
612+
613+
const result = classifyReplayTarget({
614+
recorded,
615+
token,
616+
nodes: replayNodes,
617+
platform: ANDROID,
618+
refLabel: undefined,
619+
requireRect: true,
620+
allowDisambiguation: true,
621+
});
622+
assertVerified(result, { winnerRef: expected.ref, matchCount: 1 });
623+
624+
// Contrast: recording the CONTAINER itself (no retarget) leaves a
625+
// role-only identity — every row's wrapper shares it — that a
626+
// requireUnique resolve refuses to bind under the same reorder. This is
627+
// the FDR the retarget removes.
628+
const containerChain = buildSelectorChainForNode(container, ANDROID, {
629+
action: 'click',
630+
nodes: recordNodes,
631+
});
632+
assert.deepEqual(containerChain, ['role="linearlayout"']);
633+
const containerResolved = resolveSelectorChain(
634+
replayNodes,
635+
parseSelectorChain(containerChain.join(' || ')),
636+
{ platform: ANDROID, requireRect: true, requireUnique: true },
637+
);
638+
assert.equal(
639+
containerResolved,
640+
null,
641+
'the un-retargeted container selector refuses to bind uniquely',
642+
);
643+
});
644+
645+
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)