Skip to content

Commit 9b610fb

Browse files
authored
feat(replay): recorded landmark identity for wait, is coverage — read-only step identity (#1349) (#1381)
* refactor(replay): extract shared target-evidence tree helpers into src/replay Move buildIndexMap/buildAncestryChain/filterIdentitySet out of the daemon's session-target-evidence into the shared replay zone so the commands runtime (wait's polling loop, #1349) can consume them without importing the daemon; press-retarget drops its private buildIndexMap duplicate. * feat(replay): recorded landmark identity verification for wait, get-pattern coverage for is (#1349) - New CommandDescriptor trait targetIdentityVerification pins the evidence-carrying command set and routes wait to a post-resolution phase so an annotated wait never enters the generic pre-dispatch verification (an absent landmark is its expected starting condition). - wait <selector> records landmark-mode target-v1 evidence (existence self-check; identity-empty matches record no annotation) and, on replay, keeps polling until a selector match carries the recorded identity; a deadline with only impostor matches fails closed as an identity-mismatch REPLAY_DIVERGENCE, a recorded-unverifiable annotation refuses before polling, and a plain timeout stays an action-failure divergence. - is (except exists) joins the get pattern: evidence at record time, generic pre-dispatch verification, and the post-resolution guard threaded through dispatch; direct-iOS fast paths for wait/is are gated during recording and guarded replays. - Read-only find stays intentionally unannotated (fuzzy-locator resolution has no selector-chain identity token), proven by test. * feat(publication): destination guard requires verified recorded landmark identity (#1349) A qualifying ADR 0016 guard is now a selector wait whose target-v1 annotation is verified; identity-less or unverifiable guards are refused with a recovery hint. Adds the reshuffled-screen false-pass regression: record -> publish -> replay against a same-label/different-ancestry tree diverges as identity-mismatch (matchCount >= 1 proving the selector alone would have false-passed). * refactor(replay): dedupe post-dispatch identity-mismatch shaping, trim evidence-writer complexity Shared buildPostDispatchIdentityMismatchResponse behind the guard and wait-landmark conversions; extracted payload-ceiling helpers from computeTargetEvidence; identity-refusal conversion split out of resolveReplayStepResponse. Docs: ADR 0012 decision 3 amendment (#1349), ADR 0016 guard strengthening, help workflow/save-script text. * refactor(replay): make landmark evidence's record-time verification explicit, trim ADR-restating docs The landmark-mode self-check was provably a tautology (the winner is a member of its own identity set whenever the parent walk is intact), so a membership scan defended only by a comment is replaced with the explicit decision: broken walk fails closed, landmark is verified by construction, action mode keeps decision 3's step-5 self-check. Doc comments that re-argued the ADR amendment now state behavior and point to it. * chore: untrack multitouch-helper build artifacts, ignore its build/dist dirs Generated Android helper output swept into the earlier refactor commit by accident; analogous snapshot-helper/ime-helper build dirs were already ignored. * fix(interaction): wait polls ride out content-unreadable captures (live-validated on Android) Live ADR 0016 validation on a Pixel emulator showed a destination-guard wait replayed immediately after a navigation press deterministically dies: the first poll's capture lands mid-transition and the Android helper's 'insufficient foreground app content' verdict threw out of the polling loop. iOS already yields the same state as a sparse verdict with no matches, so the loop kept polling there — this makes wait semantics platform-consistent. A content-verdict capture failure (isUnreadableCaptureContentError) now counts as a no-match poll for selector and text waits; a wait whose screen never became readable rethrows the last capture verdict at the deadline, so persistent breakage keeps its diagnosis. Other capture failures still throw immediately. * fix(snapshot): narrow unreadable-capture classification to enumerated content verdicts Android stamps androidSnapshotHelperFailureReason on mechanism failures too (helper timeouts, adb failures, missing helper artifact — free-form reason strings), so matching any string made waits poll those to their deadline instead of failing immediately. The predicate now matches only the enumerated content-recovery reasons, and AndroidHelperContentRecoveryDecision derives its reason union from the same list so a new content verdict cannot miss the predicate. Adds the realistic wrapped mechanism-error regression the synthetic test missed. * test(interaction): make the wait mechanism-failure regressions revert-sensitive Assert exactly one capture attempt: the broad any-string classifier would poll the repeated fixture error to the fake-clock deadline and rethrow the same message, passing the message-only assertion. Verified the mechanism test fails against the broadened classifier and passes against the narrowed one.
1 parent 968db8b commit 9b610fb

33 files changed

Lines changed: 1980 additions & 253 deletions

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,5 @@ android/snapshot-helper/build/
4040
android/snapshot-helper/dist/
4141
android/ime-helper/build/
4242
android/ime-helper/dist/
43+
android/multitouch-helper/build/
44+
android/multitouch-helper/dist/

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,12 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-16). See [Mig
3838
- Decision 6 amendment, repair-segment default exclusion of observation-only commands and the `--record`
3939
opt-in — stage 1 (interim guidance only, #1287) and stage 2 (the semantic default-exclusion fix,
4040
this amendment) of #1271. See the amendment under decision 6 below.
41+
- Decision 3 amendment (#1349), read-only step identity — landmark-mode `target-v1` evidence and
42+
post-resolution (in-loop) verification for selector `wait`, `get`-pattern coverage for `is` (except
43+
`exists`), the `targetIdentityVerification` descriptor trait pinning the evidence-carrying command
44+
set, explicit deferral of read-only `find`/`is exists`/non-selector waits, and ADR 0016's destination
45+
guard strengthened to require a `verified` landmark annotation, with a provider-level
46+
reshuffled-screen false-pass regression. See the amendment under decision 3 below.
4147

4248
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
4349
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -370,6 +376,58 @@ A recorded `id` never matches a node without that id.
370376
> platform-agnostic: an RN `FlatList` row (`Cell`) that is label-less with a labeled `Text` child hits
371377
> the same substitution on iOS.
372378
379+
> **Amendment (#1349): read-only step identity — `wait` landmark verification, `is` coverage, and the
380+
> per-command verification-phase trait.** Decision 3's evidence and machinery extend to the eligible
381+
> read-only steps without changing their execution semantics. The annotation format is **unchanged**
382+
> (`target-v1`, same tuple, same bounds, same writer/parser); what is new is a per-command dispatch of
383+
> WHEN verification runs, declared on the central `CommandDescriptor` as
384+
> `targetIdentityVerification: 'pre-dispatch' | 'post-resolution'` and pinned by a completeness test —
385+
> a new evidence-carrying command must choose its phase explicitly rather than silently entering the
386+
> generic pre-dispatch path.
387+
>
388+
> - **`wait <selector>` — covered, `post-resolution`, landmark (existence) semantics.** A wait's
389+
> landmark may legitimately be absent when its step starts, so the generic pre-action verification
390+
> (paths 2-6) must never run for it; only path 1 (recorded-`unverifiable`, which consults no screen)
391+
> refuses up front. The recorded annotation instead travels into the wait's own polling loop
392+
> (`internal.replayLandmarkGuard`, the same daemon-only channel as `replayTargetGuard`), which
393+
> strengthens the success condition from "the selector matches" to "a selector match carries the
394+
> recorded identity" — the identity tier only (local identity + leaf-anchored ancestry prefix). The
395+
> positional disambiguation signals are deliberately **not** compared: a destination guard proves the
396+
> landmark exists on the ready screen, not that it kept its list position, and requiring position
397+
> would produce the inverse false-failure on a legitimately reshuffled-but-correct screen. Polling is
398+
> preserved fail-open in time and fail-closed at the deadline: a same-selector impostor never aborts
399+
> the wait (a transient look-alike mid-transition is exactly what a wait exists to wait through); at
400+
> the deadline, a poll history containing selector matches that never carried the identity surfaces
401+
> as an `identity-mismatch` divergence (`matchCount` from the last matching poll, observed identity +
402+
> first ancestry mismatch in `mismatches`) BEFORE the wait reports success, while a selector that
403+
> never matched at all remains the ordinary wait timeout (`action-failure` — that failure needs a
404+
> state repair, not an identity repair, and its `repairHint` routing is already correct).
405+
> A wait poll also rides out a capture that judged the current screen UNREADABLE (the Android
406+
> helper's content verdicts, `isUnreadableCaptureContentError`; iOS surfaces the same state as a
407+
> sparse verdict with no matches): live Android validation showed a guard wait replayed right after
408+
> a navigation press deterministically dies on its first mid-transition capture otherwise. A wait
409+
> whose screen never became readable rethrows the last capture verdict at the deadline instead of
410+
> masking it as a generic timeout.
411+
> Record time writes evidence in a **landmark mode**: the step-5 self-check is identity-set
412+
> MEMBERSHIP rather than isolation (mirroring what replay will actually verify), and a matched node
413+
> with no id and no label after #1269 demotion records **no annotation** — a role-only landmark
414+
> identity is near-vacuous, and an unannotated wait keeps its existing selector-existence semantics
415+
> instead of failing closed on evidence that never discriminated anything. ADR 0016's destination
416+
> guard consumes exactly this: a qualifying guard is a selector wait with a `verified` annotation.
417+
> - **`get` — unchanged**; already covered by the pre-dispatch path and the post-resolution guard.
418+
> - **`is` (all predicates except `exists`) — covered, `pre-dispatch`, the `get` pattern end-to-end.**
419+
> `is` resolves a unique node immediately, so pre-action verification is semantically valid; the
420+
> resolved node/tree feed record-time evidence, and dispatch threads `replayTargetGuard` into
421+
> `assertExpectedResolvedTarget` exactly like `get`. The direct-iOS `is`/`wait` fast paths are gated
422+
> off during recording and guarded replays, mirroring `get`'s existing recording gate.
423+
> - **Intentionally deferred, with tests proving no annotation is recorded and no identity check runs:**
424+
> `is exists` (existence assertion with no unique winner; wait-like semantics without the
425+
> guard-critical role), every read-only `find` variant (fuzzy-locator resolution has no
426+
> selector-chain identity token for the classifier, and publication already refuses mutating `find`
427+
> as non-verifiable), and `wait text`/`wait stable`/duration waits/`wait @ref` (no element target, or
428+
> a session-local ref that ADR 0016 already refuses to publish; `wait @ref` is rejected rather than
429+
> converted to a portable selector).
430+
373431
**Ancestry.** The chain is the nearest **K = 8** ancestors of the target, ordered **leaf→root** (nearest
374432
ancestor first), each entry `{ role, label? }` under the same normalization (`role` may be the empty
375433
string when the node has no type; `label` is omitted when empty). Truncation drops entries from the
@@ -1254,6 +1312,7 @@ not restate or amend the decisions.
12541312
| 8. Agent-supervised re-record repair, base | 6 (R1-R6) | Shipped | #1228 |
12551313
| 9. Repair-transaction lifecycle | 6 (R7) | Shipped | #1235 |
12561314
| 10. Repair-segment default exclusion + `--record` | 6 amendment | Shipped | #1271 stage 2 |
1315+
| 11. Read-only step identity (wait landmark verification, `is` coverage) | 3 amendment (#1349) | Shipped | — |
12571316

12581317
Step 4 (#1209) added the `selector-miss`/`identity-mismatch`/`identity-unverifiable` divergence kinds
12591318
and a post-resolution target guard that cross-checks the dispatched winner against the verified member.

‎docs/adr/0016-active-session-script-publication.md‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -87,15 +87,27 @@ the serialized guard before filesystem work and never relies on repair-only bare
8787
`session save-script` refuses publication without this **destination guard** and tells the author to
8888
record one. V1 does not infer a screen identity from a snapshot or synthesize an implicit guard.
8989

90-
The initial guard is selector-level. `wait` does not yet carry recorded-landmark identity through its
91-
polling resolution, so the guard proves that an element matching its selector exists, not that it is the
92-
same landmark element observed while authoring. Authors must choose a selective destination-specific
93-
landmark; a reshuffled screen containing the same weak label elsewhere can false-pass.
94-
[#1349](https://github.com/callstack/agent-device/issues/1349) owns the identity design for waits and
95-
remaining read-only steps. It must preserve polling: ADR 0012's current pre-action `target-v1`
90+
> **Amendment (#1349, shipped).** The guard is now identity-level, not merely selector-level. A
91+
> selector wait recorded while ARMED captures landmark-mode `target-v1` evidence for the matched
92+
> element (ADR 0012's read-only-step amendment), and a qualifying destination guard is exactly a
93+
> selector wait whose annotation is `verification: "verified"` — a selector wait on an identity-empty
94+
> element (no id and no label) records no annotation and does not qualify, so publication refuses it
95+
> with a recovery hint naming a labeled/id-bearing landmark. On replay, polling is preserved: the wait
96+
> keeps polling until a selector match carries the recorded identity (local identity + leaf-anchored
97+
> ancestry prefix), and a deadline reached with only same-selector impostors fails closed as an
98+
> `identity-mismatch` `REPLAY_DIVERGENCE` before the wait reports success. The reshuffled-screen
99+
> false-pass below is covered by a provider-scenario regression (record → publish → replay against a
100+
> reshuffled tree whose same-label node sits under a different id/ancestry).
101+
102+
The original selector-level caveat, retained as context: the guard proved that an element matching its
103+
selector exists, not that it is the same landmark element observed while authoring, so a reshuffled
104+
screen containing the same weak label elsewhere could false-pass.
105+
[#1349](https://github.com/callstack/agent-device/issues/1349) owned the identity design for waits and
106+
remaining read-only steps. It preserves polling: ADR 0012's pre-action `target-v1`
96107
verification cannot be attached to a wait unchanged because a not-yet-present landmark is the expected
97-
starting condition. Any identity check for a wait happens after its selector resolves, before the wait
98-
reports success, or uses a distinct guard-specific mechanism.
108+
starting condition, so the identity check runs after the wait's selector resolves and before the wait
109+
reports success (see ADR 0012's #1349 amendment for the mechanism and the read-only-step coverage
110+
classification).
99111

100112
The phrase "last mutating action" is derived from a request-sensitive recording-effect trait on the
101113
central `CommandDescriptor`, required for every command that records session actions and guarded by a
@@ -203,7 +215,9 @@ executing that script, not the artifact being saved.
203215
- The artifact replays from a cold start, completes its destination guard, returns the live session id,
204216
and accepts a subsequent command on that session.
205217
- Every action in ADR 0012's existing target-binding command set has canonical identity evidence and no
206-
unresolved `@ref` reaches disk; the destination guard remains selector-level until #1349 lands.
218+
unresolved `@ref` reaches disk; since #1349 the destination guard additionally requires verified
219+
recorded landmark identity, with a reshuffled-screen false-pass regression proving replay fails
220+
closed.
207221
- Existing-target refusal preserves the original bytes; `--force` replaces atomically; a failed publish
208222
remains retryable.
209223
- After PUBLISHED, later ordinary actions remain usable, repeated `session save-script` fails, plain

‎src/cli/parser/cli-help.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,7 @@ Reusable open-to-destination scripts:
234234
agent-device press 'id="continue"' --settle
235235
agent-device wait 'role="heading" label="Screen X"'
236236
agent-device session save-script
237-
session save-script [path] [--force] publishes the sole recorded open through the destination guard, omits close, and leaves the session active. A duration wait, wait stable, or wait @ref is not a destination guard. A second successful open aborts publication; start a fresh session to author again.
237+
session save-script [path] [--force] publishes the sole recorded open through the destination guard, omits close, and leaves the session active. The guard is a selector wait on a labeled or id-bearing landmark: its recorded identity is captured while armed, and replay verifies that identity after the wait's selector resolves, so a reshuffled screen with the same label elsewhere fails closed instead of false-passing. A duration wait, wait stable, wait @ref, or a selector wait on an unlabeled element is not a destination guard. A second successful open aborts publication; start a fresh session to author again.
238238
Recorded fill/type inputs are written literally to the .ad file. Do not record passwords, tokens, or other secrets; use pre-authenticated test state or non-secret fixture credentials until parameterized input authoring is available.
239239
240240
Snapshots and refs:

0 commit comments

Comments
 (0)