Skip to content

Commit f83b9e6

Browse files
committed
fix: rank divergence screen.refs within the cap so overlays are not buried
The #1264 root cause is cap burial, not capture scope: buildReplayDivergenceScreenRefs sliced candidates in document order, so a fully-captured separate-window overlay (volume dialog, QS shade, permission dialog) that enumerates after the app window's ~77 nodes lands past position 20 and is truncated away — the report shows a healthy-looking app under a covering overlay it cannot see (archived evidence: screen.truncated: true, zero volume refs). - Rank within the cap instead of document-order slicing: foreign-window (non-app-bundleId) hittable nodes — the dismiss targets for whatever covers the app — are promoted ahead of app content, otherwise stable (document order preserved within each tier; equal-priority app nodes never reshuffled). The 20-cap is a byte bound, not a first-20-in-tree-order policy. - Occlusion fallback: when a system overlay mass-covers the app (every app node annotated interactionBlocked: 'covered'), surface those covered nodes rather than emitting an empty screen.refs — a report whose capture holds meaningful nodes but whose refs is empty is broken by construction. - repairHint/suggestions consume the full captured node list, not the capped refs slice, so hint routing is unaffected; only screen.refs selection changes. Detection keys off node.bundleId (Android-only, from the a11y package); iOS/macOS leave per-node bundleId undefined, so ranking degrades to document order there (safe — those platforms surface modals via the probe path, not by cap-competing). Guarded on a known appBundleId so a sessionless capture never reorders. Tests: replaces the small-fixture #1264 test (which the overlay fit inside the cap regardless of order, so it did not prove the invariant) with a realistic full fixture (24 app controls + overlay dismiss-target captured LAST) that fails on document-order slicing and passes with ranking; plus occlusion tests (mass-covered app -> overlay surfaces, refs non-empty; bare-scrim fallback -> covered app nodes surfaced, refs non-empty). Both were verified to fail before the fix. ADR-0012 decision 4 amendment reworded to cover ref-selection ranking and the occlusion guarantee, not only capture scope. Refs #1264
1 parent dbc5b09 commit f83b9e6

3 files changed

Lines changed: 353 additions & 39 deletions

File tree

‎docs/adr/0012-interactive-replay.md‎

Lines changed: 33 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,11 @@ Accepted (2026-07-10); partially implemented (last updated 2026-07-13). See [Mig
1212
`REPLAY_DIVERGENCE` repair-loop fix (#1223).
1313
- Decision 6, the base agent-supervised re-record repair — `replay --save-script` arming, the
1414
post-watermark healed slice, `repairHint`, and the writer's bare-`@ref` fail-loud guard (#1228).
15-
- Decision 4 amendment, `screen`'s capture scope — the divergence `screen` capture is unified with
16-
`snapshot`'s own full-window scope, with the chrome and meaningful-target filters layered on top as
17-
filters, not scopings (#1264).
15+
- Decision 4 amendment, `screen`'s capture scope and ref selection — the divergence `screen` capture is
16+
unified with `snapshot`'s own full-window scope (chrome and meaningful-target filters layered on top as
17+
filters, not scopings), and `screen.refs` is ranked within the byte cap (foreign-window dismiss targets
18+
ahead of app content; mass-covered app nodes surfaced rather than emptied) instead of sliced in document
19+
order, so a captured overlay is never buried past the cap (#1264).
1820

1921
**Accepted but NOT yet implemented** (this amendment; tracked by #1235 — repair-transaction lifecycle):
2022
the R7 repair-transaction keep-alive and its distinct `resume.repairSessionHeld` signal, the ARMED →
@@ -441,21 +443,34 @@ healthy snapshot digest and the only form that issues actionable refs. `{ state:
441443
hint }` is returned when capture fails or is sparse; it has no refs or generation and must not fall back to
442444
the old session tree. Screen-capture failure never replaces or masks the original replay cause.
443445

444-
> **Amendment (#1264): `screen`'s capture scope.** `refs` is a filtered digest of the exact same tree a
445-
> plain `snapshot` would return at that moment — the capture underneath `screen` is built with `snapshot`'s
446-
> own full-window scope (Android: the snapshot-helper route, with the existing graceful app-scoped fallback
447-
> only when the helper is unavailable; iOS: the same capture semantics as the bounded system-modal probe
448-
> path; macOS/Linux: their surface-scoped branches), never a narrower, app-window-only capture assembled
449-
> separately for divergence reporting. The chrome filter (#1233/#1256, `collectSettleChromeRefs`) and the
450-
> meaningful-target filter (label/id or `hittable`) are layered ON TOP of that full capture as **filters**,
451-
> not as a separate, narrower scoping — a filter may drop a node the full capture contains, but the capture
452-
> itself must never omit content `snapshot` would show. This is a hard invariant: **an agent must never see
453-
> a healthier `screen` in a divergence report than a plain `snapshot` would show it.** Concretely, a
454-
> separate-window system overlay covering the app at the moment of capture — a held volume dialog, a
455-
> persistent quick-settings shade, a permission dialog — must appear in `screen.refs` (its actionable/
456-
> hittable/labeled nodes surviving the filters) exactly as `snapshot` would present it, and `repairHint`
457-
> (decision 6) is computed over that same full, correctly-scoped capture so it is never routed as if the
458-
> app underneath a covering overlay were healthy.
446+
> **Amendment (#1264): `screen`'s capture scope and ref selection.** `refs` is a filtered, ranked digest
447+
> of the exact same tree a plain `snapshot` would return at that moment. Two guarantees back the invariant:
448+
>
449+
> 1. **Capture scope.** The capture underneath `screen` is built with `snapshot`'s own full-window scope
450+
> (Android: the snapshot-helper route, with the existing graceful app-scoped fallback only when the
451+
> helper is unavailable; iOS: the same capture semantics as the bounded system-modal probe path;
452+
> macOS/Linux: their surface-scoped branches), never a narrower, app-window-only capture assembled
453+
> separately for divergence reporting. The chrome filter (#1233/#1256, `collectSettleChromeRefs`) and the
454+
> meaningful-target filter (label/id or `hittable`) are layered ON TOP of that full capture as
455+
> **filters**, not as a separate, narrower scoping — a filter may drop a node the full capture contains,
456+
> but the capture itself must never omit content `snapshot` would show.
457+
> 2. **Ref selection within the cap.** The `screen.refs` cap is a **byte bound**, not a "first N in document
458+
> order" policy. A separate-window overlay enumerates AFTER the app window's nodes, so a document-order
459+
> slice truncates a fully-captured overlay away (its dismiss target sits past the cap) — reporting a
460+
> healthy-looking app even though the capture holds the covering window. So `refs` is RANKED before it is
461+
> capped: foreign-window (non-app-`bundleId`) hittable nodes — the dismiss targets for whatever covers
462+
> the app — are promoted ahead of app content, with document order otherwise preserved (stable within
463+
> each tier). And when a system overlay MASS-COVERS the app so every app node is annotated `covered`,
464+
> those covered nodes are surfaced rather than emitting an empty `refs`: a report whose capture holds
465+
> meaningful nodes but whose `refs` is empty is broken by construction.
466+
>
467+
> This is a hard invariant: **an agent must never see a healthier `screen` in a divergence report than a
468+
> plain `snapshot` would show it.** Concretely, a separate-window system overlay covering the app at the
469+
> moment of capture — a held volume dialog, a persistent quick-settings shade, a permission dialog — must
470+
> appear in `screen.refs` (its actionable/hittable/labeled nodes surviving the filters AND the cap) exactly
471+
> as `snapshot` would present it. `repairHint` (decision 6) and `suggestions` (decision 1) consume the FULL
472+
> captured node list, not the capped `refs` slice, so hint routing is computed over the same full,
473+
> correctly-scoped capture and is never routed as if the app underneath a covering overlay were healthy.
459474
460475
Response levels bound the entire serialized UTF-8 `details.divergence` object, not merely its arrays:
461476
compact (`--level digest`) is at most **8 KiB**, default at most **24 KiB**, and full at most **64 KiB**.

‎src/daemon/handlers/__tests__/session-replay-divergence.test.ts‎

Lines changed: 267 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -472,20 +472,17 @@ test('buildReplayFailureDivergence excludes Android status-bar/IME chrome from s
472472
expect(screen.truncated).toBeUndefined();
473473
});
474474

475-
// #1264 invariant: an agent must never see a healthier `screen` in a
476-
// divergence report than a plain `snapshot` would show it. A separate-window
477-
// system overlay (e.g. the volume dialog) that a full `snapshot` capture would
478-
// include must also survive into divergence `screen.refs` — its actionable
479-
// (hittable) content passing the meaningful-target filter — while ordinary
475+
// #1264 coexistence (chrome filter + overlay, real walked fixture): a
476+
// separate-window system overlay (volume dialog) that a full `snapshot` capture
477+
// includes survives into divergence `screen.refs` — its actionable (hittable)
478+
// content passing the meaningful-target filter — while ordinary
480479
// status-bar/nav-bar/IME chrome in the SAME capture is still excluded. This
481-
// extends the real (walked) Android fixture from the test above with a
482-
// systemui overlay run using real, live-verified volume-dialog leaf ids
483-
// (`volume_dialog_container`, `volume_new_ringer_active_icon_container`, see
484-
// `core/__tests__/snapshot-chrome-android-statusbar.test.ts`), appended to the
485-
// RAW array so the whole tree — app content, status bar, AND the overlay —
486-
// goes through the real non-raw walk together, exactly as a single full
487-
// capture would.
488-
test('buildReplayFailureDivergence: a system-overlay window in the full capture survives into screen.refs while status/nav chrome is filtered (#1264)', async () => {
480+
// runs the real non-raw Android walk over app content + status bar + overlay
481+
// together, proving the chrome filter and the overlay coexist. NOTE: this
482+
// fixture is small enough that the overlay fits inside the 20-cap regardless of
483+
// ordering, so it does NOT by itself prove cap-burial resistance — that is the
484+
// dedicated realistic-tree test below.
485+
test('buildReplayFailureDivergence: a system-overlay window survives into screen.refs while status/nav chrome is filtered (#1264)', async () => {
489486
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-overlay-'));
490487
const sessionStore = new SessionStore(path.join(root, 'sessions'));
491488
const sessionName = 'default';
@@ -561,3 +558,260 @@ test('buildReplayFailureDivergence: a system-overlay window in the full capture
561558
expect(screen.refs.some((ref) => ref.label === 'Checkout form')).toBe(true);
562559
expect(screen.refs.some((ref) => ref.label === 'review name')).toBe(true); // field-name
563560
});
561+
562+
// Realistic full capture: 24 app controls enumerate in document order BEFORE a
563+
// separate-window overlay, whose lone actionable dismiss target is captured
564+
// LAST (index 24) — exactly the shape of a live tree where the app window's
565+
// ~77 nodes precede a volume dialog / QS shade. 24 app + 1 overlay = 25
566+
// meaningful candidates > the 20-cap. Under the previous
567+
// `candidates.slice(0, 20)` in document order, the overlay sits at position 24
568+
// and is TRUNCATED away (the archived #1264 evidence: `screen.truncated: true`,
569+
// zero volume refs). Under within-cap ranking (foreign-bundle hittable dismiss
570+
// targets promoted ahead of app content) it survives. This test fails on
571+
// document-order slicing and passes with the ranking.
572+
function appSwampWithTrailingOverlay(appBundleId: string) {
573+
const appControls = Array.from({ length: 24 }, (_, i) => ({
574+
index: i,
575+
depth: 1,
576+
type: 'android.widget.Button',
577+
bundleId: appBundleId,
578+
label: `App control ${String(i + 1).padStart(2, '0')}`,
579+
identifier: `${appBundleId}:id/control_${i + 1}`,
580+
// Non-overlapping small rects down the app window: no occlusion fires.
581+
rect: { x: 0, y: 10 * i, width: 120, height: 8 },
582+
hittable: true,
583+
}));
584+
return [
585+
...appControls,
586+
// The overlay's dismiss target, captured LAST — a foreign (systemui) window.
587+
{
588+
index: 24,
589+
depth: 1,
590+
type: 'android.widget.ImageButton',
591+
bundleId: 'com.android.systemui',
592+
identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container',
593+
label: 'Ringer volume',
594+
rect: { x: 300, y: 300, width: 44, height: 44 },
595+
hittable: true,
596+
},
597+
];
598+
}
599+
600+
test('buildReplayFailureDivergence: a fully-captured overlay dismiss-target enumerating LAST still lands within the screen.refs cap (#1264 cap burial)', async () => {
601+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-burial-'));
602+
const sessionStore = new SessionStore(path.join(root, 'sessions'));
603+
const sessionName = 'default';
604+
const appBundleId = 'com.callstack.agentdevicelab';
605+
sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId }));
606+
607+
mockDispatchCommand.mockResolvedValue({
608+
nodes: appSwampWithTrailingOverlay(appBundleId),
609+
truncated: false,
610+
backend: 'android',
611+
});
612+
613+
const action = {
614+
ts: 0,
615+
command: 'press',
616+
positionals: ['label="App control 01"'],
617+
flags: {},
618+
result: { selectorChain: ['label="App control 01"'] },
619+
};
620+
const divergence = await buildReplayFailureDivergence({
621+
error: { code: 'COMMAND_FAILED', message: 'not hittable' },
622+
action,
623+
index: 0,
624+
sourcePath: path.join(root, 'flow.ad'),
625+
sourceLine: 1,
626+
session: sessionStore.get(sessionName),
627+
sessionName,
628+
sessionStore,
629+
logPath: path.join(root, 'daemon.log'),
630+
responseLevel: 'default',
631+
planActions: [action],
632+
planDigest: 'test-plan-digest',
633+
});
634+
635+
expect(divergence.screen.state).toBe('available');
636+
const screen = divergence.screen as Extract<typeof divergence.screen, { state: 'available' }>;
637+
638+
// The overlay dismiss target, captured LAST (position 24, past the 20-cap in
639+
// document order), survives because foreign-window hittable nodes are ranked
640+
// ahead of app content within the cap.
641+
expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true);
642+
// It is ranked first, not merely present.
643+
expect(screen.refs[0]?.label).toBe('Ringer volume');
644+
// Ranking is stable for equal-priority app content: the surviving app refs are
645+
// still the earliest ones in document order (no reshuffle).
646+
expect(screen.refs.some((ref) => ref.label === 'App control 01')).toBe(true);
647+
// The report still honestly reports it dropped candidates beyond the cap.
648+
expect(screen.truncated).toBe(true);
649+
});
650+
651+
// Occlusion interaction (#1264): when a system overlay MASS-COVERS the app,
652+
// every app node is annotated `interactionBlocked: 'covered'` (and
653+
// `hittable: false`) by `annotateCoveredSnapshotNodes`. The covering overlay's
654+
// own actionable node is NOT covered and IS the repair surface. A report whose
655+
// capture holds these nodes but whose `refs` is empty is broken by
656+
// construction. The overlay-like `Dialog` container (a later document node with
657+
// a full-screen rect over the app buttons) triggers the occlusion annotation.
658+
function overlayMassCoveringApp(appBundleId: string) {
659+
const appButtons = Array.from({ length: 3 }, (_, i) => ({
660+
index: i,
661+
depth: 1,
662+
type: 'android.widget.Button',
663+
bundleId: appBundleId,
664+
label: `Field ${String.fromCharCode(65 + i)}`,
665+
identifier: `${appBundleId}:id/field_${i + 1}`,
666+
rect: { x: 20, y: 100 + 60 * i, width: 300, height: 44 },
667+
hittable: true,
668+
}));
669+
const dialogIndex = appButtons.length;
670+
return [
671+
...appButtons,
672+
// Overlay-like container (`Dialog` type) covering the whole app area, later
673+
// in document order — marks every app button `covered`. Non-hittable, so it
674+
// is not itself a dismiss target.
675+
{
676+
index: dialogIndex,
677+
depth: 0,
678+
type: 'android.app.Dialog',
679+
bundleId: 'com.android.systemui',
680+
identifier: 'com.android.systemui:id/volume_dialog_container',
681+
rect: { x: 0, y: 0, width: 402, height: 874 },
682+
},
683+
// The overlay's actionable dismiss target, a child of the dialog.
684+
{
685+
index: dialogIndex + 1,
686+
parentIndex: dialogIndex,
687+
depth: 1,
688+
type: 'android.widget.ImageButton',
689+
bundleId: 'com.android.systemui',
690+
identifier: 'com.android.systemui:id/volume_new_ringer_active_icon_container',
691+
label: 'Ringer volume',
692+
rect: { x: 300, y: 300, width: 44, height: 44 },
693+
hittable: true,
694+
},
695+
];
696+
}
697+
698+
test('buildReplayFailureDivergence: when a system overlay mass-covers the app, the overlay dismiss-target surfaces in screen.refs and refs is non-empty (#1264 occlusion)', async () => {
699+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-occlusion-'));
700+
const sessionStore = new SessionStore(path.join(root, 'sessions'));
701+
const sessionName = 'default';
702+
const appBundleId = 'com.callstack.agentdevicelab';
703+
sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId }));
704+
705+
mockDispatchCommand.mockResolvedValue({
706+
nodes: overlayMassCoveringApp(appBundleId),
707+
truncated: false,
708+
backend: 'android',
709+
});
710+
711+
const action = {
712+
ts: 0,
713+
command: 'press',
714+
positionals: ['label="Field A"'],
715+
flags: {},
716+
result: { selectorChain: ['label="Field A"'] },
717+
};
718+
const divergence = await buildReplayFailureDivergence({
719+
error: { code: 'COMMAND_FAILED', message: 'not hittable' },
720+
action,
721+
index: 0,
722+
sourcePath: path.join(root, 'flow.ad'),
723+
sourceLine: 1,
724+
session: sessionStore.get(sessionName),
725+
sessionName,
726+
sessionStore,
727+
logPath: path.join(root, 'daemon.log'),
728+
responseLevel: 'default',
729+
planActions: [action],
730+
planDigest: 'test-plan-digest',
731+
});
732+
733+
expect(divergence.screen.state).toBe('available');
734+
const screen = divergence.screen as Extract<typeof divergence.screen, { state: 'available' }>;
735+
736+
// A capture with meaningful nodes must never yield an empty screen.refs.
737+
expect(screen.refs.length).toBeGreaterThan(0);
738+
// The overlay's non-covered actionable node is the repair surface and surfaces.
739+
expect(screen.refs.some((ref) => ref.label === 'Ringer volume')).toBe(true);
740+
// The mass-covered app buttons are correctly excluded while a non-covered
741+
// dismiss target exists (they are not tappable under the overlay).
742+
expect(screen.refs.some((ref) => ref.label === 'Field A')).toBe(false);
743+
});
744+
745+
// Occlusion fallback (#1264): the covering overlay has NO actionable node of its
746+
// own (a bare scrim), so once the app is mass-covered there is no non-covered
747+
// meaningful candidate at all. Rather than emit an empty `screen.refs` — a
748+
// report broken by construction — the covered app nodes are surfaced so the
749+
// agent at least sees what is underneath.
750+
function bareScrimMassCoveringApp(appBundleId: string) {
751+
const appButtons = Array.from({ length: 3 }, (_, i) => ({
752+
index: i,
753+
depth: 1,
754+
type: 'android.widget.Button',
755+
bundleId: appBundleId,
756+
label: `Field ${String.fromCharCode(65 + i)}`,
757+
identifier: `${appBundleId}:id/field_${i + 1}`,
758+
rect: { x: 20, y: 100 + 60 * i, width: 300, height: 44 },
759+
hittable: true,
760+
}));
761+
return [
762+
...appButtons,
763+
// Bare overlay scrim: overlay-like `Dialog` type (covers the app) but no
764+
// label, no identifier, non-hittable — not itself a meaningful candidate.
765+
{
766+
index: appButtons.length,
767+
depth: 0,
768+
type: 'android.app.Dialog',
769+
bundleId: 'com.android.systemui',
770+
rect: { x: 0, y: 0, width: 402, height: 874 },
771+
},
772+
];
773+
}
774+
775+
test('buildReplayFailureDivergence: a mass-covered app with no actionable overlay node still returns non-empty screen.refs (#1264 occlusion fallback)', async () => {
776+
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-scrim-'));
777+
const sessionStore = new SessionStore(path.join(root, 'sessions'));
778+
const sessionName = 'default';
779+
const appBundleId = 'com.callstack.agentdevicelab';
780+
sessionStore.set(sessionName, makeAndroidSession(sessionName, { appBundleId }));
781+
782+
mockDispatchCommand.mockResolvedValue({
783+
nodes: bareScrimMassCoveringApp(appBundleId),
784+
truncated: false,
785+
backend: 'android',
786+
});
787+
788+
const action = {
789+
ts: 0,
790+
command: 'press',
791+
positionals: ['label="Field A"'],
792+
flags: {},
793+
result: { selectorChain: ['label="Field A"'] },
794+
};
795+
const divergence = await buildReplayFailureDivergence({
796+
error: { code: 'COMMAND_FAILED', message: 'not hittable' },
797+
action,
798+
index: 0,
799+
sourcePath: path.join(root, 'flow.ad'),
800+
sourceLine: 1,
801+
session: sessionStore.get(sessionName),
802+
sessionName,
803+
sessionStore,
804+
logPath: path.join(root, 'daemon.log'),
805+
responseLevel: 'default',
806+
planActions: [action],
807+
planDigest: 'test-plan-digest',
808+
});
809+
810+
expect(divergence.screen.state).toBe('available');
811+
const screen = divergence.screen as Extract<typeof divergence.screen, { state: 'available' }>;
812+
813+
// No non-covered meaningful candidate exists, but the capture is not empty —
814+
// the covered app content is surfaced rather than returning an empty report.
815+
expect(screen.refs.length).toBeGreaterThan(0);
816+
expect(screen.refs.some((ref) => ref.label === 'Field A')).toBe(true);
817+
});

0 commit comments

Comments
 (0)