Skip to content

Commit 5a1c7cf

Browse files
committed
DRAFT fix(settle): arm the confirmation on the dismissed surface, not on screen replacement
DRAFT / BLOCKED BY #3565 — do not merge. The rule below reads a property of the baseline CAPTURE, and #3565 replaces it with a fact stamped by the capture producer. Kept as the reviewed starting point and the device evidence, not as a shippable rule. The layer under this one arms on a complete-viewport post-action projection plus a <50% identity overlap. A tab switch and a push are exactly that: both screens project completely and the rows of one tab are not the rows of the next, so ordinary navigation pays the 1.5s floor (#3470's whole complaint). This narrows the trigger to the pre-action tree showing a surface presented over app content, which an interactive capture reports as the surface alone: a presented pageSheet projects 4 nodes, a system permission prompt 5 nodes under its own Alert root, while an ordinary screen of the same app projects 24+ including its nav or tab bar. Whether the surface was actually replaced stays per-capture. Why it is still a draft: on device the same automation sheet projects 4 nodes through an interactive capture and 6 through a full one, so arming survives only on the route whose baseline happens to be the interactive flavour. A dismissal preceded by a plain `snapshot`, or taken on the generic `back` route, loses the #1831 coverage this rule exists to keep. The chrome heuristic was considered and rejected: it regresses modals that carry their own nav bar against `main`. The durable rule is a typed `presentedSurface` fact from the capture producer (#3565); reading it must keep this parent's arming when the producer answers 'unknown', and the count bound and the Alert special case both go with `showsPresentedSurface`. The three navigation cases are mutation-checked against the parent's arming: restoring the viewport-completeness arm fails exactly those three and nothing else.
1 parent c381817 commit 5a1c7cf

4 files changed

Lines changed: 422 additions & 141 deletions

File tree

‎src/commands/interaction/runtime/stable-capture.fixtures.ts‎

Lines changed: 176 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import type { SnapshotState } from '@agent-device/kernel/snapshot';
1+
import type { RawSnapshotNode, Rect, SnapshotState } from '@agent-device/kernel/snapshot';
22
import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures';
33

44
export function elementSettingsSnapshot(offscreenRowYs: number[]): SnapshotState {
@@ -133,3 +133,178 @@ export const elementSettledRoomSnapshot = makeSnapshotState(
133133
],
134134
{ backend: 'xctest', snapshotQuality: { state: 'healthy', backend: 'tree' } },
135135
);
136+
137+
// Screen-replacement fixtures in the SHAPE a live iPhone 17 (iOS 27) session projects — an
138+
// `Application` root, the screen's own content, and a tab bar whose buttons keep the same identities
139+
// across tabs — kept compact on purpose: a live screen of this app reports 24-34 nodes and the same
140+
// proportion here is 18 (`snapshot -i`, October 2026). What the tests need is the ratio, not the
141+
// volume: each fixture is a BROAD replacement of its siblings — fewer than half the node identities
142+
// survive — which is what made the old complete-viewport-projection arm treat an ordinary tab switch
143+
// or push as a screen replacement (#3470).
144+
145+
const TAB_BAR_LABELS = ['Home', 'Catalog', 'Form', 'Settings'];
146+
147+
function iosScreenSnapshot(
148+
content: Array<{ type: string; label: string; rect: Rect }>,
149+
): SnapshotState {
150+
const nodes: RawSnapshotNode[] = [
151+
{
152+
index: 0,
153+
depth: 0,
154+
type: 'Application',
155+
label: 'Agent Device Tester',
156+
rect: { x: 0, y: 0, width: 402, height: 874 },
157+
},
158+
...content.map((node, offset) => ({
159+
index: offset + 1,
160+
depth: 1,
161+
parentIndex: 0,
162+
...node,
163+
})),
164+
];
165+
const tabBarIndex = nodes.length;
166+
nodes.push({
167+
index: tabBarIndex,
168+
depth: 1,
169+
parentIndex: 0,
170+
type: 'TabBar',
171+
label: 'Tab Bar',
172+
rect: { x: 0, y: 791, width: 402, height: 83 },
173+
});
174+
TAB_BAR_LABELS.forEach((label, offset) => {
175+
nodes.push({
176+
index: nodes.length,
177+
depth: 2,
178+
parentIndex: tabBarIndex,
179+
type: 'Button',
180+
label,
181+
rect: { x: 12 + offset * 96, y: 795, width: 92, height: 54 },
182+
});
183+
});
184+
return makeSnapshotState(nodes, {
185+
backend: 'xctest',
186+
snapshotQuality: { state: 'healthy', backend: 'tree' },
187+
});
188+
}
189+
190+
/** A tab's list screen: the shared tab bar plus that tab's own rows. */
191+
export function elementCatalogTabSnapshot(): SnapshotState {
192+
return iosScreenSnapshot(
193+
Array.from({ length: 12 }, (_, offset) => ({
194+
type: 'Cell',
195+
label: `Catalog product ${offset + 1}`,
196+
rect: { x: 0, y: 120 + offset * 90, width: 402, height: 84 },
197+
})),
198+
);
199+
}
200+
201+
/** The tab you switch to: different rows, same tab bar, same app root. */
202+
export function elementSettingsTabSnapshot(): SnapshotState {
203+
return iosScreenSnapshot(
204+
Array.from({ length: 12 }, (_, offset) => ({
205+
type: 'Button',
206+
label: `Open settings fixture ${offset + 1}`,
207+
rect: { x: 16, y: 120 + offset * 60, width: 370, height: 45 },
208+
})),
209+
);
210+
}
211+
212+
/** The screen a push opens: its own nav bar and content, and the tab bar's row is gone. */
213+
export function elementPushedDetailSnapshot(): SnapshotState {
214+
return makeSnapshotState(
215+
[
216+
{
217+
index: 0,
218+
depth: 0,
219+
type: 'Application',
220+
label: 'Agent Device Tester',
221+
rect: { x: 0, y: 0, width: 402, height: 874 },
222+
},
223+
{
224+
index: 1,
225+
depth: 1,
226+
parentIndex: 0,
227+
type: 'NavigationBar',
228+
label: 'Automation',
229+
rect: { x: 0, y: 62, width: 402, height: 54 },
230+
},
231+
{
232+
index: 2,
233+
depth: 2,
234+
parentIndex: 1,
235+
type: 'Button',
236+
label: '(tabs)',
237+
rect: { x: 16, y: 62, width: 44, height: 44 },
238+
},
239+
...['Open automation sheet', 'Press canary', 'Long press canary'].map((label, offset) => ({
240+
index: 3 + offset,
241+
depth: 1,
242+
parentIndex: 0,
243+
type: 'Button',
244+
label,
245+
rect: { x: 16, y: 240 + offset * 120, width: 370, height: 45 },
246+
})),
247+
],
248+
{ backend: 'xctest', snapshotQuality: { state: 'healthy', backend: 'tree' } },
249+
);
250+
}
251+
252+
/**
253+
* A system permission prompt's own projection: the runner roots it at an `Alert` node and the app
254+
* content it presents over is not in the tree. Larger than `TINY_STABLE_TREE_NODE_COUNT`, so only the
255+
* root names it (#2660).
256+
*/
257+
export function systemPermissionPromptSnapshot(): SnapshotState {
258+
return makeSnapshotState(
259+
[
260+
{
261+
index: 0,
262+
depth: 0,
263+
type: 'Alert',
264+
label: 'Allow “Agent Device Tester” to access your microphone?',
265+
rect: { x: 40, y: 309, width: 322, height: 283 },
266+
},
267+
{
268+
index: 1,
269+
depth: 1,
270+
parentIndex: 0,
271+
type: 'StaticText',
272+
label: 'Allow Agent Device Tester to exercise microphone permissions.',
273+
rect: { x: 60, y: 420, width: 282, height: 42 },
274+
},
275+
{
276+
index: 2,
277+
depth: 1,
278+
parentIndex: 0,
279+
type: 'StaticText',
280+
label: 'Microphone access lets the lab record a clip.',
281+
rect: { x: 60, y: 470, width: 282, height: 38 },
282+
},
283+
{
284+
index: 3,
285+
depth: 1,
286+
parentIndex: 0,
287+
type: 'Button',
288+
label: 'Don’t Allow',
289+
rect: { x: 60, y: 528, width: 130, height: 48 },
290+
},
291+
{
292+
index: 4,
293+
depth: 1,
294+
parentIndex: 0,
295+
type: 'Button',
296+
label: 'Allow',
297+
rect: { x: 212, y: 528, width: 130, height: 48 },
298+
},
299+
{
300+
index: 5,
301+
depth: 1,
302+
parentIndex: 0,
303+
type: 'Button',
304+
label: 'Learn More',
305+
rect: { x: 136, y: 580, width: 130, height: 48 },
306+
},
307+
],
308+
{ backend: 'xctest', snapshotQuality: { state: 'healthy', backend: 'tree' } },
309+
);
310+
}

‎src/commands/interaction/runtime/stable-capture.test.ts‎

Lines changed: 65 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,14 @@ import { createAgentDevice } from '../../../runtime.ts';
99
import { localCommandPolicy } from '@agent-device/contracts/command-policy';
1010
import { runStableCaptureLoop } from './stable-capture.ts';
1111
import {
12+
elementCatalogTabSnapshot,
13+
elementPushedDetailSnapshot,
1214
elementSettingsSnapshot,
1315
elementSettledRoomSnapshot,
16+
elementSettingsTabSnapshot,
1417
elementThreadsNoticeSnapshot,
1518
elementTransientRoomSnapshot,
19+
systemPermissionPromptSnapshot,
1620
} from './stable-capture.fixtures.ts';
1721
import { createMemorySessionStore } from '../../../command-runtime/runtime-factory.ts';
1822

@@ -326,6 +330,58 @@ test('settle confirms a tiny modal replacement through a partial post-action pro
326330
);
327331
});
328332

333+
test('settle confirms the dismissal of a surface the projection names outright', async () => {
334+
// A blocking system prompt projects 6 nodes — over `TINY_STABLE_TREE_NODE_COUNT` — and names
335+
// itself with its `Alert` root (#2660), so that root is what arms the confirmation.
336+
const prompt = systemPermissionPromptSnapshot();
337+
const { runtime } = transitionRuntime({
338+
presented: prompt,
339+
dismissesTo: elementCatalogTabSnapshot(),
340+
settledAtMs: 1_200,
341+
});
342+
343+
const outcome = await runStableCaptureLoop(
344+
runtime,
345+
{ session: 'default' },
346+
{ quietMs: 150, timeoutMs: 5_000, broadTransitionBaselineNodes: prompt.nodes },
347+
);
348+
349+
assert.equal(outcome.settled, true);
350+
assert.ok(outcome.waitedMs >= 1_500, `settled prompt dismissal after ${outcome.waitedMs}ms`);
351+
});
352+
353+
// The three navigation cases #3470 complains about. Each is a BROAD replacement by the identity
354+
// ratio — both screens project a complete viewport and share under half their node identities — so
355+
// they are exactly what the parent layer still confirms for 1.5s, and what arming on the dismissed
356+
// surface is meant to stop charging.
357+
for (const [label, from, to] of [
358+
['a tab switch', elementCatalogTabSnapshot(), elementSettingsTabSnapshot()],
359+
['a push', elementCatalogTabSnapshot(), elementPushedDetailSnapshot()],
360+
[
361+
// The other side of the size bound: the screen a push LANDS ON can be tiny, and that must not
362+
// arm the confirmation either — only the surface being dismissed is the trigger.
363+
'a push onto a nearly-empty screen',
364+
elementCatalogTabSnapshot(),
365+
elementThreadsNoticeSnapshot,
366+
],
367+
] as const) {
368+
test(`settle keeps the quiet window across ${label}`, async () => {
369+
const { runtime } = staticSnapshotRuntime(to);
370+
371+
const outcome = await runStableCaptureLoop(
372+
runtime,
373+
{ session: 'default' },
374+
{ quietMs: 500, timeoutMs: 5_000, broadTransitionBaselineNodes: from.nodes },
375+
);
376+
377+
assert.equal(outcome.settled, true);
378+
assert.ok(
379+
outcome.waitedMs < 800,
380+
`${label} settled after ${outcome.waitedMs}ms with ${outcome.captures} captures`,
381+
);
382+
});
383+
}
384+
329385
test('a dismissal confirmation is not dropped by a later empty projection', async () => {
330386
// A confirmation is a fact about the transition, so once armed it holds the floor for the rest of
331387
// the run: later captures that cannot support the comparison (here, empty projections that match
@@ -534,10 +590,16 @@ function transitionRuntime(
534590
rootlessUntilMs?: number;
535591
/** 1-based capture indices that declare a non-XCTest capture channel. */
536592
otherChannelAtCaptures?: number[];
593+
/** The surface the action dismisses; defaults to #1831's notice. */
594+
presented?: SnapshotState;
595+
/** What the screen under it looks like once the dismissal completes. */
596+
dismissesTo?: SnapshotState;
537597
} = {},
538598
) {
539599
const captureBackend = options.captureBackend ?? 'tree';
540600
const settledAtMs = options.settledAtMs ?? 800;
601+
const presented = options.presented ?? elementThreadsNoticeSnapshot;
602+
const dismissesTo = options.dismissesTo ?? elementSettledRoomSnapshot;
541603
let elapsedMs = 0;
542604
let captures = 0;
543605
const clock = {
@@ -561,9 +623,9 @@ function transitionRuntime(
561623
};
562624
}
563625
const snapshot = keepsBaseline
564-
? elementThreadsNoticeSnapshot
626+
? presented
565627
: withCaptureBackend(
566-
elapsedMs < settledAtMs ? elementTransientRoomSnapshot : elementSettledRoomSnapshot,
628+
elapsedMs < settledAtMs ? elementTransientRoomSnapshot : dismissesTo,
567629
captureBackend,
568630
);
569631
const rootless =
@@ -589,7 +651,7 @@ function transitionRuntime(
589651
} satisfies AgentDeviceBackend,
590652
artifacts: createLocalArtifactAdapter(),
591653
sessions: createMemorySessionStore([
592-
{ name: 'default', snapshot: options.sessionSnapshot ?? elementThreadsNoticeSnapshot },
654+
{ name: 'default', snapshot: options.sessionSnapshot ?? presented },
593655
]),
594656
policy: localCommandPolicy(),
595657
clock,

‎src/commands/interaction/runtime/stable-capture.ts‎

Lines changed: 46 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ import type {
55
SnapshotQualityVerdict,
66
SnapshotState,
77
} from '@agent-device/kernel/snapshot';
8-
import { isViewportRootNode } from '@agent-device/contracts/snapshot';
98
import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract';
109
import { now, sleep } from '../../runtime-common.ts';
1110
import {
@@ -407,12 +406,13 @@ type StableCaptureConfirmation = {
407406
};
408407

409408
/**
410-
* The pre-action tree to confirm against, or `undefined` when the capture in hand cannot support the
411-
* comparison: the capture must prove an iOS XCTest session (`SnapshotState.backend` is optional at
412-
* the runtime boundary, so the bound platform stays authoritative when a capture omits that
413-
* provenance; a declared non-XCTest backend still wins so this confirmation cannot leak onto another
414-
* capture implementation), and it must be trustworthy evidence about the screen. `undefined` asks the
415-
* next capture, rather than ruling the transition out for the run.
409+
* The pre-action surface to confirm against, or `undefined` when the transition in front of the loop
410+
* is not one this confirmation exists for: the chrome-free signal for the pre-action tree, when the
411+
* capture proves an iOS XCTest session (`SnapshotState.backend` is optional at the runtime boundary,
412+
* so the bound platform stays authoritative when a capture omits that provenance; a declared
413+
* non-XCTest backend still wins so this confirmation cannot leak onto another capture
414+
* implementation) and that tree shows a surface presented over app content. Whether the surface was
415+
* actually replaced stays per-capture — the first capture after a dismissal can still show it.
416416
*/
417417
function broadTransitionBaseline(params: {
418418
baselineNodes: SnapshotNode[];
@@ -422,18 +422,45 @@ function broadTransitionBaseline(params: {
422422
}): StableCaptureSignal | undefined {
423423
const { baselineNodes, capture, platform, appBundleId } = params;
424424
if (!isBoundIosXCTestCapture(capture.backend, platform)) return undefined;
425-
// A broad *screen* replacement needs a complete post-action viewport projection. The baseline
426-
// comes from settle's authoritative pre-action capture, but a presented iOS modal can omit its
427-
// Application root and still be the complete interaction surface. Requiring the root only from
428-
// the candidate prevents scoped/synthetic post-action fragments from extending settle latency.
429-
const hasCompleteViewportProjection = capture.nodes.some(isViewportRootNode);
430-
// Tiny pre-action surfaces are the other trustworthy completeness signal: iOS presents alerts
431-
// and sheets as a handful of nodes, and their first post-dismissal capture can temporarily omit
432-
// the viewport root. Treating that rootless replacement as an arbitrary scoped projection lets
433-
// a coherent transitional tree win the default 500ms quiet window.
434-
const hasTinyPresentedBaseline = baselineNodes.length <= TINY_STABLE_TREE_NODE_COUNT;
435-
if (!hasCompleteViewportProjection && !hasTinyPresentedBaseline) return undefined;
436-
return stableCaptureComparisonSignal(capture, baselineNodes, appBundleId);
425+
const surfaceNodes = withoutSettleChrome(baselineNodes, appBundleId);
426+
return showsPresentedSurface(surfaceNodes)
427+
? stableCaptureComparisonSignal(capture, baselineNodes, appBundleId)
428+
: undefined;
429+
}
430+
431+
/**
432+
* Whether a tree shows a presented surface rather than an app screen. UIKit presents a modal over the
433+
* content it presents from, and an interactive capture of a presented surface offers only the
434+
* surface: the content underneath is not hittable, so it drops out of the projection. Live ground
435+
* truth (iPhone 18 Pro sim, `snapshot -i`): a presented `pageSheet` projects to 4 nodes under the
436+
* app root and a system permission prompt to 5 nodes under its own `Alert` root — while an ordinary
437+
* screen of the same app projects 24+ nodes including its own nav or tab bar.
438+
*
439+
* `TINY_STABLE_TREE_NODE_COUNT` is the size bound for that reading, shared with the tiny-tree read.
440+
* It is also what stops an ordinary screen from arming the confirmation — which is the #3470
441+
* complaint about tab switches and pushes. An empty projection never reaches the comparison at all:
442+
* `stableCaptureSignalsHaveBroadReplacement` refuses a side with no nodes.
443+
*
444+
* The `Alert` arm names the one surface the projection says outright: the runner projects a blocking
445+
* system prompt as its own `Alert` accessibility root, and a prompt with more actions than the bound
446+
* is still a prompt.
447+
*
448+
* DRAFT — BLOCKED BY #3565. This rule reads a property of the BASELINE CAPTURE, not of the screen:
449+
* on device (see the evidence on #3512) the same automation sheet projects 4 nodes through an
450+
* interactive capture and 6 through a full one, so arming survives only on the route whose baseline
451+
* happens to be the interactive flavour, and a dismissal preceded by a plain `snapshot` or taken on
452+
* the generic `back` route loses the #1831 coverage this rule exists to keep. The durable rule is a
453+
* presented-surface fact stamped by the capture producer (#3565); `presentedSurface: 'unknown'` must
454+
* keep this layer's parent's arming, and the count bound and the `Alert` special case both go with
455+
* this function.
456+
*/
457+
function showsPresentedSurface(nodes: readonly SnapshotNode[]): boolean {
458+
return nodes.length <= TINY_STABLE_TREE_NODE_COUNT || isBlockingSystemAlertRoot(nodes[0]);
459+
}
460+
461+
/** The root a blocking system prompt's own projection carries (#2660), if this tree has one. */
462+
function isBlockingSystemAlertRoot(root: SnapshotNode | undefined): boolean {
463+
return root !== undefined && root.parentIndex === undefined && root.type === 'Alert';
437464
}
438465

439466
/**

0 commit comments

Comments
 (0)