Skip to content

Commit 4ae5441

Browse files
authored
feat(snapshot): carry a presented-surface fact through capture and settle (#3577)
* feat(ios): carry the presented-surface fact from capture producers to consumers A settle confirmation that has to prove a dismissal (#1831) has nothing to compare against: the capture contract says nothing about whether a modal presentation was over the app, so a consumer can only infer it from tree shape — and a tree shape that happens to hold before a tap is exactly what makes the confirmation arm on a no-op. iOS is where it matters most, because there an overlay is usually the same app process, so it moves neither the surface identity nor the foreground-owner fact. Publish the producer's own reading instead, on the state that is already built from the capture's unfiltered nodes: - `SnapshotState.presentedSurface` takes 'presented' | 'absent' | 'unknown', and `readPresentedSurfaceFact` is the one wire reader. A producer that inspected its whole subject and saw nothing presenting says 'absent'; one whose evidence cannot answer says 'unknown'; only a producer that never classified stays silent. Anything the vocabulary cannot name reads as 'unknown', so an unrecognised value can never become a false absence. - The fact rides the response envelope on every iOS capture route, including a digest, and travels with the nodes wherever a capture crosses a layer: the backend capture and interactor snapshot results, the daemon capture cache, the interactive-retry and deferred-re-observation nodes, the ref-target frame, and the settle baseline. A consumer that adopts an interaction's own capture therefore keeps the fact that capture established rather than re-deriving a weaker one. - Both settle routes mint that baseline from one function, which moves to capture-kit beside the assembly that builds the state it reads: the interaction route lives in `commands/` and the generic route in `daemon/`, and the layering forbids the second from importing the first. So the next fact a baseline owes cannot be carried by one route and dropped by the other. - The runner forwards the reading of the tier that answered. Its accessibility tree carries no window classes, so it states 'presented' from a resolved blocking system modal and never 'absent'; the shared classifier comes next. Consumers are untouched: nothing decides on the fact yet. * fix(kernel): keep the presented-surface read off the shared wire reader's closure Review of #3577 found four problems with how the fact crosses the wire. `readPresentedSurfaceFact` validated against `PRESENTED_SURFACE_STATES`, which made every eager consumer of `kernel/record` evaluate `kernel/snapshot.ts` as well. ADR 0019's loading-shape probe measures exactly that: thirteen carried entries grew by one module and 70 eager statements, which `scripts/__tests__/eager-closure-budgets.test.ts` refuses — the growth is a pre-existing module made eager, so no split explains it and shrinkage elsewhere cannot fund it. The reader now decides from a table it owns, keyed by `PresentedSurface`, so a vocabulary state without a row fails to compile rather than reading as `unknown`. The lookup is own-property, which also closes the inherited-member path an `in` test would have opened: a payload naming `constructor` is not a state. `PRESENTED_SURFACE_STATES` is frozen. `as const` was only a type-level promise on an array the reader trusted payloads against, so any consumer could move the boundary between a state and `unknown` for the whole process. The client normalized the fact on the default response path and returned a digest payload verbatim, which meant the same field was read two ways depending on `--response-level`. Passing a leveled payload through without reshaping it is not a licence to skip the fact's own vocabulary, so the digest path reads it through the same reader; a payload with no key stays byte-identical. Two doc comments claimed a capture "came from read" something. Both now name the producer that did the reading. * refactor(interaction): hand find's leaf the baseline surfaceScopedNodes minted PreresolvedInteractionTarget carries one SurfaceScopedNodes minted from find's own capture instead of mirroring nodes, surface and presentation field by field, so the adopting leaf's settle baseline is that value as-is. PresentedSurface becomes a plain union, and the fact travels only inside SnapshotState: the top-level BackendSnapshotResult field had no production writer. Moves the moving-target gate onto the capture-kit baseline mint. * fix(daemon): carry the presented-surface fact on the rest-proof baseline The moving-target rest tree copied the baseline shape by hand and left out presentedSurface, so the baseline after a rest proof differed from every other one. It now carries the same facts surfaceScopedNodes mints; it stays local because the snapshot-baseline import would grow the eager closure of most daemon entries.
1 parent ff3947c commit 4ae5441

45 files changed

Lines changed: 715 additions & 69 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,11 @@ struct DataPayload: Codable {
438438
var snapshotQuality: SnapshotQuality?
439439
/// Set when the capture describes an in-place system surface, not the app itself (#2438).
440440
var systemSurface: SystemSurfaceProvenancePayload?
441+
/// Whether this capture read a modal presentation over the app's own content (#3565). The runner's
442+
/// accessibility tree reports no UIKit class names, so a runner capture states `presented` from
443+
/// positive evidence and `unknown` otherwise; `absent` belongs to a producer that can read window
444+
/// structure, and a tier that classified nothing publishes no key at all.
445+
var presentedSurface: PresentedSurface?
441446
/// The keyboard band this capture measured, when the tier that answered reads keyboards at all
442447
/// (#2660). Absent means the query-sweep or private-AX tier answered, and the daemon's tap guard
443448
/// keeps deriving the band from the tree.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SystemModal.swift‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,17 @@ extension RunnerTests {
6969
nodes.append(contentNode)
7070
}
7171

72-
return DataPayload(nodes: nodes, truncated: false)
72+
// The probe resolved a blocking modal over the app, so this payload states its own subject rather
73+
// than leaving a reader to infer it from a tree that holds only the modal (#3565).
74+
return Self.blockingSystemModalPayload(nodes: nodes)
75+
}
76+
77+
/// The payload a resolved blocking system modal publishes: the modal's own tree, and the fact that
78+
/// the modal is what it describes. The probe is the one capture that states `presented` without
79+
/// reading a window structure — its tree holds only the modal, so nothing downstream could read the
80+
/// presentation out of it (#3565). Split from the probe body because that body needs a device.
81+
static func blockingSystemModalPayload(nodes: [PresentedNode]) -> DataPayload {
82+
DataPayload(nodes: nodes, truncated: false, presentedSurface: .presented)
7383
}
7484

7585
func firstBlockingSystemModal(
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
import XCTest
2+
import AgentDeviceSnapshotPresentation
3+
4+
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
5+
// #3565: a capture has to say whether it read a modal presentation over the app. The blocking
6+
// system-modal probe is the one capture that can state `presented` without reading any window
7+
// structure: the tree it publishes holds only the modal, so nothing downstream could read a
8+
// presentation out of it. Only the payload factory is pinned here, because the probe body that calls it
9+
// enumerates XCUIElements and needs a device.
10+
extension RunnerTests {
11+
func testResolvedBlockingSystemModalPayloadStatesAPresentedSurface() {
12+
XCTAssertEqual(Self.blockingSystemModalPayload(nodes: []).presentedSurface, .presented)
13+
}
14+
15+
// Silence is the answer a consumer must not be able to confuse with absence: a capture the runner
16+
// never classified (an older runner, a tier that published no reading) encodes no key at all.
17+
func testUnclassifiedSnapshotPayloadEncodesNoPresentedSurfaceKey() throws {
18+
let encoded = try JSONEncoder().encode(DataPayload(nodes: [], truncated: false))
19+
let object = try XCTUnwrap(
20+
JSONSerialization.jsonObject(with: encoded) as? [String: Any],
21+
"the snapshot payload encodes as an object"
22+
)
23+
XCTAssertNil(object["presentedSurface"])
24+
}
25+
}
26+
#endif
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
import Foundation
2+
3+
/// Whether this capture read a modal presentation over the app's own content (#3565). The producer owns
4+
/// the answer, and `absent` is a reading rather than a default, because a consumer arms a dismissal
5+
/// confirmation on it: a capture whose evidence cannot answer says `unknown` instead of staying silent,
6+
/// since silence is also what a producer that never classified publishes.
7+
///
8+
/// Mirrors the TypeScript `PresentedSurface` in `@agent-device/kernel/snapshot`.
9+
public enum PresentedSurface: String, Codable, Equatable {
10+
/// A presentation covers the content this capture describes.
11+
case presented
12+
/// Every window this capture read showed only its own content container.
13+
case absent
14+
/// The capture supports neither answer.
15+
case unknown
16+
}

‎packages/capture-kit/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,10 @@
230230
"types": "./src/snapshot/scroll-edge-state.ts",
231231
"default": "./src/snapshot/scroll-edge-state.ts"
232232
},
233+
"./snapshot-baseline": {
234+
"types": "./src/snapshot-baseline.ts",
235+
"default": "./src/snapshot-baseline.ts"
236+
},
233237
"./snapshot-chrome": {
234238
"types": "./src/snapshot-chrome.ts",
235239
"default": "./src/snapshot-chrome.ts"
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import { expect, test } from 'vitest';
2+
import type { SnapshotState } from '@agent-device/kernel/snapshot';
3+
import { surfaceScopedNodes } from './snapshot-baseline.ts';
4+
5+
const nodes = [{ index: 0, type: 'Application', ref: 'e1' }];
6+
7+
/**
8+
* #3565 and #2438: this is the one mint both settle routes take their baseline from, so every fact a
9+
* capture established about what it read has to leave here with the nodes. A baseline that lost the
10+
* presented-surface fact would read to #1831's dismissal confirmation as "no producer spoke" instead of
11+
* the reading the capture actually took.
12+
*/
13+
test('the mint carries the presented-surface fact beside the nodes', () => {
14+
for (const presentedSurface of ['presented', 'absent', 'unknown'] as const) {
15+
expect(surfaceScopedNodes(snapshot({ presentedSurface }))).toEqual({
16+
nodes,
17+
presentedSurface,
18+
});
19+
}
20+
});
21+
22+
test('the mint carries no presented surface a capture never read', () => {
23+
expect(surfaceScopedNodes(snapshot({}))).toEqual({ nodes });
24+
});
25+
26+
test('the mint carries the system surface the nodes describe', () => {
27+
expect(
28+
surfaceScopedNodes(snapshot({ iosSystemSurfaceBundleId: 'com.apple.SafariViewService' })),
29+
).toEqual({ nodes, surfaceBundleId: 'com.apple.SafariViewService' });
30+
});
31+
32+
test('a session with no stored snapshot still mints an empty baseline', () => {
33+
expect(surfaceScopedNodes(undefined)).toEqual({ nodes: [] });
34+
});
35+
36+
function snapshot(fields: Partial<SnapshotState>): SnapshotState {
37+
return { nodes, createdAt: 0, presentationKey: '', ...fields } as SnapshotState;
38+
}
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import type { SnapshotState } from '@agent-device/kernel/snapshot';
2+
import type { SurfaceScopedNodes } from '@agent-device/contracts/interaction';
3+
4+
/**
5+
* Mints the one baseline value a post-action comparison carries: the captured nodes together with the
6+
* surface they describe and the presentation the capture read (#2438, #3565).
7+
*
8+
* Both settle routes take their baseline from here — the element-targeted interaction route and the
9+
* generic daemon route (`scroll`/`back --settle`) — so a fact a capture established cannot be carried
10+
* by one route and dropped by the other. The pair sits beside {@link buildSnapshotState} because that
11+
* is the layer both routes can reach: the interaction route lives in `commands/`, the generic route in
12+
* `daemon/`, and the layering forbids the second from importing the first.
13+
*
14+
* One value, never two channels: every boundary that carries a baseline carries the type this returns,
15+
* so nodes cannot arrive without the surface they describe.
16+
*/
17+
export function surfaceScopedNodes(
18+
snapshot:
19+
| Pick<SnapshotState, 'nodes' | 'iosSystemSurfaceBundleId' | 'presentedSurface'>
20+
| undefined,
21+
): SurfaceScopedNodes {
22+
return {
23+
nodes: snapshot?.nodes ?? [],
24+
...(snapshot?.iosSystemSurfaceBundleId
25+
? { surfaceBundleId: snapshot.iosSystemSurfaceBundleId }
26+
: {}),
27+
...(snapshot?.presentedSurface ? { presentedSurface: snapshot.presentedSurface } : {}),
28+
};
29+
}

‎packages/capture-kit/src/snapshot-state.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import {
1010
snapshotPresentationOptionsFromFlags,
1111
snapshotRefMinting,
1212
type IosTargetActivation,
13+
type PresentedSurface,
1314
type RawSnapshotNode,
1415
type SnapshotBackend,
1516
type SnapshotCaptureProvenance,
@@ -55,6 +56,11 @@ export function buildSnapshotState(
5556
keyboard?: SnapshotKeyboardBandFact;
5657
/** The box the producer measured the rects in (#3182), carried to the response that reads this state. */
5758
viewport?: SnapshotViewportSize;
59+
/**
60+
* iOS: the producer's reading of a modal presentation over the app (#3565), decided on the
61+
* acquired tree so every projection of one screen carries the same value.
62+
*/
63+
presentedSurface?: PresentedSurface;
5864
} & SnapshotCaptureProvenance,
5965
flags:
6066
| (Pick<CommandFlags, 'snapshotDepth' | 'snapshotInteractiveOnly' | 'snapshotRaw'> &
@@ -92,6 +98,7 @@ export function buildSnapshotState(
9298
? { comparisonKey: iosSnapshotComparisonIdentityKey(data.comparisonIdentity) }
9399
: {}),
94100
...(data.systemSurface ? { iosSystemSurfaceBundleId: data.systemSurface.bundleId } : {}),
101+
...(data.presentedSurface ? { presentedSurface: data.presentedSurface } : {}),
95102
...(data.targetActivation ? { targetActivation: data.targetActivation } : {}),
96103
...(data.keyboard ? { keyboard: data.keyboard } : {}),
97104
...(data.viewport ? { viewport: data.viewport } : {}),

‎packages/contracts/src/client-capture.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import type { SessionSurface } from './session-surface.ts';
44
import type { PublicSnapshotCaptureAnnotations } from './snapshot-capture-annotations.ts';
55
import type { SnapshotDiagnosticsSummary } from './snapshot-diagnostics.ts';
66
import type {
7+
PresentedSurface,
78
SnapshotCommandOptionFields,
89
SnapshotKeyboardBandFact,
910
SnapshotNode,
@@ -78,6 +79,13 @@ export type CaptureSnapshotResult = {
7879
* screen size only; content-safe gesture bounds stay with #1821.
7980
*/
8081
viewport?: SnapshotViewportSize;
82+
/**
83+
* iOS: whether the capture read a modal presentation over the app (#3565), decided on the acquired
84+
* tree before interactive filtering, compaction, depth, or scope. `'absent'` is a positive reading;
85+
* a capture whose producer could not decide — or did not speak — reports `'unknown'`, and absence
86+
* of the field means a non-iOS producer or an older runner published nothing.
87+
*/
88+
presentedSurface?: PresentedSurface;
8189
/**
8290
* Screenshot captured automatically when the semantic snapshot was sparse.
8391
* Remote clients receive a materialized local path through the daemon artifact channel.

‎packages/contracts/src/interaction.ts‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
1-
import type { Point, SnapshotKeyboardBandFact, SnapshotNode } from '@agent-device/kernel/snapshot';
1+
import type {
2+
PresentedSurface,
3+
Point,
4+
SnapshotKeyboardBandFact,
5+
SnapshotNode,
6+
} from '@agent-device/kernel/snapshot';
27
import type { ResponseCost } from '@agent-device/kernel/contracts';
38
import type { ClickButton } from './click-button.ts';
49
import type { FillUnconfirmedVerification } from './fill-evidence.ts';
@@ -119,16 +124,14 @@ export type PreresolvedInteractionTarget = {
119124
ref: string;
120125
/** The node the caller resolved, after the caller's own promotion. */
121126
node: SnapshotNode;
122-
/** The tree `node` came from — the guards read its siblings for occlusion/viewport. */
123-
nodes: SnapshotNode[];
124127
/**
125-
* The in-place iOS system surface `nodes` describes (#2438), absent for ordinary app content.
126-
* Travels with the tree so the adopting consumer's post-action comparison knows which surface
127-
* its baseline came from.
128+
* The tree `node` came from, minted by `surfaceScopedNodes` from the caller's capture. The guards
129+
* read its siblings for occlusion/viewport, and the adopting leaf's post-action baseline is this
130+
* value as-is, so the surface (#2438) and presentation (#3565) facts travel with the nodes.
128131
*/
129-
iosSystemSurfaceBundleId?: string;
132+
tree: SurfaceScopedNodes;
130133
/**
131-
* The keyboard band the capture of `nodes` measured, carried so the adopting consumer's keyboard
134+
* The keyboard band the capture of `tree` measured, carried so the adopting consumer's keyboard
132135
* guard reads the same band the tree was captured with instead of re-deriving one (#2660).
133136
*/
134137
keyboard?: SnapshotKeyboardBandFact;
@@ -148,6 +151,13 @@ export type SurfaceScopedNodes = {
148151
nodes: SnapshotNode[];
149152
/** Bundle id of the in-place iOS system surface; absent for ordinary app content. */
150153
surfaceBundleId?: string;
154+
/**
155+
* Whether the producer that captured these nodes read a modal presentation over the app (#3565).
156+
* The settle baseline is carried in this type, so the fact has to travel with the nodes: a consumer
157+
* that arms #1831's dismissal confirmation reads it here instead of guessing from the tree.
158+
* Absent means the capture's producer published no fact.
159+
*/
160+
presentedSurface?: PresentedSurface;
151161
};
152162

153163
/** Which tree a resolved `@ref`'s node was read from (ADR 0014 step 5). */

0 commit comments

Comments
 (0)