Skip to content

Commit 5dee3be

Browse files
authored
fix(ios): carry the snapshot viewport across the runner boundary as a declared fact (#2908)
* fix(ios): carry the snapshot viewport across the runner boundary as a declared fact The runner spelled "viewport unknown" as CGRect.infinite, which SnapshotGeometry.isGeometricallyActionable read as "everything is actionable" before computing any center, while the host-AX path models the same state as IosViewportEvidence and declines to publish the bit. One state, two encodings, opposite directions, across a language boundary. SnapshotViewport now carries the three cases the host already uses -- reported, derived, and missing { reason } -- and refuses an unusable box at declaration, so no caller has to re-check what it was handed. The unknown-viewport policy is stated once, at the Swift decision site, and fails CLOSED: no box means no supportable claim about where a tap lands. Clipping stays a separate question from containment; a capture with no box has no clip and no root clip for the cumulative invariant to violate, which is an absent answer rather than an unbounded one. * test(ios): pin the hittable predicate with a cross-language vector table The fixed 320x240 fold fixture cannot reach an edge-of-viewport center, a node rect neither language's null/empty check refuses, or an unknown viewport, so the predicate's two implementations disagreed on two of them: CGRect.infinite is built of finite Doubles and its center is (0,0), and a negative width passes an empty check while its center lands inside -- both actionable in Swift and not actionable in TypeScript. The private AX bridge's JSON frame parser can hand over either. contracts/fixtures/snapshot-actionability-policy.json is replayed by ActionabilityPolicyTests (Swift) and by scripts/ios-snapshot-differential .test.ts (TypeScript), which also asserts the host's own instrument: resolveViewportEvidence refuses a regular presentation without a positive finite viewport, which is why the TypeScript predicate has no unknown viewport case. Rows come in pairs sharing one center so a `false` that the guard, rather than containment, decides cannot go quiet. * refactor(ios): plot the quarter-turn test through the shared rect predicate `SnapshotGeometrySpace.isPlottable` and `SnapshotGeometry.isPositiveFinite` were the same five comparisons, written twice in one package, which is how #2891 started: one question with two encodings that can drift apart. `.null`, an inverted box, and an infinite box are refused identically by both, so the coordinate-space table needs no new rows. * fix(ios): make the viewport type the only place a box becomes a fact `SnapshotViewport.reported(box:)` normalised an unusable box to `.missing(reason: .invalid)`, and the commit said no caller had to re-check what it was handed — while `.reported(rect:)` stayed a public case that accepted `CGRect.infinite` on the way past that claim. The associated value is now a box with an internal initialiser, so outside this package a viewport fact is only reachable through the two factories, and the case labels cannot be named around them. MSG2 * test(ios): write down the one input the two predicates cannot share The table replayed `{"infinite": true}` as `CGRect.infinite` in Swift and as a rect with infinite components in TypeScript, so its infinity row never put the same number in front of both sides. `CGRect.infinite` is made of finite Doubles: TypeScript would call those components actionable, and only the platform that can name the value can refuse it. The row is now Swift-only and carries `asymmetry` naming that, every row declares both `swift` and `typescript`, and a shared non-finite row replaces it with numbers both sides actually see. Also states three claims the code had not been checked against: a published `hittable: false` is retained rather than filtered out, the runner's viewport fact decides only runner bits because the host re-derives its own from the capture's root node, and `viewportFromRoot` — not only `resolveViewportEvidence` — is what makes the TypeScript `viewport` argument total. * fix(ios): leave hittable absent without a viewport box; only a reported viewport anchors a rotation A capture with no viewport box can still decide a disabled or degenerate node, but not containment, so the runner now omits hittable instead of declaring false, matching the host bridge's undefined bit. The interface orientation moves into SnapshotViewport.reported, so a derived or missing viewport cannot anchor a rotation, and SnapshotGeometrySpace takes the viewport fact rather than a sentinel CGRect. * chore(gates): route the actionability vector table to the differential lane contracts/fixtures/snapshot-actionability-policy.json is read by both sides of scripts/ios-snapshot-differential, and packages/kernel/src/rect.ts holds the TypeScript predicate that table replays, so a change to either must select that lane rather than rely on the generic golden-table rule reaching only the unit and Swift builds.
1 parent 6508ef8 commit 5dee3be

33 files changed

Lines changed: 837 additions & 249 deletions

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 14 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -217,12 +217,11 @@ extension RunnerTests {
217217
}
218218

219219
let rootFrame = privateAXRect(root["frame"])
220-
let geometry = privateAXSnapshotGeometry(
220+
let viewport = privateAXSnapshotViewport(
221221
app: app,
222222
bundleId: target.bundleId,
223223
rootFrame: rootFrame
224224
)
225-
let viewport = geometry.viewport
226225
let nodes = privateAXAcquisition(
227226
rawRoot: root,
228227
hint: hint
@@ -259,8 +258,7 @@ extension RunnerTests {
259258
customActions: Self.privateAXCustomActionCoverage(
260259
response[RunnerAXSnapshotCustomActionsKey]
261260
),
262-
viewport: viewport,
263-
interfaceOrientation: geometry.interfaceOrientation
261+
viewport: viewport
264262
)
265263
#else
266264
return nil
@@ -275,37 +273,32 @@ extension RunnerTests {
275273
!hasAbandonedMainThreadWork() && !isSnapshotXCTestChannelPenalized(bundleId: bundleId)
276274
}
277275

278-
/// The geometry this tier may anchor a rotation on. The bridge's own root frame is one more
279-
/// reported box rather than the app's frame, so a capture anchored on it reports no interface
280-
/// orientation and normalizes nothing: rotated system surfaces then stay as reported, which the
281-
/// consumers already treat as geometry they cannot measure (#2612).
282-
private func privateAXSnapshotGeometry(
276+
/// The app's reported viewport when XCTest can read it, else the bridge's own root frame declared
277+
/// `.derived`, which cannot anchor a rotation: rotated system surfaces then stay as reported (#2612).
278+
private func privateAXSnapshotViewport(
283279
app: XCUIApplication,
284280
bundleId: String?,
285281
rootFrame: CGRect
286-
) -> (viewport: CGRect, interfaceOrientation: Int) {
287-
let fallback = rootFrame.isEmpty ? CGRect.infinite : rootFrame
282+
) -> SnapshotViewport {
283+
let fallback = SnapshotViewport.derived(box: rootFrame)
288284
guard shouldReadPrivateAXViewportViaXCTest(bundleId: bundleId) else {
289-
return (fallback, RunnerInterfaceOrientation.unknown)
285+
return fallback
290286
}
291287
do {
292-
let anchor = try runMainThreadWork(
288+
let reported = try runMainThreadWork(
293289
"private_ax_viewport",
294290
timeout: 1,
295291
timeoutError: snapshotMainThreadTimeoutError("reading private AX viewport")
296292
) {
297-
(
298-
viewport: self.safeSnapshotViewport(app: app),
299-
interfaceOrientation: self.capturedInterfaceOrientation(app: app)
300-
)
293+
self.safeSnapshotViewport(app: app, readingOrientation: true)
301294
}
302-
if anchor.viewport.isInfinite || anchor.viewport.isNull || anchor.viewport.isEmpty {
303-
return (fallback, RunnerInterfaceOrientation.unknown)
295+
if case .missing = reported {
296+
return fallback
304297
}
305-
return (anchor.viewport, anchor.interfaceOrientation)
298+
return reported
306299
} catch {
307300
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_VIEWPORT_FALLBACK=%@", String(describing: error))
308-
return (fallback, RunnerInterfaceOrientation.unknown)
301+
return fallback
309302
}
310303
}
311304

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import AgentDeviceSnapshotPresentation
12
import XCTest
23

34
extension RunnerTests {
@@ -120,16 +121,9 @@ extension RunnerTests {
120121
return navigationBackKeywords.firstIndex { text.contains($0) }
121122
}
122123

123-
// isFinite/>0 alone don't reject CGRect.infinite — its origin (~-9e307) is finite.
124-
static func isUsableNavigationFrame(_ frame: CGRect) -> Bool {
125-
guard frame.width.isFinite, frame.height.isFinite, frame.width > 0, frame.height > 0 else {
126-
return false
127-
}
128-
return !frame.isInfinite
129-
}
130-
131124
static func isTopNavigationControlFrame(_ candidate: CGRect, in window: CGRect) -> Bool {
132-
guard isUsableNavigationFrame(candidate), isUsableNavigationFrame(window) else {
125+
guard SnapshotGeometry.isPositiveFinite(candidate), SnapshotGeometry.isPositiveFinite(window)
126+
else {
133127
return false
134128
}
135129
// Accept the compact navigation/search header band without matching deep content controls.
@@ -138,7 +132,7 @@ extension RunnerTests {
138132
}
139133

140134
static func topLeadingNavigationFallbackPoint(in frame: CGRect) -> CGPoint? {
141-
guard isUsableNavigationFrame(frame) else {
135+
guard SnapshotGeometry.isPositiveFinite(frame) else {
142136
return nil
143137
}
144138
// Aim at the standard leading navigation slot, bounded for compact and tablet widths.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift‎

Lines changed: 10 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,8 @@ extension RunnerTests {
2121
struct SnapshotTraversalContext {
2222
let queryRoot: XCUIElement
2323
let rootSnapshot: XCUIElementSnapshot
24-
let viewport: CGRect
25-
/** Which way the app's interface is turned from the device's native space (#2612). */
26-
let interfaceOrientation: Int
24+
/** Carries which way the app's interface is turned from the device's native space (#2612). */
25+
let viewport: SnapshotViewport
2726
/**
2827
* The keyboard band this capture measured, published beside the tree so the daemon's tap guard
2928
* measures against the producer's own reading rather than a band it derives from these rects
@@ -278,8 +277,7 @@ extension RunnerTests {
278277
nodes: nodes,
279278
truncated: false,
280279
effectiveDepth: nil,
281-
viewport: context.viewport,
282-
interfaceOrientation: context.interfaceOrientation
280+
viewport: context.viewport
283281
)
284282
}
285283

@@ -435,8 +433,7 @@ extension RunnerTests {
435433
nodes: nodes,
436434
truncated: false,
437435
effectiveDepth: nil,
438-
viewport: context.viewport,
439-
interfaceOrientation: context.interfaceOrientation
436+
viewport: context.viewport
440437
)
441438
}
442439

@@ -455,14 +452,13 @@ extension RunnerTests {
455452
nodes: nodes,
456453
truncated: false,
457454
effectiveDepth: nil,
458-
viewport: .infinite,
459-
interfaceOrientation: RunnerInterfaceOrientation.unknown
455+
viewport: .missing(reason: .notProvided)
460456
),
461457
.completed
462458
)
463459
}
464460

465-
let viewport = safeSnapshotViewport(app: app)
461+
let viewport = safeSnapshotViewport(app: app, readingOrientation: false)
466462
var seen = Set<String>()
467463
var candidates: [RawAXNode] = []
468464
let flatElements = flatInteractiveElements(app: app, deadline: deadline)
@@ -492,11 +488,9 @@ extension RunnerTests {
492488
}
493489

494490
// The synthetic root doubles as the daemon's viewport (find.ts prefers on-screen matches
495-
// inside nodes[0].rect): use the real screen viewport when capture produced a finite one,
496-
// so off-screen candidates can never inflate the root and masquerade as on-screen.
497-
let rootRect = viewport.isInfinite || viewport.isNull || viewport.isEmpty
498-
? interactiveRootFrame(for: candidates)
499-
: viewport
491+
// inside nodes[0].rect): use the real screen viewport when the capture resolved one, so
492+
// off-screen candidates can never inflate the root and masquerade as on-screen.
493+
let rootRect = viewport.rect ?? interactiveRootFrame(for: candidates)
500494
nodes[0] = interactiveRootNode(rect: rootRect)
501495
for candidate in candidates {
502496
nodes.append(
@@ -524,8 +518,7 @@ extension RunnerTests {
524518
nodes: nodes,
525519
truncated: outcome == .deadlineExhausted,
526520
effectiveDepth: nil,
527-
viewport: viewport,
528-
interfaceOrientation: RunnerInterfaceOrientation.unknown
521+
viewport: viewport
529522
),
530523
outcome
531524
)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotAcquisition.swift‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -41,18 +41,13 @@ extension RunnerTests {
4141
// The viewport and the interface orientation are one hop: geometry that arrives in the device's
4242
// native space can only be placed relative to the app's own frame and rotation, and asking for
4343
// the pair twice would read them at two different moments of a rotation.
44-
let geometry = try runMainThreadWork(
44+
let viewport = try runMainThreadWork(
4545
"snapshot_viewport",
4646
timeout: min(1.0, max(0.1, captureDeadline.timeIntervalSinceNow)),
4747
timeoutError: snapshotMainThreadTimeoutError("preparing tree snapshot")
4848
) {
49-
(
50-
viewport: self.safeSnapshotViewport(app: app),
51-
interfaceOrientation: self.capturedInterfaceOrientation(app: app)
52-
)
49+
self.safeSnapshotViewport(app: app, readingOrientation: true)
5350
}
54-
let viewport = geometry.viewport
55-
let interfaceOrientation = geometry.interfaceOrientation
5651
let treeSliceBudget = treeCaptureSliceBudgetOverride ?? treeCaptureSliceBudget
5752
let slice = min(treeSliceBudget, max(0.5, captureDeadline.timeIntervalSinceNow))
5853
guard let rootSnapshot = try captureSnapshotRootBounded(app, sliceSeconds: slice) else {
@@ -70,7 +65,6 @@ extension RunnerTests {
7065
queryRoot: app,
7166
rootSnapshot: rootSnapshot,
7267
viewport: viewport,
73-
interfaceOrientation: interfaceOrientation,
7468
keyboardBand: keyboardBand
7569
)
7670
}
@@ -140,8 +134,18 @@ extension RunnerTests {
140134
return nil
141135
}
142136

143-
func safeSnapshotViewport(app: XCUIApplication) -> CGRect {
144-
safely("SNAPSHOT_VIEWPORT", CGRect.infinite) { snapshotViewport(app: app) }
137+
/// The viewport as a declared fact; a read that raises is `.missing(reason: .notProvided)` (#2891).
138+
/// `readingOrientation` reads the interface orientation in the same hop, for tiers whose frames can
139+
/// arrive in the device's native space; without it the viewport cannot anchor a rotation.
140+
func safeSnapshotViewport(app: XCUIApplication, readingOrientation: Bool) -> SnapshotViewport {
141+
safely("SNAPSHOT_VIEWPORT", .missing(reason: .notProvided)) {
142+
.reported(
143+
box: snapshotAppFrame(app: app),
144+
interfaceOrientation: readingOrientation
145+
? capturedInterfaceOrientation(app: app)
146+
: RunnerInterfaceOrientation.unknown
147+
)
148+
}
145149
}
146150

147151
private func describeSnapshotError(_ error: Error) -> String {
@@ -231,16 +235,12 @@ extension RunnerTests {
231235
return text.isEmpty ? nil : text
232236
}
233237

234-
private func snapshotViewport(app: XCUIApplication) -> CGRect {
238+
private func snapshotAppFrame(app: XCUIApplication) -> CGRect {
235239
#if os(iOS)
236-
let appFrame = onScreenWindowFrame(app: app)
240+
return onScreenWindowFrame(app: app)
237241
#else
238-
let appFrame = app.frame
242+
return app.frame
239243
#endif
240-
if !appFrame.isNull && !appFrame.isEmpty {
241-
return appFrame
242-
}
243-
return .infinite
244244
}
245245

246246
static func snapshotTraversalIdentity(

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -513,8 +513,7 @@ extension RunnerTests {
513513
let normalizedAcquisition = acquisition.replacingNodes(
514514
SnapshotGeometrySpace.normalized(
515515
nodes: acquisition.nodes,
516-
viewport: acquisition.viewport,
517-
interfaceOrientation: acquisition.interfaceOrientation
516+
viewport: acquisition.viewport
518517
)
519518
)
520519

@@ -641,7 +640,7 @@ extension RunnerTests {
641640
guard Self.structuralOnlyNodeTypes.contains(node.type) else { return false }
642641
guard !isRootContainer else { return true }
643642

644-
let isFullScreenContainer = !node.hittable && rootRects.contains { rootRect in
643+
let isFullScreenContainer = node.hittable != true && rootRects.contains { rootRect in
645644
rootRect.x == node.rect.x && rootRect.y == node.rect.y
646645
&& rootRect.width == node.rect.width && rootRect.height == node.rect.height
647646
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AXSnapshotFallbackTests.swift‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -464,15 +464,14 @@ extension RunnerTests {
464464
interactiveOnly: true, customActions: false)
465465
let acquired = SnapshotGeometrySpace.normalized(
466466
nodes: privateAXAcquisition(rawRoot: tree, hint: hint),
467-
viewport: viewport,
468-
interfaceOrientation: RunnerInterfaceOrientation.portrait
467+
viewport: .reported(box: viewport, interfaceOrientation: RunnerInterfaceOrientation.portrait)
469468
)
470469
// Acquisition serializes the drawer too; the shared fold is what hides it (#1797).
471470
XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings"))
472471

473472
let capture = try SnapshotPresentation.presentRegular(
474473
SnapshotAcquisition(
475-
hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: viewport),
474+
hint: hint, nodes: acquired, truncated: false, effectiveDepth: nil, viewport: .reported(box: viewport)),
476475
options: PresentationOptions(interactiveOnly: true, depth: nil, scope: nil, raw: false),
477476
policy: .cursorProjected
478477
)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+PrivateAXPresentationTests.swift‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,11 @@ extension RunnerTests {
4242
interfaceOrientation: RunnerInterfaceOrientation.portrait)
4343
return try SnapshotPresentation.presentRegular(
4444
SnapshotAcquisition(
45-
hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil, viewport: viewport,
46-
interfaceOrientation: RunnerInterfaceOrientation.portrait),
45+
hint: hint, nodes: nodes, truncated: false, effectiveDepth: nil,
46+
viewport: .reported(
47+
box: viewport,
48+
interfaceOrientation: RunnerInterfaceOrientation.portrait
49+
)),
4750
options: PresentationOptions(
4851
interactiveOnly: interactiveOnly, depth: nil, scope: nil, raw: false),
4952
policy: .cursorProjected
@@ -59,8 +62,7 @@ extension RunnerTests {
5962
) -> [RawAXNode] {
6063
SnapshotGeometrySpace.normalized(
6164
nodes: privateAXAcquisition(rawRoot: rawRoot, hint: hint),
62-
viewport: viewport,
63-
interfaceOrientation: interfaceOrientation
65+
viewport: .reported(box: viewport, interfaceOrientation: interfaceOrientation)
6466
)
6567
}
6668

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SnapshotCapturePlanOccupancyTests.swift‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,8 +94,7 @@ extension RunnerTests {
9494
XCTAssertFalse(app.frame.isEmpty)
9595
// The traversal context reads the viewport under a 1 s cap; a cold first read can overrun it
9696
// and abandon the wrong block, so pay it here, uncapped.
97-
_ = safeSnapshotViewport(app: app)
98-
_ = capturedInterfaceOrientation(app: app)
97+
_ = safeSnapshotViewport(app: app, readingOrientation: true)
9998
currentApp = app
10099
currentBundleId = "com.callstack.agentdevice.runner.tree-capture-test"
101100
snapshotXCTestPenaltyWarmupExemption.isPending = true

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SnapshotCapturePlanTests.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -282,7 +282,7 @@ extension RunnerTests {
282282
nodes: [],
283283
truncated: false,
284284
effectiveDepth: nil,
285-
viewport: .infinite
285+
viewport: .reported(box: CGRect(x: 0, y: 0, width: 402, height: 874))
286286
),
287287
options: options
288288
)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SnapshotHittabilityTests.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ extension RunnerTests {
9797
nodes: nodes,
9898
truncated: false,
9999
effectiveDepth: nil,
100-
viewport: CGRect(x: 0, y: 0, width: 100, height: 100)
100+
viewport: .reported(box: CGRect(x: 0, y: 0, width: 100, height: 100))
101101
),
102102
options: options
103103
)

0 commit comments

Comments
 (0)