Skip to content

Commit 6cfe6a5

Browse files
committed
refactor(ios): declare bridge failure scope at the throw site, disclose unplaced surface hosts, move the coordinate-space rule into the presentation package
Three follow-ups to #2653 (fixes #2612), each inside that PR's own new code. Failure scope. Whether a Simulator AX bridge failure retires the app generation from the bridge was decided at the route by matching the failure's kind and code. SnapshotSourceFailure now carries scope: 'capture' | 'generation', set where the failure is thrown (snapshotSourceCaptureError for bridge-preparation-pending and window-coordinate-space-unresolved), and the route reads the field. Disclosure. A runner capture that could not turn a rotated surface host back (orientation unreadable, or the private-AX tier anchored on the bridge's root box) published two coordinate spaces silently. SnapshotGeometrySpace.unplacedSurfaceHostCount runs after every tier's acquisition; a non-zero count rides the quality verdict as unresolvedCoordinateSpaceWindows and the daemon renders a warning. Package. The rotation table and the coordinate-space rule are pure CoreGraphics and now live in AgentDeviceSnapshotPresentation (SnapshotCoordinateSpace.swift) with their tests in the package suite, so swift test replays the golden table without a simulator. isSurfaceHost takes booleans instead of an XCUIElement.ElementType.
1 parent 91652a8 commit 6cfe6a5

25 files changed

Lines changed: 598 additions & 363 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,15 @@
3434
in the app's own orientation space.` — and the route sends that one capture to the runner instead of
3535
retiring the app generation, so the next capture of a healthy app still uses the bridge. One table
3636
proves both languages apply one rule: `contracts/fixtures/window-coordinate-space.json` (#2612).
37+
- Changed (ios): a runner capture that could not turn a rotated system surface back into the app's
38+
orientation space — the interface orientation was unreadable, or the tier had only the bridge's
39+
root box to anchor on — now says so instead of publishing two coordinate spaces silently: the
40+
quality verdict carries `unresolvedCoordinateSpaceWindows` and the snapshot warns that rects under
41+
those windows are not in the same space as the rest of the tree. Whether a Simulator AX bridge
42+
failure retires the app generation from the bridge is now declared where the failure is thrown
43+
(`scope: capture | generation`) rather than matched on its code at the route. The rotation table
44+
and the coordinate-space rule moved from the XCTest bundle into the `AgentDeviceSnapshotPresentation`
45+
package, where `swift test` replays the golden table without a simulator.
3746
- Added (limrun): `longpress` on Limrun iOS direct sessions. The interactor refused it as
3847
unsupported although the SDK exposes the HID primitives; it now holds one touch as a
3948
`performActions` batch of `touchDown`, `wait`, `touchUp`, defaulting to the 800 ms the Android

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerScrollViewportPolicy.swift‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import XCTest
2+
import AgentDeviceSnapshotPresentation
23

34
// The scroll viewport rule the runner shares with the TS runtime (#2500).
45
//

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+PrivateAXPresentation.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ extension RunnerTests {
4444
let fields = privateAXFields(raw)
4545
let nodeSpace = SnapshotGeometrySpace.space(
4646
reportedBySurfaceHost: SnapshotGeometrySpace.isSurfaceHost(
47-
elementType: fields.elementType,
47+
isWindow: isWindowElement(fields.elementType),
4848
parentIsWindow: parentIsWindow
4949
),
5050
reportedFrame: fields.rect,

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Snapshot.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1069,7 +1069,7 @@ extension RunnerTests {
10691069
) -> SnapshotGeometrySpace {
10701070
SnapshotGeometrySpace.space(
10711071
reportedBySurfaceHost: SnapshotGeometrySpace.isSurfaceHost(
1072-
elementType: snapshot.elementType,
1072+
isWindow: isWindowElement(snapshot.elementType),
10731073
parentIsWindow: parentIsWindow
10741074
),
10751075
reportedFrame: snapshot.frame,

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotCapturePlan.swift‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ struct SnapshotQuality: Codable {
2727
let collapsedLeafIndexes: [Int]?
2828
/// Coverage of the bounded custom-action pass, when the capture asked for one.
2929
let customActions: SnapshotCustomActionCoverage?
30+
/// Surface hosts the published tree still carries in the device's native space: the capture could
31+
/// not name the app's interface orientation to turn them back, so rects under them are not in the
32+
/// space the rest of the tree publishes (#2612). Absent when every host was placed.
33+
var unresolvedCoordinateSpaceWindows: Int? = nil
3034
/// Response-level timing for the accepted backend attempt, never repeated per node.
3135
var timing: SnapshotCaptureTiming? = nil
3236
}
@@ -61,6 +65,8 @@ struct SnapshotBackendCapture {
6165
let effectiveDepth: Int?
6266
var customActions: SnapshotCustomActionCoverage? = nil
6367
var qualityPayload: DataPayload? = nil
68+
/// Turned surface hosts the acquisition left as reported; nil when the tree holds none.
69+
var unresolvedCoordinateSpaceWindows: Int? = nil
6470
var timing: SnapshotCaptureTiming? = nil
6571
}
6672

@@ -503,6 +509,13 @@ extension RunnerTests {
503509

504510
var capture = presented
505511
capture.timing = timer.timing
512+
// A host the walk could not place is disclosed, never silently published as an address: the
513+
// orientation read failed, or the tier had only the bridge's root box to anchor on (#2612).
514+
let unplaced = SnapshotGeometrySpace.unplacedSurfaceHostCount(
515+
in: acquisition.nodes,
516+
viewport: acquisition.viewport
517+
)
518+
capture.unresolvedCoordinateSpaceWindows = unplaced > 0 ? unplaced : nil
506519
return SnapshotBackendAttempt(
507520
outcome: .captured(capture),
508521
timing: timer.timing
@@ -625,6 +638,7 @@ extension RunnerTests {
625638
effectiveDepth: capture.effectiveDepth,
626639
collapsedLeafIndexes: Self.collapsedLeafIndexes(payload.nodes ?? []),
627640
customActions: capture.customActions,
641+
unresolvedCoordinateSpaceWindows: capture.unresolvedCoordinateSpaceWindows,
628642
timing: capture.timing
629643
)
630644
return DataPayload(
@@ -876,6 +890,25 @@ extension RunnerTests {
876890
XCTAssertEqual(payload.nodes?.count, 1)
877891
}
878892

893+
/// A tree that still carries a turned host says so on the verdict, and one that holds none says
894+
/// nothing: the daemon's warning keys off presence, so a zero must not ride the wire as a fact.
895+
func testStampedPayloadDisclosesUnplacedSurfaceHosts() {
896+
let payload = DataPayload(
897+
nodes: [planTestNode(index: 0, type: "Application", label: "App")],
898+
truncated: false
899+
)
900+
let unplaced = stampedSnapshotPayload(
901+
SnapshotBackendCapture(
902+
payload: payload, effectiveDepth: nil, unresolvedCoordinateSpaceWindows: 1),
903+
backend: .recursiveTree, state: "healthy", reason: nil)
904+
XCTAssertEqual(unplaced.snapshotQuality?.unresolvedCoordinateSpaceWindows, 1)
905+
906+
let placed = stampedSnapshotPayload(
907+
SnapshotBackendCapture(payload: payload, effectiveDepth: nil),
908+
backend: .recursiveTree, state: "healthy", reason: nil)
909+
XCTAssertNil(placed.snapshotQuality?.unresolvedCoordinateSpaceWindows)
910+
}
911+
879912
func testStampedPayloadTruncationTracksCompletenessNotRecoveryProvenance() {
880913
let complete = SnapshotBackendCapture(
881914
payload: DataPayload(
Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
import Foundation
2+
import CoreGraphics
3+
4+
// Where a piece of geometry lives, and how it gets where the caller means it.
5+
//
6+
// DISPATCH: synthesized events skip XCTest's orientation handling, so a point planned in an app's
7+
// interface space has to be rotated into the device's native (portrait-up) space before it is
8+
// performed. Without that, a landscape tap lands somewhere else.
9+
//
10+
// CAPTURE: iOS hosts some system surfaces in that native space even while the app is rotated, so
11+
// their whole subtree arrives quarter-turned. Measured on iPhone 17 Pro (iOS 26.2) with the system
12+
// keyboard up in landscape: the app viewport is 874 x 402, `UIRemoteKeyboardWindow` reports its own
13+
// box as 402 x 874, a key arrives 45 x 72 at x 154, and the key plane as 202 x 724 — a strip down
14+
// the left edge — while the screenshot draws a 724 x 204 band docked at y 198, which is exactly
15+
// what the live `app.keyboards` query measures. Rules that read the reported numbers refused app
16+
// content the keyboard was nowhere near and let a tap through into a key (#2612, #2589).
17+
//
18+
// Both directions live in ONE table on purpose: capture rotates back with the exact inverse of what
19+
// dispatch rotates forward, so a performed tap and a reported rect can never disagree about which
20+
// pixel is which. The round-trip test in the package suite is that proof, not a comment asking for
21+
// care. This file is pure geometry, which is why it lives here rather than in the XCTest bundle:
22+
// nothing in it needs a simulator to run.
23+
24+
/// The UIInterfaceOrientation raw values the rotation table switches on. XCTest exposes the app's
25+
/// interface orientation as an integer, and 0 (unknown) deliberately rotates nothing.
26+
public enum RunnerInterfaceOrientation {
27+
public static let unknown = 0
28+
public static let portrait = 1
29+
public static let portraitUpsideDown = 2
30+
public static let landscapeRight = 3
31+
public static let landscapeLeft = 4
32+
}
33+
34+
/// The quarter-turn between an app's interface space and the device's native (portrait-up) space.
35+
public enum CoordinateSpaceRotation {
36+
/// An interface-space point, in the native space synthesized events are performed in.
37+
public static func native(
38+
point: CGPoint,
39+
in frame: CGRect,
40+
interfaceOrientation: Int
41+
) -> CGPoint {
42+
let localX = Double(point.x) - Double(frame.minX)
43+
let localY = Double(point.y) - Double(frame.minY)
44+
let width = Double(frame.width)
45+
let height = Double(frame.height)
46+
switch interfaceOrientation {
47+
case RunnerInterfaceOrientation.landscapeRight:
48+
return CGPoint(x: height - localY, y: localX)
49+
case RunnerInterfaceOrientation.landscapeLeft:
50+
return CGPoint(x: localY, y: width - localX)
51+
case RunnerInterfaceOrientation.portraitUpsideDown:
52+
return CGPoint(x: width - localX, y: height - localY)
53+
default: // portrait, or an orientation the platform did not name
54+
return CGPoint(x: localX, y: localY)
55+
}
56+
}
57+
58+
/// An interface-space translation vector, in the same native space as `native(point:)`.
59+
public static func native(vector: CGVector, interfaceOrientation: Int) -> CGVector {
60+
switch interfaceOrientation {
61+
case RunnerInterfaceOrientation.landscapeRight:
62+
return CGVector(dx: -vector.dy, dy: vector.dx)
63+
case RunnerInterfaceOrientation.landscapeLeft:
64+
return CGVector(dx: vector.dy, dy: -vector.dx)
65+
case RunnerInterfaceOrientation.portraitUpsideDown:
66+
return CGVector(dx: -vector.dx, dy: -vector.dy)
67+
default: // portrait, or an orientation the platform did not name
68+
return vector
69+
}
70+
}
71+
72+
/// The inverse of `native(point:)`: a native-space point, in the app's interface space.
73+
public static func oriented(
74+
point: CGPoint,
75+
in frame: CGRect,
76+
interfaceOrientation: Int
77+
) -> CGPoint {
78+
let width = Double(frame.width)
79+
let height = Double(frame.height)
80+
let localX: Double
81+
let localY: Double
82+
switch interfaceOrientation {
83+
case RunnerInterfaceOrientation.landscapeRight:
84+
localX = Double(point.y)
85+
localY = height - Double(point.x)
86+
case RunnerInterfaceOrientation.landscapeLeft:
87+
localX = width - Double(point.y)
88+
localY = Double(point.x)
89+
case RunnerInterfaceOrientation.portraitUpsideDown:
90+
localX = width - Double(point.x)
91+
localY = height - Double(point.y)
92+
default: // portrait, or an orientation the platform did not name
93+
localX = Double(point.x)
94+
localY = Double(point.y)
95+
}
96+
return CGPoint(x: localX + Double(frame.minX), y: localY + Double(frame.minY))
97+
}
98+
99+
/// The inverse of `native(point:)` for a rect. A quarter turn swaps the axes, so opposite corners
100+
/// of the reported rect become opposite corners of the result; ordering them keeps the origin
101+
/// top-left, which is what every consumer of a published rect reads.
102+
public static func oriented(
103+
rect: CGRect,
104+
in frame: CGRect,
105+
interfaceOrientation: Int
106+
) -> CGRect {
107+
let leading = oriented(
108+
point: CGPoint(x: rect.minX, y: rect.minY),
109+
in: frame,
110+
interfaceOrientation: interfaceOrientation
111+
)
112+
let trailing = oriented(
113+
point: CGPoint(x: rect.maxX, y: rect.maxY),
114+
in: frame,
115+
interfaceOrientation: interfaceOrientation
116+
)
117+
return CGRect(
118+
x: min(leading.x, trailing.x),
119+
y: min(leading.y, trailing.y),
120+
width: abs(trailing.x - leading.x),
121+
height: abs(trailing.y - leading.y)
122+
)
123+
}
124+
}
125+
126+
/// The space one captured subtree reports its rects in.
127+
///
128+
/// RULE: the capture publishes every rect in the app's orientation space. A surface host declares the
129+
/// space of its own subtree — its box is either the app's box, or the app's box turned through a
130+
/// quarter, which is how a surface hosted in the device's native space announces itself — and anything
131+
/// below it inherits that space. Nothing is declared, and every rect stays exactly as the platform
132+
/// reported it, when the app frame is unusable, when the interface orientation names no quarter turn
133+
/// (portrait, or an orientation the platform did not name), or when the app frame is square. Geometry
134+
/// the capture cannot place is not a claim about where it is, and consumers already fail open on it the
135+
/// way they do on any missing platform fact — but the capture says so: `unplacedSurfaceHostCount`
136+
/// counts the hosts a published tree still carries turned, so the daemon can warn instead of
137+
/// publishing two spaces silently.
138+
///
139+
/// A half turn is a different claim: an upside-down app's own box is already the native box, so no
140+
/// reported window distinguishes the two spaces and nothing is turned back. That is the same limit the
141+
/// box-only rule accepts for a square app frame, and it is why the consumer rules in
142+
/// `tap-keyboard-occlusion.ts` stay rather than becoming an assertion here.
143+
public enum SnapshotGeometrySpace: Equatable {
144+
/** The app's own orientation space: the one every consumer of a published tree reads. */
145+
case appOrientation
146+
/** The device's native (portrait-up) space, with the way back to the app's space attached. */
147+
case deviceNative(appFrame: CGRect, interfaceOrientation: Int)
148+
149+
/// How far a window's side lengths may miss the app's swapped side lengths and still be that
150+
/// quarter turn. Measured captures match to the point; this absorbs float representation only.
151+
public static let quarterTurnTolerance: Double = 1
152+
153+
/// The reported rect, in the space the capture publishes.
154+
public func orientedFrame(of reportedFrame: CGRect) -> CGRect {
155+
switch self {
156+
case .appOrientation:
157+
return reportedFrame
158+
case .deviceNative(let appFrame, let interfaceOrientation):
159+
return CoordinateSpaceRotation.oriented(
160+
rect: reportedFrame,
161+
in: appFrame,
162+
interfaceOrientation: interfaceOrientation
163+
)
164+
}
165+
}
166+
167+
/// The space one node's subtree reports in.
168+
///
169+
/// Only a surface host may declare a space: the window itself, or the surface the window hands its
170+
/// content to. XCTest reports `UIRemoteKeyboardWindow` with the app's own box and its child with
171+
/// the turned one, so the window alone is not where the turn shows up, while a deep node claiming a
172+
/// turned box is content whose reported bounds happen to be large and is left in the inherited
173+
/// space. A host that reports anything other than a turned box declares the app's space for its
174+
/// subtree, which is how a tree with no rotated surface stays exactly as reported.
175+
public static func space(
176+
reportedBySurfaceHost isSurfaceHost: Bool,
177+
reportedFrame: CGRect,
178+
inheritedFrom inherited: SnapshotGeometrySpace,
179+
appFrame: CGRect,
180+
interfaceOrientation: Int
181+
) -> SnapshotGeometrySpace {
182+
guard isSurfaceHost else { return inherited }
183+
guard namesQuarterTurn(interfaceOrientation) else { return .appOrientation }
184+
guard isQuarterTurned(reportedFrame, relativeTo: appFrame) else { return .appOrientation }
185+
return .deviceNative(appFrame: appFrame, interfaceOrientation: interfaceOrientation)
186+
}
187+
188+
/// Whether this orientation is a turn the reported box can be measured against. Portrait and an
189+
/// unnamed orientation leave the device's native space upright, so nothing under them is turned.
190+
public static func namesQuarterTurn(_ interfaceOrientation: Int) -> Bool {
191+
interfaceOrientation == RunnerInterfaceOrientation.landscapeLeft
192+
|| interfaceOrientation == RunnerInterfaceOrientation.landscapeRight
193+
}
194+
195+
/// Whether this node, or the window above it, is where a hosted surface's box appears.
196+
public static func isSurfaceHost(isWindow: Bool, parentIsWindow: Bool) -> Bool {
197+
isWindow || parentIsWindow
198+
}
199+
200+
/// The element type names under which a producer publishes the app's windows: the app's own
201+
/// window, `UITextEffectsWindow`, `UIRemoteKeyboardWindow`. XCTest models each as its own element
202+
/// typed `.application` or `.window`, and every producer names them this way in a published tree.
203+
public static func isWindowType(_ type: String) -> Bool {
204+
type == "Application" || type == "Window"
205+
}
206+
207+
/// Surface hosts a published tree still carries in the device's native space.
208+
///
209+
/// A capture that could name the app's interface orientation turned every such host back, so its
210+
/// published tree holds none. One that could not — the orientation read failed, or the tier had
211+
/// only the bridge's own root box to anchor on — publishes the host as the platform reported it,
212+
/// and this is where that shows: a host whose box is the viewport quarter-turned. The count is a
213+
/// disclosure, not a repair; it rides the quality verdict so the daemon can say which captures hold
214+
/// two spaces instead of letting rects under those hosts read as addresses (#2612).
215+
public static func unplacedSurfaceHostCount(in nodes: [RawAXNode], viewport: CGRect) -> Int {
216+
guard isPlottable(viewport) else { return 0 }
217+
var count = 0
218+
for node in nodes {
219+
let parentIsWindow = node.parentIndex.map { index in
220+
nodes.indices.contains(index) && isWindowType(nodes[index].type)
221+
} ?? false
222+
guard isSurfaceHost(isWindow: isWindowType(node.type), parentIsWindow: parentIsWindow)
223+
else { continue }
224+
let frame = CGRect(
225+
x: node.rect.x, y: node.rect.y, width: node.rect.width, height: node.rect.height
226+
)
227+
if isQuarterTurned(frame, relativeTo: viewport) { count += 1 }
228+
}
229+
return count
230+
}
231+
232+
/// A window that reports the app's two side lengths on the other axes is hosted in the device's
233+
/// native space. A square app frame cannot be told from its own quarter turn, so it is left alone
234+
/// rather than guessed at.
235+
private static func isQuarterTurned(_ frame: CGRect, relativeTo appFrame: CGRect) -> Bool {
236+
guard isPlottable(frame), isPlottable(appFrame),
237+
abs(appFrame.width - appFrame.height) > quarterTurnTolerance
238+
else {
239+
return false
240+
}
241+
return abs(frame.width - appFrame.height) <= quarterTurnTolerance
242+
&& abs(frame.height - appFrame.width) <= quarterTurnTolerance
243+
}
244+
245+
private static func isPlottable(_ frame: CGRect) -> Bool {
246+
frame.origin.x.isFinite && frame.origin.y.isFinite
247+
&& frame.width.isFinite && frame.height.isFinite
248+
&& frame.width > 0 && frame.height > 0
249+
}
250+
}

0 commit comments

Comments
 (0)