Skip to content

Commit bc80c28

Browse files
committed
fix(ios): publish captured keyboard geometry in the app's orientation space
iOS hosts the system keyboard in the device's native portrait space, so in landscape its whole subtree arrives quarter-turned: a key measured 45x72 at x 154 in an 874x402 app, drawing a strip down the left edge where the screenshot shows a 724x204 band docked at y 198. Rules that read those numbers refused app content the keyboard was nowhere near and let a tap land on a key (#2612). The Apple runner now reads the app's interface orientation at capture time and publishes every rect under a turned surface host in the app's space, through the exact inverse of the rotation its synthesized touches rotate forward by, so a reported rect and a performed tap cannot disagree about which pixel is which. One table serves both directions, and the new coordinate-space module owns it. An orientation that names no quarter turn declares nothing, and the collapsed-tab expansion, which reads live element frames, stays out of a declared native space rather than turning them twice. The Simulator AX bridge reader carries no interface orientation, so it counts surface hosts reporting the app box quarter-turned and refuses that capture with window-coordinate-space-unresolved; the route serves the runner for that capture alone rather than retiring the app generation, so the next capture of a healthy app still uses the bridge. Both languages replay one table, contracts/fixtures/window-coordinate-space.json, which pins the turn and the box that declares it.
1 parent 849ad7d commit bc80c28

21 files changed

Lines changed: 1642 additions & 188 deletions

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,21 @@
1919
(a disabled title field presented as an enabled Button for the whole row) no longer carries the
2020
field's `hittable: false`; on the XCTest runner path that Button now counts as interactive and
2121
becomes a Maestro atomic-dispatch candidate where it was excluded before.
22+
- Fixed (ios): a landscape iPhone snapshot reports the system keyboard's rects in the app's own
23+
orientation space. iOS hosts `UIRemoteKeyboardWindow` in the device's native portrait space, so its
24+
whole subtree arrived quarter-turned — a key measured 45 pt wide and 72 pt tall at `x 154` in an
25+
874 x 402 app, drawing a strip down the left edge where the screenshot shows a 724 x 204 band
26+
docked at `y 198`. Rules that read those numbers refused app content the keyboard was nowhere near
27+
and let a tap land on a key. The Apple runner now reads the app's interface orientation at capture
28+
time and publishes every rect under a turned surface host in the app's space, using the exact
29+
inverse of the rotation its synthesized touches already rotate through, so a reported rect and a
30+
performed tap cannot disagree about which pixel is which. The Simulator AX bridge reader has no
31+
interface orientation in its attribute set, so it refuses a capture holding a surface host reporting
32+
the app box quarter-turned — `Simulator AX snapshot unavailable
33+
(window-coordinate-space-unresolved); used XCTest for this capture, which reports captured geometry
34+
in the app's own orientation space.` — and the route sends that one capture to the runner instead of
35+
retiring the app generation, so the next capture of a healthy app still uses the bridge. One table
36+
proves both languages apply one rule: `contracts/fixtures/window-coordinate-space.json` (#2612).
2237
- Added (limrun): `longpress` on Limrun iOS direct sessions. The interactor refused it as
2338
unsupported although the SDK exposes the HID primitives; it now holds one touch as a
2439
`performActions` batch of `touchDown`, `wait`, `touchUp`, defaulting to the 800 ms the Android

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerCoordinateSpace.swift‎

Lines changed: 608 additions & 0 deletions
Large diffs are not rendered by default.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerScrollViewportPolicy.swift‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ enum RunnerScrollViewport {
3333
/**
3434
* The band to plan the swipe inside, the frame to rotate its coordinates against, and the keyboard
3535
* top when the band was clipped for one. The two frames are separate on purpose: a clip shortens
36-
* only the band, while `nativeSynthesizedPoint` derives a `landscapeRight` native x from the
36+
* only the band, while `CoordinateSpaceRotation.native(point:)` derives a `landscapeRight` native x from the
3737
* frame's HEIGHT, so rotating inside the band moves the dispatched path sideways off the planned
3838
* one.
3939
*/
@@ -329,7 +329,7 @@ extension RunnerTests {
329329
XCTAssertEqual(constants.accessoryAllowance, ScrollViewportPolicy.accessoryAllowance)
330330
}
331331

332-
/// A clipped landscape band shortens the frame, and `nativeSynthesizedPoint` derives a
332+
/// A clipped landscape band shortens the frame, and `CoordinateSpaceRotation.native(point:)` derives a
333333
/// `landscapeRight` native x from the frame's HEIGHT. Rotating inside the band therefore moves the
334334
/// dispatched path sideways by exactly what the keyboard took, off the lane the plan was built for,
335335
/// so the plan band and the coordinate basis stay separate values through dispatch (#2500).
@@ -374,20 +374,18 @@ extension RunnerTests {
374374
XCTAssertEqual(reported.data?.keyboardAvoided, true)
375375

376376
let orientedStartY = gesture.planFrame.minY + gesture.plan.y1
377-
let dispatched = nativeSynthesizedPoint(
378-
orientedX: gesture.planFrame.minX + gesture.plan.x1,
379-
orientedY: orientedStartY,
377+
let dispatchedFromViewport = CoordinateSpaceRotation.native(
378+
point: CGPoint(x: gesture.planFrame.minX + gesture.plan.x1, y: orientedStartY),
380379
in: gesture.coordinateFrame,
381380
interfaceOrientation: RunnerInterfaceOrientation.landscapeRight
382381
)
383-
let clippedBasis = nativeSynthesizedPoint(
384-
orientedX: gesture.planFrame.minX + gesture.plan.x1,
385-
orientedY: orientedStartY,
382+
let dispatchedFromBand = CoordinateSpaceRotation.native(
383+
point: CGPoint(x: gesture.planFrame.minX + gesture.plan.x1, y: orientedStartY),
386384
in: gesture.planFrame,
387385
interfaceOrientation: RunnerInterfaceOrientation.landscapeRight
388386
)
389387
XCTAssertEqual(
390-
dispatched.x - clippedBasis.x,
388+
dispatchedFromViewport.x - dispatchedFromBand.x,
391389
viewport.height - band.height,
392390
accuracy: 0.001,
393391
"rotating inside the clipped band would shift native x by what the keyboard took"

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -216,11 +216,13 @@ extension RunnerTests {
216216
}
217217

218218
let rootFrame = privateAXRect(root["frame"])
219-
let viewport = privateAXSnapshotViewport(app: app, rootFrame: rootFrame)
219+
let geometry = privateAXSnapshotGeometry(app: app, rootFrame: rootFrame)
220+
let viewport = geometry.viewport
220221
let nodes = privateAXAcquisition(
221222
rawRoot: root,
222223
hint: hint,
223-
viewport: viewport
224+
viewport: viewport,
225+
interfaceOrientation: geometry.interfaceOrientation
224226
)
225227
// Serialization-level emptiness only: an acquired-but-fully-clipped tree is presentation's
226228
// verdict now, surfaced by the plan's sparse classifier on the presented payload (#1797).
@@ -269,26 +271,36 @@ extension RunnerTests {
269271
!hasAbandonedMainThreadWork() && !isSnapshotXCTestChannelPenalized(bundleId: currentBundleId)
270272
}
271273

272-
private func privateAXSnapshotViewport(app: XCUIApplication, rootFrame: CGRect) -> CGRect {
274+
/// The geometry this tier may anchor a rotation on. The bridge's own root frame is one more
275+
/// reported box rather than the app's frame, so a capture anchored on it reports no interface
276+
/// orientation and normalizes nothing: rotated system surfaces then stay as reported, which the
277+
/// consumers already treat as geometry they cannot measure (#2612).
278+
private func privateAXSnapshotGeometry(
279+
app: XCUIApplication,
280+
rootFrame: CGRect
281+
) -> (viewport: CGRect, interfaceOrientation: Int) {
273282
let fallback = rootFrame.isEmpty ? CGRect.infinite : rootFrame
274283
guard shouldReadPrivateAXViewportViaXCTest() else {
275-
return fallback
284+
return (fallback, RunnerInterfaceOrientation.unknown)
276285
}
277286
do {
278-
let viewport = try runMainThreadWork(
287+
let anchor = try runMainThreadWork(
279288
"private_ax_viewport",
280289
timeout: 1,
281290
timeoutError: snapshotMainThreadTimeoutError("reading private AX viewport")
282291
) {
283-
self.safeSnapshotViewport(app: app)
292+
(
293+
viewport: self.safeSnapshotViewport(app: app),
294+
interfaceOrientation: self.capturedInterfaceOrientation(app: app)
295+
)
284296
}
285-
if viewport.isInfinite || viewport.isNull || viewport.isEmpty {
286-
return fallback
297+
if anchor.viewport.isInfinite || anchor.viewport.isNull || anchor.viewport.isEmpty {
298+
return (fallback, RunnerInterfaceOrientation.unknown)
287299
}
288-
return viewport
300+
return (anchor.viewport, anchor.interfaceOrientation)
289301
} catch {
290302
NSLog("AGENT_DEVICE_RUNNER_PRIVATE_AX_VIEWPORT_FALLBACK=%@", String(describing: error))
291-
return fallback
303+
return (fallback, RunnerInterfaceOrientation.unknown)
292304
}
293305
}
294306

@@ -686,7 +698,8 @@ extension RunnerTests {
686698
hint: CaptureHint(
687699
projection: .regular, depth: nil, regularPresentedDepth: nil,
688700
interactiveOnly: false, customActions: false),
689-
viewport: CGRect(x: 0, y: 0, width: 390, height: 844)
701+
viewport: CGRect(x: 0, y: 0, width: 390, height: 844),
702+
interfaceOrientation: RunnerInterfaceOrientation.portrait
690703
)
691704

692705
let card = nodes.first { $0.label == "feedItem-by-whiskers.test" }
@@ -720,7 +733,8 @@ extension RunnerTests {
720733
raw: false
721734
)
722735
),
723-
viewport: .infinite
736+
viewport: .infinite,
737+
interfaceOrientation: RunnerInterfaceOrientation.portrait
724738
)
725739

726740
let labels = nodes.compactMap { $0.label ?? $0.identifier }
@@ -798,7 +812,9 @@ extension RunnerTests {
798812
let hint = CaptureHint(
799813
projection: .regular, depth: nil, regularPresentedDepth: nil,
800814
interactiveOnly: true, customActions: false)
801-
let acquired = privateAXAcquisition(rawRoot: tree, hint: hint, viewport: viewport)
815+
let acquired = privateAXAcquisition(
816+
rawRoot: tree, hint: hint, viewport: viewport,
817+
interfaceOrientation: RunnerInterfaceOrientation.portrait)
802818
// Acquisition serializes the drawer too; the shared fold is what hides it (#1797).
803819
XCTAssertTrue(acquired.compactMap(\.label).contains("Admin settings"))
804820

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift‎

Lines changed: 17 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,6 @@ private struct RunnerUnsupportedOperationError: LocalizedError {
1111
var errorDescription: String? { message }
1212
}
1313

14-
enum RunnerInterfaceOrientation {
15-
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
16-
static let unknown = 0
17-
#endif
18-
static let portrait = 1
19-
static let portraitUpsideDown = 2
20-
static let landscapeRight = 3
21-
static let landscapeLeft = 4
22-
}
23-
2414
extension RunnerTests {
2515
enum PlannedGestureExecution: Equatable {
2616
case fastSwipe
@@ -497,50 +487,6 @@ extension RunnerTests {
497487
return performCoordinateDrag(app: app, x: x, y: y, x2: x2, y2: y2, holdDuration: holdDuration)
498488
}
499489

500-
/// Rotates an interface-oriented point into the device-native (portrait) space the
501-
/// synthesized event path consumes — synthesized events skip XCTest's orientation
502-
/// handling, so without this a landscape tap lands in the wrong place.
503-
func nativeSynthesizedPoint(
504-
orientedX x: Double,
505-
orientedY y: Double,
506-
in frame: CGRect,
507-
interfaceOrientation: Int
508-
) -> CGPoint {
509-
let localX = x - Double(frame.minX)
510-
let localY = y - Double(frame.minY)
511-
let width = Double(frame.width)
512-
let height = Double(frame.height)
513-
switch interfaceOrientation {
514-
case RunnerInterfaceOrientation.landscapeRight:
515-
return CGPoint(x: height - localY, y: localX)
516-
case RunnerInterfaceOrientation.landscapeLeft:
517-
return CGPoint(x: localY, y: width - localX)
518-
case RunnerInterfaceOrientation.portraitUpsideDown:
519-
return CGPoint(x: width - localX, y: height - localY)
520-
default: // portrait or unknown
521-
return CGPoint(x: localX, y: localY)
522-
}
523-
}
524-
525-
/// Rotates an interface-oriented translation vector into the same native
526-
/// coordinate space as `nativeSynthesizedPoint`.
527-
func nativeSynthesizedVector(
528-
orientedDx dx: Double,
529-
orientedDy dy: Double,
530-
interfaceOrientation: Int
531-
) -> CGVector {
532-
switch interfaceOrientation {
533-
case RunnerInterfaceOrientation.landscapeRight:
534-
return CGVector(dx: -dy, dy: dx)
535-
case RunnerInterfaceOrientation.landscapeLeft:
536-
return CGVector(dx: dy, dy: -dx)
537-
case RunnerInterfaceOrientation.portraitUpsideDown:
538-
return CGVector(dx: -dx, dy: -dy)
539-
default: // portrait or unknown
540-
return CGVector(dx: dx, dy: dy)
541-
}
542-
}
543-
544490
func synthesizedDragAt(
545491
app: XCUIApplication,
546492
x: Double,
@@ -569,8 +515,16 @@ extension RunnerTests {
569515
)
570516
}
571517
let frame = context.referenceFrame
572-
let start = nativeSynthesizedPoint(orientedX: x, orientedY: y, in: frame, interfaceOrientation: orientation)
573-
let end = nativeSynthesizedPoint(orientedX: x2, orientedY: y2, in: frame, interfaceOrientation: orientation)
518+
let start = CoordinateSpaceRotation.native(
519+
point: CGPoint(x: x, y: y),
520+
in: frame,
521+
interfaceOrientation: orientation
522+
)
523+
let end = CoordinateSpaceRotation.native(
524+
point: CGPoint(x: x2, y: y2),
525+
in: frame,
526+
interfaceOrientation: orientation
527+
)
574528
let message = switch profile {
575529
case .continuous:
576530
RunnerSynthesizedGesture.synthesizeContinuousDrag(
@@ -644,7 +598,11 @@ extension RunnerTests {
644598
)
645599
}
646600
let frame = context.referenceFrame
647-
let point = nativeSynthesizedPoint(orientedX: x, orientedY: y, in: frame, interfaceOrientation: orientation)
601+
let point = CoordinateSpaceRotation.native(
602+
point: CGPoint(x: x, y: y),
603+
in: frame,
604+
interfaceOrientation: orientation
605+
)
648606
if let message = RunnerSynthesizedGesture.synthesizeTap(
649607
withApplication: app,
650608
x: Double(point.x),
@@ -1055,9 +1013,8 @@ extension RunnerTests {
10551013
)
10561014
let pointerSamples: [[[String: NSNumber]]] = plan.pointers.map { pointer in
10571015
pointer.samples.map { sample in
1058-
let point = nativeSynthesizedPoint(
1059-
orientedX: sample.point.x,
1060-
orientedY: sample.point.y,
1016+
let point = CoordinateSpaceRotation.native(
1017+
point: CGPoint(x: sample.point.x, y: sample.point.y),
10611018
in: frame,
10621019
interfaceOrientation: orientation
10631020
)
@@ -1178,53 +1135,6 @@ extension RunnerTests {
11781135
#endif
11791136

11801137
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
1181-
// Identity in portrait/unknown, 90° per landscape, 180° upside-down.
1182-
func testNativeSynthesizedPointRotatesByInterfaceOrientation() {
1183-
let portrait = CGRect(x: 0, y: 0, width: 834, height: 1210)
1184-
let landscape = CGRect(x: 0, y: 0, width: 1210, height: 834)
1185-
let offsetLandscape = CGRect(x: 10, y: 20, width: 1210, height: 834)
1186-
// (frame, UIInterfaceOrientation, expected native point) for a tap at (170, 268).
1187-
let cases: [(CGRect, Int, CGPoint)] = [
1188-
(portrait, RunnerInterfaceOrientation.portrait, CGPoint(x: 170, y: 268)),
1189-
(landscape, RunnerInterfaceOrientation.landscapeRight, CGPoint(x: 566, y: 170)),
1190-
(landscape, RunnerInterfaceOrientation.landscapeLeft, CGPoint(x: 268, y: 1040)),
1191-
(portrait, RunnerInterfaceOrientation.portraitUpsideDown, CGPoint(x: 664, y: 942)),
1192-
(portrait, RunnerInterfaceOrientation.unknown, CGPoint(x: 170, y: 268)),
1193-
]
1194-
for (frame, orientation, expected) in cases {
1195-
XCTAssertEqual(
1196-
nativeSynthesizedPoint(orientedX: 170, orientedY: 268, in: frame, interfaceOrientation: orientation),
1197-
expected,
1198-
"interfaceOrientation \(orientation)"
1199-
)
1200-
}
1201-
XCTAssertEqual(
1202-
nativeSynthesizedPoint(
1203-
orientedX: 180,
1204-
orientedY: 288,
1205-
in: offsetLandscape,
1206-
interfaceOrientation: RunnerInterfaceOrientation.landscapeLeft
1207-
),
1208-
CGPoint(x: 268, y: 1040),
1209-
"non-zero frame origin is localized before rotation"
1210-
)
1211-
}
1212-
1213-
func testNativeSynthesizedVectorRotatesByInterfaceOrientation() {
1214-
let cases: [(Int, CGVector)] = [
1215-
(RunnerInterfaceOrientation.portrait, CGVector(dx: 40, dy: -20)),
1216-
(RunnerInterfaceOrientation.landscapeRight, CGVector(dx: 20, dy: 40)),
1217-
(RunnerInterfaceOrientation.landscapeLeft, CGVector(dx: -20, dy: -40)),
1218-
(RunnerInterfaceOrientation.portraitUpsideDown, CGVector(dx: -40, dy: 20)),
1219-
(RunnerInterfaceOrientation.unknown, CGVector(dx: 40, dy: -20)),
1220-
]
1221-
for (orientation, expected) in cases {
1222-
let vector = nativeSynthesizedVector(orientedDx: 40, orientedDy: -20, interfaceOrientation: orientation)
1223-
XCTAssertEqual(vector.dx, expected.dx, "dx interfaceOrientation \(orientation)")
1224-
XCTAssertEqual(vector.dy, expected.dy, "dy interfaceOrientation \(orientation)")
1225-
}
1226-
}
1227-
12281138
func testSynthesizedScreenshotReferenceFrameUsesScreenshotSize() throws {
12291139
let resolved = try XCTUnwrap(
12301140
orientedSynthesizedScreenshotReferenceFrame(

0 commit comments

Comments
 (0)