Skip to content

Commit c8ad8fb

Browse files
thymikeeclaude
andauthored
fix(ios): size sole-display gestures from XCTest screen metrics (callstack#4157)
* fix(ios): size the sole display from the system app, not the runner's UIScreen The generated AgentDeviceRunnerUITests-Runner.app has no launch screen, so the runner process can read UIScreen.main in compatibility mode (768x1024 on an iPad, a letterbox on an iPhone). The synthesized rotation then turned landscape touches about that box, and every press landed a fixed offset away while reporting ok. Read the display from the AXRuntime system application frame, the same screen the fills-screen proof already compares against. Fixes callstack#4091 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): log a failed system-app screen read and harden its test Log AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=system-app when the AXRuntime read fails, rename the reference-frame size to portraitSize (it is points), resolve every UIScreen getter before swapping any, and skip the display test unless the host app fills the screen. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): read the expected screen from the host app, within the fill tolerance The gesture-viewport test now expects the host app's frame instead of the runner's UIScreen, and both display tests compare within the 0.5 pt the fills-screen proof allows. The sole-screen frame doc names its unit. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): refuse a synthesized gesture when the sole display cannot be measured A sole display whose size AXRuntime would not report fell back to the resolved window's box, which on a placed iPad window is the callstack#3899 basis. synthesizedRotationFrame now answers nil only for that case, and tap, long press, drag and planned gestures refuse with UNSUPPORTED_OPERATION before they log or dispatch. A multi-screen host keeps the window route. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): log every failed system-app screen read and keep the hint key-free The system-app size read now logs an unavailable AXRuntime class too, the refusal hint points at the runner log without quoting its key, the rotation frame doc says an unknown orientation keeps the window box (no turn reads it), and the display test clears its cached target. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(ios): say what the sole-screen route pays for after the display read Admission on an iPhone still skips the fills-screen probe, but every device now pays one AXRuntime read (~0.6-0.9 ms) when the caller sizes the display. Co-Authored-By: Claude Opus 5.5 <[email protected]> * perf(ios): read the sole display size once per runner A device's display size does not change while the runner runs, so the first successful AXRuntime read (~0.6-0.9 ms) is kept on the main-owned state and later gestures reuse it; a failed read is not kept, so the refusal stays reachable and the next gesture reads again. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): keep the sole display size for one command, not the runner's life Display scaling can change the system application frame while the runner runs, so a size kept for the whole runner could turn later gestures about a stale box. Each command clears it and reads again; a command's gestures still share one successful read. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): turn about a full-screen app's window when the display size is unreadable A failed AXRuntime display read refused every synthesized gesture on a single-screen device, including iPhone taps that worked through the window before. When the sole-screen rule (RunnerResolveSoleScreen) proves the app fills the screen, its window is the display and stays the rotation basis; otherwise the gesture still refuses with UNSUPPORTED_OPERATION. Both routes log AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=rotation-basis. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): read the display box only for turns that move the axes Portrait and an unknown orientation never read the rotation box, so an unreadable display no longer refuses their gestures and they skip the AXRuntime read entirely. The AX-failure tap tests now run in landscape, where the box is read. Co-Authored-By: Claude Opus 5.5 <[email protected]> * ci(ios): keep the nightly iPad leg at five tests The display-box test is stub-based and runs on the iPhone full suite, so the iPad leg keeps the refusal test, which only an iPad can exercise. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): share the AX-failure tap tests' launch, landscape and stub setup Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(ios): name the sole-screen admission rule and the turns that read the box Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): pin that a portrait tap dispatches when the display is unreadable Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(ios): place the display-size read after admission and spell longpress as the CLI does Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): dispatch a window-less portrait tap when the display size is unreadable With the window query rejected and the AXRuntime display read failing there is no box at all. A portrait tap or long press keeps its point, so on an admitted sole screen it is now dispatched as given; any other turn still refuses with UNSUPPORTED_OPERATION. Both log stage=rotation-basis window=none. Tap and long press share one point-gesture resolver. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): name exactly which turns read the display box Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): read sole-display size from XCTest screen metrics --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent d47bc9d commit c8ad8fb

7 files changed

Lines changed: 550 additions & 66 deletions

File tree

‎.github/workflows/xctest-nightly.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,8 +150,9 @@ jobs:
150150
'
151151
152152
# The iPad leg runs a named selection: the smallest set that can execute the off-phone half of
153-
# the sole-screen rule — the admission boundary itself, one hostile-tree command that must
154-
# reach synthesis, and `back`'s route witness.
153+
# the sole-screen rule — the admission boundary itself, the refusal when the display box a
154+
# synthesized turn swaps cannot be read and the app is not proven full-screen (#4091), one
155+
# hostile-tree command that must reach synthesis, and `back`'s route witness.
155156
# Geometry and error cases live in the host-runnable evidence matrix instead. `xcodebuild`
156157
# exits 0 when a selection matches nothing, so the reporter below asserts the executed count
157158
# against this lane's reach (`XCTEST_LANE=nightly-ipad`, derived by check:xctest-selection).
@@ -169,6 +170,7 @@ jobs:
169170
-xctestrun "$XCTESTRUN_PATH" \
170171
-destination "platform=iOS Simulator,id=$UDID" \
171172
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen \
173+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testTapRefusesBeforeDispatchWhenTheSoleDisplayCannotBeMeasured \
172174
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testLongPressOnAnUnserializableTreeIsSynthesizedOnTheSoleScreen \
173175
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryTapsThroughTheRouteThatOpens \
174176
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryWithUnprovableForegroundTapsThroughTheWindow \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerMainOwnedState.swift‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ final class RunnerMainOwnedState {
99
var processIdentifier: Int?
1010
var accessibilityHealth: RunnerAccessibilityHealth = .unknown
1111
var needsPostSnapshotInteractionDelay = false
12+
/// The device's only screen in portrait points, kept for the rest of the command after its first
13+
/// successful read. Each command reads it again, because display scaling can change it while the
14+
/// runner runs; orientation is applied per gesture.
15+
var soleDisplayPortraitSize: CGSize?
1216

1317
nonisolated init() {}
1418
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ extension RunnerTests {
1010
alertDeadline: Date? = nil
1111
) throws -> Response {
1212
var activeApp = activeApp
13+
mainOwned.soleDisplayPortraitSize = nil
1314
if command.invalidatesRememberedTextEntryTap {
1415
clearRememberedTextEntryTap()
1516
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift‎

Lines changed: 172 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,11 @@ extension RunnerTests {
2828
)
2929
}
3030
let frame = context.referenceFrame
31-
let rotationFrame = synthesizedRotationFrame(fallback: frame, interfaceOrientation: orientation)
31+
guard
32+
let rotationFrame = synthesizedRotationFrame(app: app, fallback: frame, interfaceOrientation: orientation)
33+
else {
34+
return Self.unmeasuredSoleDisplayOutcome(gesture: "drag")
35+
}
3236
let start = CoordinateSpaceRotation.native(
3337
point: CGPoint(x: x, y: y),
3438
in: rotationFrame,
@@ -104,28 +108,28 @@ extension RunnerTests {
104108
)
105109
}
106110
let orientation = Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
107-
guard let context = context ?? synthesizedCoordinateContext(app: app) else {
108-
return .unsupported(
109-
message: "synthesized coordinate tap could not resolve an app window with a finite screen frame",
110-
hint: "Retry after the app is foregrounded, or use a plain screenshot to choose coordinates."
111-
)
112-
}
113-
let point = CoordinateSpaceRotation.native(
114-
point: CGPoint(x: x, y: y),
115-
in: synthesizedRotationFrame(fallback: context.referenceFrame, interfaceOrientation: orientation),
116-
interfaceOrientation: orientation
111+
let resolution = pointGestureTarget(
112+
app: app,
113+
x: x,
114+
y: y,
115+
orientation: orientation,
116+
context: context ?? synthesizedCoordinateContext(app: app),
117+
gesture: .tap
117118
)
119+
guard case .target(let point, let resolvedWindow, let referenceFrame) = resolution else {
120+
return resolution.refusal
121+
}
118122
logSynthesizedDispatch(
119123
kind: "tap",
120124
start: point,
121125
end: nil,
122-
referenceFrame: context.referenceFrame,
126+
referenceFrame: referenceFrame,
123127
orientation: orientation
124128
)
125129
var message: NSString?
126130
let status = RunnerSynthesizedGesture.synthesizeTap(
127131
withApplication: app,
128-
resolvedWindow: context.resolvedWindow,
132+
resolvedWindow: resolvedWindow,
129133
x: Double(point.x),
130134
y: Double(point.y),
131135
deadline: deadline,
@@ -177,23 +181,23 @@ extension RunnerTests {
177181
hint: "Retry with finite x and y values and a positive durationMs."
178182
)
179183
}
180-
guard let context else {
181-
return .unsupported(
182-
message: "synthesized coordinate long press could not resolve an app window with a finite screen frame",
183-
hint: "Retry after the app is foregrounded, or use a plain screenshot to choose coordinates."
184-
)
185-
}
186184
let orientation = Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
187-
let point = CoordinateSpaceRotation.native(
188-
point: CGPoint(x: x, y: y),
189-
in: synthesizedRotationFrame(fallback: context.referenceFrame, interfaceOrientation: orientation),
190-
interfaceOrientation: orientation
185+
let resolution = pointGestureTarget(
186+
app: app,
187+
x: x,
188+
y: y,
189+
orientation: orientation,
190+
context: context,
191+
gesture: .longpress
191192
)
193+
guard case .target(let point, let resolvedWindow, let referenceFrame) = resolution else {
194+
return resolution.refusal
195+
}
192196
logSynthesizedDispatch(
193197
kind: "longPress",
194198
start: point,
195199
end: nil,
196-
referenceFrame: context.referenceFrame,
200+
referenceFrame: referenceFrame,
197201
orientation: orientation
198202
)
199203
let samples: [[String: NSNumber]] = [0, durationMs].map { offsetMs in
@@ -205,7 +209,7 @@ extension RunnerTests {
205209
}
206210
if let message = RunnerSynthesizedGesture.synthesizeGesture(
207211
withApplication: app,
208-
resolvedWindow: context.resolvedWindow,
212+
resolvedWindow: resolvedWindow,
209213
pointerSamples: [samples]
210214
) {
211215
return .unsupported(
@@ -486,47 +490,160 @@ extension RunnerTests {
486490
/// anything else (a windowed iPad, a multi-screen foldable) keeps requiring a window.
487491
@MainActor
488492
func soleScreenReferenceFrame(app: XCUIApplication) -> CGRect? {
489-
var resolvedScreen: AnyObject?
490-
var resolvedDisplayID: UInt = 0
491-
guard RunnerResolveSoleScreen(app, &resolvedScreen, &resolvedDisplayID) == nil else {
492-
return nil
493+
guard soleScreenRouteAdmits(app) else { return nil }
494+
let orientation = Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
495+
let frame = soleDisplayPortraitSize().flatMap {
496+
Self.soleScreenReferenceFrame(portraitSize: $0, interfaceOrientation: orientation)
493497
}
494-
let frame = soleDisplayFrame(
495-
interfaceOrientation: Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
496-
)
497498
NSLog(
498499
"AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved=%@",
499500
frame.map { NSCoder.string(for: $0) } ?? "none"
500501
)
501502
return frame
502503
}
503504

505+
/// The sole-screen admission rule `RunnerResolveSoleScreen` owns: an iPhone by its idiom, an iPad
506+
/// only on the fills-screen proof.
507+
@MainActor
508+
private func soleScreenRouteAdmits(_ app: XCUIApplication) -> Bool {
509+
var resolvedScreen: AnyObject?
510+
var resolvedDisplayID: UInt = 0
511+
return RunnerResolveSoleScreen(app, &resolvedScreen, &resolvedDisplayID) == nil
512+
}
513+
504514
/// The box a synthesized turn swaps side lengths about. The digitizer turns about its display, so a
505515
/// placed iPad window's smaller box is not the basis (#3899); a multi-screen host keeps the
506-
/// resolved window, which names its panel (ADR 0025).
507-
func synthesizedRotationFrame(fallback: CGRect, interfaceOrientation: Int) -> CGRect {
508-
soleDisplayFrame(interfaceOrientation: interfaceOrientation) ?? fallback
516+
/// resolved window `fallback`, which names its panel (ADR 0025). When the device's only screen
517+
/// will not report its size, `fallback` is the basis only while the app provably fills that
518+
/// screen; otherwise this is nil and the gesture refuses before it dispatches. Portrait and an
519+
/// unknown orientation leave the point as it is, so they keep `fallback` without a read; landscape
520+
/// swaps the box's sides and upside-down mirrors the point across them, so both read it.
521+
@MainActor
522+
func synthesizedRotationFrame(app: XCUIApplication, fallback: CGRect, interfaceOrientation: Int) -> CGRect? {
523+
guard XCUIScreen.screens.count == 1 else { return fallback }
524+
switch interfaceOrientation {
525+
case RunnerInterfaceOrientation.landscapeLeft, RunnerInterfaceOrientation.landscapeRight,
526+
RunnerInterfaceOrientation.portraitUpsideDown:
527+
break
528+
default:
529+
return fallback
530+
}
531+
guard let portraitSize = soleDisplayPortraitSize() else {
532+
let routeAdmits = soleScreenRouteAdmits(app)
533+
NSLog(
534+
"AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=rotation-basis basis=%@",
535+
routeAdmits ? "window" : "refused"
536+
)
537+
return routeAdmits ? fallback : nil
538+
}
539+
return Self.soleScreenReferenceFrame(portraitSize: portraitSize, interfaceOrientation: interfaceOrientation)
540+
?? fallback
509541
}
510542

511-
private func soleDisplayFrame(interfaceOrientation: Int) -> CGRect? {
512-
guard XCUIScreen.screens.count == 1 else { return nil }
513-
let screen = UIScreen.main
514-
let scale = screen.nativeScale
515-
guard scale > 0 else { return nil }
516-
return Self.soleScreenReferenceFrame(
517-
nativePortraitSize: CGSize(
518-
width: screen.nativeBounds.width / scale,
519-
height: screen.nativeBounds.height / scale
520-
),
521-
interfaceOrientation: interfaceOrientation
543+
private enum SynthesizedPointGesture: String {
544+
case tap
545+
case longpress
546+
547+
/// The gesture's name in the unresolved-window refusal, which predates this enum.
548+
var unresolvedWindowName: String { self == .tap ? "tap" : "long press" }
549+
}
550+
551+
private enum PointGestureResolution {
552+
/// A nil `resolvedWindow` routes the record to the sole screen.
553+
case target(point: CGPoint, resolvedWindow: XCUIElement?, referenceFrame: CGRect)
554+
case refused(RunnerInteractionOutcome)
555+
556+
/// Read only on the branch that is not a target, so its `.performed` answer is never returned.
557+
var refusal: RunnerInteractionOutcome {
558+
if case .refused(let outcome) = self { return outcome }
559+
return .performed
560+
}
561+
}
562+
563+
/// Where a synthesized point gesture lands: its native point and the window that routes it, or the
564+
/// refusal that stops it before dispatch.
565+
@MainActor
566+
private func pointGestureTarget(
567+
app: XCUIApplication,
568+
x: Double,
569+
y: Double,
570+
orientation: Int,
571+
context: SynthesizedCoordinateContext?,
572+
gesture: SynthesizedPointGesture
573+
) -> PointGestureResolution {
574+
if let context {
575+
guard
576+
let rotationFrame = synthesizedRotationFrame(
577+
app: app,
578+
fallback: context.referenceFrame,
579+
interfaceOrientation: orientation
580+
)
581+
else {
582+
return .refused(Self.unmeasuredSoleDisplayOutcome(gesture: gesture.rawValue))
583+
}
584+
let point = CoordinateSpaceRotation.native(
585+
point: CGPoint(x: x, y: y),
586+
in: rotationFrame,
587+
interfaceOrientation: orientation
588+
)
589+
return .target(point: point, resolvedWindow: context.resolvedWindow, referenceFrame: context.referenceFrame)
590+
}
591+
if let unrotated = unrotatedSoleScreenPoint(app: app, x: x, y: y, interfaceOrientation: orientation) {
592+
return .target(point: unrotated, resolvedWindow: nil, referenceFrame: .null)
593+
}
594+
return .refused(
595+
.unsupported(
596+
message: "synthesized coordinate \(gesture.unresolvedWindowName) could not resolve an app window with a finite screen frame",
597+
hint: "Retry after the app is foregrounded, or use a plain screenshot to choose coordinates."
598+
)
599+
)
600+
}
601+
602+
/// A point gesture with no context: no window resolved and the sole screen's size is unreadable,
603+
/// so there is no box (#4091). On an admitted sole screen a portrait turn leaves the point as it
604+
/// is, so it is dispatched as given; any other turn needs the box and refuses. Nil refuses.
605+
@MainActor
606+
private func unrotatedSoleScreenPoint(
607+
app: XCUIApplication,
608+
x: Double,
609+
y: Double,
610+
interfaceOrientation: Int
611+
) -> CGPoint? {
612+
guard XCUIScreen.screens.count == 1, soleScreenRouteAdmits(app) else { return nil }
613+
let unrotated = interfaceOrientation == RunnerInterfaceOrientation.portrait
614+
NSLog(
615+
"AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=rotation-basis basis=%@ window=none",
616+
unrotated ? "unrotated" : "refused"
617+
)
618+
return unrotated ? CGPoint(x: x, y: y) : nil
619+
}
620+
621+
/// Not `UIScreen.main`: the runner process can read that in compatibility mode (#4091). A failed
622+
/// read is not kept, so the next gesture reads again.
623+
@MainActor
624+
private func soleDisplayPortraitSize() -> CGSize? {
625+
if let cached = mainOwned.soleDisplayPortraitSize { return cached }
626+
let screen = RunnerSoleScreenSize()
627+
guard screen.width > 0, screen.height > 0 else { return nil }
628+
let size = CGSize(width: min(screen.width, screen.height), height: max(screen.width, screen.height))
629+
mainOwned.soleDisplayPortraitSize = size
630+
return size
631+
}
632+
633+
static func unmeasuredSoleDisplayOutcome(gesture: String) -> RunnerInteractionOutcome {
634+
.unsupported(
635+
message:
636+
"synthesized \(gesture) could not read the size of the device's only screen, and the app is not "
637+
+ "proven to fill it, so its coordinates cannot be rotated",
638+
hint: "Retry after the app is foregrounded; the runner log records why the screen size could not be read."
522639
)
523640
}
524641
#endif
525642

526-
/// The screen in the app's interface orientation, anchored at the origin like a window frame. An
527-
/// unknown orientation cannot say which way to lay the screen, so it names no frame.
643+
/// The screen, in points, in the app's interface orientation, anchored at the origin like a window
644+
/// frame. An unknown orientation cannot say which way to lay the screen, so it names no frame.
528645
static func soleScreenReferenceFrame(
529-
nativePortraitSize size: CGSize,
646+
portraitSize size: CGSize,
530647
interfaceOrientation: Int
531648
) -> CGRect? {
532649
guard size.width > 0, size.height > 0 else { return nil }
@@ -678,6 +795,7 @@ extension RunnerTests {
678795
: .sampled
679796
}
680797

798+
@MainActor
681799
func sampledPlannedGesture(
682800
app: XCUIApplication,
683801
plan: RunnerGesturePlan
@@ -692,7 +810,11 @@ extension RunnerTests {
692810
width: plan.viewport.width,
693811
height: plan.viewport.height
694812
)
695-
let rotationFrame = synthesizedRotationFrame(fallback: frame, interfaceOrientation: orientation)
813+
guard
814+
let rotationFrame = synthesizedRotationFrame(app: app, fallback: frame, interfaceOrientation: orientation)
815+
else {
816+
return Self.unmeasuredSoleDisplayOutcome(gesture: "gesture")
817+
}
696818
let pointerSamples: [[[String: NSNumber]]] = plan.pointers.map { pointer in
697819
pointer.samples.map { sample in
698820
let point = CoordinateSpaceRotation.native(

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,13 @@ FOUNDATION_EXPORT NSString * _Nullable RunnerResolveSoleScreen(
7171
NSUInteger *displayID
7272
);
7373

74+
// The sole XCTest display's pixel bounds divided by its scale, in points. Neither the runner's
75+
// compatibility-mode `UIScreen` nor the system application's possibly empty AX frame supplies
76+
// this measurement. Any failed read or a screen count other than one answers `CGSizeZero`.
77+
#if TARGET_OS_IOS
78+
FOUNDATION_EXPORT CGSize RunnerSoleScreenSize(void);
79+
#endif
80+
7481
// What the full-screen proof concluded. The decision is a function of these values, so a test
7582
// supplies them directly instead of restating the device rule.
7683
typedef NS_ENUM(NSUInteger, RunnerFullScreenEvidence) {

0 commit comments

Comments
 (0)