Skip to content

Commit 1fbcbfa

Browse files
committed
fix(ios): book every surface on the lit panel, not the first window
On an iPhone Duo home screen the application's first window is SpringBoard's status-bar window: a 223x317 box on a panel that may be dark. Taps, the snapshot viewport, the capture display and the synthesized record all took that window as "the screen", so every one of them answered for a screen nobody was looking at. One rule now names the window a command books its surface against (RunnerPanelSurface): the window the host reports as the lit panel shows. The host reads panel power it already has and sends the lit panel as a typed fact on the requests that resolve a screen; a request that names no panel keeps the historic first-window walk, so single-panel iPhone, iPad and tvOS behavior is byte-for-byte the request and the read it always were. Geometry never guesses a panel: the dark panel's windows have perfectly good frames. A regular capture additionally applies the panel fact to its tree (RunnerPanelSurfaceTree): the root carries the surface box the host reads as the payload's viewport, surfaces off the panel are dropped with their subtrees, and when the application's subtree carries no window for the lit panel the resolved window is read once and grafted, which is what makes the home screen's icons addressable. Raw retains reported frames. A device whose every window sits on the dark panel refuses with APP_SCREEN_OFF_LIT_PANEL rather than capturing black. Device evidence (iPhone Duo 9542F4F5, iPhone 17 Pro BC54AC11): - folded: lit=1(466x678) selection=litPanelWindow frame=(0,0 466x678); home-screen snapshot lists icons; icon long-press opens the icon menu; Safari tap opens Safari; SYNTHESIZED_DISPATCH reference=(0,0 466x678), SYNTHESIZED_RECORD displayID=1; screenshot 466x678. - unfolded: lit=3(669x951) frame=(0,0 951x669), root republished from the dark panel's 466x678, 4 off-panel surfaces dropped, tree grafted to 67 nodes; snapshot lists icons; long-press opens the context menu; Settings tap opens Settings; reference=(0,0 951x669), record displayID=3; screenshot 951x669. - iPhone 17 Pro: tap and snapshot unchanged, no panel fact on the wire, no panel markers in runner.log, reference=(0,0 402x874).
1 parent ff8c228 commit 1fbcbfa

43 files changed

Lines changed: 2428 additions & 215 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerAppScreenCapture.swift‎

Lines changed: 37 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,16 @@ enum RunnerAppScreenCaptureFailure: String, Error {
2121
case unresolvedWindow = "APP_SCREEN_WINDOW_UNRESOLVED"
2222
case unresolvedScreen = "APP_SCREEN_UNRESOLVED"
2323
case unrenderableImage = "APP_SCREEN_CAPTURE_UNRENDERABLE"
24+
/// The request named a lit panel and every window it found is on the other one. Guessing the bigger
25+
/// window would capture a screen nobody is looking at, so the capture refuses with this instead.
26+
case offLitPanel = "APP_SCREEN_OFF_LIT_PANEL"
2427

2528
init(_ failure: RunnerApplicationScreenFailure) {
2629
switch failure {
2730
case .unresolvedWindow:
2831
self = .unresolvedWindow
32+
case .offLitPanel:
33+
self = .offLitPanel
2934
default:
3035
self = .unresolvedScreen
3136
}
@@ -39,6 +44,8 @@ enum RunnerAppScreenCaptureFailure: String, Error {
3944
return "The resolved app window did not report the display it is shown on."
4045
case .unrenderableImage:
4146
return "The resolved display handed back an image that could not be drawn upright."
47+
case .offLitPanel:
48+
return "Every window the app and the system surface reported is on the panel the host reports as dark."
4249
}
4350
}
4451

@@ -50,19 +57,26 @@ enum RunnerAppScreenCaptureFailure: String, Error {
5057
return "Retry the capture. If it persists, the XCTest runtime is not reporting the window's display; runner.log names the window frame it resolved."
5158
case .unrenderableImage:
5259
return "Retry the capture. The display was resolved, so this is the capture itself failing, not a panel question."
60+
case .offLitPanel:
61+
return "Read the device's fold pose and panel table: the host named a lit panel no window occupies. runner.log names the panel and the window frames it refused."
5362
}
5463
}
5564
}
5665

5766
extension RunnerTests {
58-
/// The target rule, kept apart from the two queries so the rule itself is testable: an unresolved
59-
/// window asks the system surface, and nothing else does.
67+
/// The target rule, kept apart from the two queries so the rule itself is testable: a process that
68+
/// names no window, or none on the lit panel, asks the system surface, and nothing else does.
69+
///
70+
/// A panel refusal moves on for the same reason an unresolved window does: the home screen is
71+
/// SpringBoard's window, so an app with nothing on the lit panel says nothing about where the lit
72+
/// panel's content is. The system surface is asked, and if neither names a surface on it, the
73+
/// panel refusal is the answer the host needs.
6074
func selectObservedScreenCapture(
6175
resolving: () -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure>,
6276
fallingBack: () -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure>
6377
) -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> {
6478
let outcome = resolving()
65-
if case .failure(.unresolvedWindow) = outcome {
79+
if case .failure(let failure) = outcome, failure == .unresolvedWindow || failure == .offLitPanel {
6680
return fallingBack()
6781
}
6882
return outcome
@@ -82,14 +96,20 @@ extension RunnerTests {
8296
///
8397
/// Never activates `app`: an observation must not change which app is foregrounded just to get a
8498
/// screen, so the caller passes the app context it already intends to observe.
99+
///
100+
/// The panel arrives as an argument rather than being read here: an operation books its whole screen
101+
/// on the panel its request named, and a snapshot of that fact cannot be replaced under it by the
102+
/// command that follows. Callers that live for one command read `requestSurfacePanel` on the command
103+
/// queue and pass what they got, so the panel survives any hop to main.
85104
func captureResolvedAppScreen(
86-
app: XCUIApplication
105+
app: XCUIApplication,
106+
litPanel: RunnerLitPanel
87107
) -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> {
88108
var outcome: Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> = .failure(
89109
.unresolvedScreen
90110
)
91111
let capture = {
92-
outcome = Self.resolveCapturedAppScreen(app: app)
112+
outcome = Self.resolveCapturedAppScreen(app: app, litPanel: litPanel)
93113
}
94114
if Thread.isMainThread {
95115
capture()
@@ -106,25 +126,29 @@ extension RunnerTests {
106126
/// keeps that capture honest — it names a display a window actually occupies instead of reaching
107127
/// for `XCUIScreen.main`, which on a foldable is one fixed panel whether or not it is dark (#2728).
108128
///
109-
/// Only an unresolved *window* moves on to the system surface. A window that resolved and then
110-
/// refused to name its display is the failure the host has to see, and no second question about a
111-
/// different process should bury it.
112-
func captureObservedScreen(app: XCUIApplication) -> Result<
129+
/// Only an unresolved window or a panel refusal moves on to the system surface. A window that
130+
/// resolved on the right panel and then refused to name its display is the failure the host has to
131+
/// see, and no second question about a different process should bury it.
132+
func captureObservedScreen(
133+
app: XCUIApplication,
134+
litPanel: RunnerLitPanel
135+
) -> Result<
113136
CapturedAppScreen, RunnerAppScreenCaptureFailure
114137
> {
115138
selectObservedScreenCapture(
116-
resolving: { captureResolvedAppScreen(app: app) },
117-
fallingBack: { captureResolvedAppScreen(app: springboard) }
139+
resolving: { captureResolvedAppScreen(app: app, litPanel: litPanel) },
140+
fallingBack: { captureResolvedAppScreen(app: springboard, litPanel: litPanel) }
118141
)
119142
}
120143

121144
private static func resolveCapturedAppScreen(
122-
app: XCUIApplication
145+
app: XCUIApplication,
146+
litPanel: RunnerLitPanel
123147
) -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> {
124148
var screen: AnyObject?
125149
var displayID: UInt = 0
126150
var resolution: RunnerApplicationScreenFailure = .unresolvedScreen
127-
guard RunnerResolveApplicationScreen(app, &screen, &displayID, &resolution),
151+
guard RunnerResolveApplicationScreen(app, litPanel, &screen, &displayID, &resolution),
128152
let captureScreen = screen as? XCUIScreen
129153
else {
130154
return .failure(RunnerAppScreenCaptureFailure(resolution))
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
// RunnerPanelSurface.h — the one rule that names the window a command books its surface against.
2+
//
3+
// Four runner paths need "the window the screen is": the interaction anchor and reference frame, the
4+
// snapshot viewport, the display a capture names, and the display a synthesized event record routes
5+
// to. Each used to take the application's first window. On a foldable home screen that is
6+
// SpringBoard's status-bar window — a 223x317 box on a panel that may be dark — so taps, the
7+
// snapshot viewport, and captures all landed on a screen nobody was looking at. They resolve through
8+
// this rule instead.
9+
//
10+
// Only this file declares which window wins. The reads that feed it stay with their owners: the
11+
// XCTest bridge reads an application's windows and their display identity, and hands the facts here,
12+
// so a device the rule cannot settle is refused the same way by every caller.
13+
14+
#import <Foundation/Foundation.h>
15+
#import <CoreGraphics/CoreGraphics.h>
16+
17+
NS_ASSUME_NONNULL_BEGIN
18+
19+
// The panel the host reports as lit, in the units the panel table reports it: which display is
20+
// powered, and that panel's own native box in points. The runner cannot read a hinge, and geometry
21+
// cannot say which of two panels is powered, so only the host can name this (ADR 0025): it reads the
22+
// panel table for the request and sends the answer with it. A request that carried no panel leaves
23+
// `displayID` 0.
24+
//
25+
// The size is the panel's *native* box, not the box an app shows on it. The system presents the inner
26+
// panel turned, so its interaction space swaps the side lengths (ADR 0025), which is why the rule
27+
// compares a box either way up.
28+
typedef struct {
29+
NSUInteger displayID;
30+
CGFloat widthPt;
31+
CGFloat heightPt;
32+
} RunnerLitPanel;
33+
34+
// Whether a request named a panel at all. Only a real display identity counts: a panel with no
35+
// display cannot be compared against a window, so it reads as no panel.
36+
static inline BOOL RunnerLitPanelIsNamed(RunnerLitPanel panel) {
37+
return panel.displayID != 0;
38+
}
39+
40+
// One window the runner measured: the display it is shown on, and the box it occupies in the app's
41+
// coordinate space.
42+
typedef struct {
43+
NSUInteger displayID;
44+
CGRect frame;
45+
} RunnerPanelWindowFact;
46+
47+
// What the rule made of an application's windows. Every outcome but `NoWindow` names the window it
48+
// chose or the first one it looked at, through `selected`, so a refusal can report the frame it
49+
// refused instead of returning a silent nil.
50+
typedef NS_ENUM(NSUInteger, RunnerPanelSurfaceSelection) {
51+
// A window on the lit display whose box is that panel's box, either way up: the screen to book.
52+
RunnerPanelSurfaceSelectionLitPanelWindow,
53+
// Windows exist on the lit display but none carries the panel's box, so the app occupies part of
54+
// it (Split View, a stage, a sheet). The largest is the surface to book: being on the lit panel is
55+
// the fact a first-window walk could never establish.
56+
RunnerPanelSurfaceSelectionLitDisplayWindow,
57+
// Every window the application reported is on a display the host reports as dark.
58+
RunnerPanelSurfaceSelectionOffLitPanel,
59+
// No window answered a usable frame, so there is no surface to book right now.
60+
RunnerPanelSurfaceSelectionNoWindow,
61+
};
62+
63+
// How many windows one walk reads. A device presenting more qualifying windows than this is not a
64+
// surface this rule can settle, and reading past the cap would only widen the risk of reading a
65+
// window that has already vanished (#2995).
66+
#define AGENT_DEVICE_MAX_PANEL_SURFACE_WINDOWS 8
67+
68+
// Half a point: the rounding tolerance the full-screen proof already uses. Sub-pixel layout is not a
69+
// different panel.
70+
#define AGENT_DEVICE_PANEL_SURFACE_TOLERANCE 0.5
71+
72+
// Whether `frame` is the lit panel's surface: anchored at the origin, and the panel's size either way
73+
// up. A frame that is not a measurement at all never qualifies.
74+
FOUNDATION_EXPORT BOOL RunnerPanelFrameIsLitPanelSurface(
75+
CGRect frame,
76+
RunnerLitPanel litPanel
77+
);
78+
79+
// Decides which of `windows` a command books against, from the facts alone: no queries, no device
80+
// state, no clock. `litPanel` must name a display: a caller whose request carried no panel has
81+
// nothing to compare a window against and keeps its historic first-window route instead of calling
82+
// this. `count` is capped by the caller at `AGENT_DEVICE_MAX_PANEL_SURFACE_WINDOWS`.
83+
//
84+
// `selected` reports the window the decision was made from and `selectedIndex` its slot in `windows`,
85+
// so a caller keeps the window element it measured instead of reading it again. On a refusal
86+
// `selected` is the first window it looked at, or a null frame when there were none, so the caller's
87+
// refusal names what it saw.
88+
FOUNDATION_EXPORT RunnerPanelSurfaceSelection RunnerPanelSurfaceSelectWindow(
89+
const RunnerPanelWindowFact *windows,
90+
NSUInteger count,
91+
RunnerLitPanel litPanel,
92+
RunnerPanelWindowFact *selected,
93+
NSUInteger *selectedIndex
94+
);
95+
96+
NS_ASSUME_NONNULL_END
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#import "RunnerPanelSurface.h"
2+
#import "ForegroundApplicationGeometry.h"
3+
#import <math.h>
4+
5+
BOOL RunnerPanelFrameIsLitPanelSurface(CGRect frame, RunnerLitPanel litPanel) {
6+
if (!AgentDeviceForegroundFrameIsMeasurable(frame)) {
7+
return NO;
8+
}
9+
if (fabs(frame.origin.x) > AGENT_DEVICE_PANEL_SURFACE_TOLERANCE ||
10+
fabs(frame.origin.y) > AGENT_DEVICE_PANEL_SURFACE_TOLERANCE) {
11+
return NO;
12+
}
13+
if (!isfinite(litPanel.widthPt) || !isfinite(litPanel.heightPt)) {
14+
return NO;
15+
}
16+
BOOL upright =
17+
fabs(frame.size.width - litPanel.widthPt) <= AGENT_DEVICE_PANEL_SURFACE_TOLERANCE &&
18+
fabs(frame.size.height - litPanel.heightPt) <= AGENT_DEVICE_PANEL_SURFACE_TOLERANCE;
19+
// A panel the system presents turned reports its window in that panel's own native space, so the
20+
// iPhone Duo inner panel's 951x669 interaction box is the transpose of the 669x951 the panel table
21+
// names for it (ADR 0025).
22+
BOOL turned =
23+
fabs(frame.size.width - litPanel.heightPt) <= AGENT_DEVICE_PANEL_SURFACE_TOLERANCE &&
24+
fabs(frame.size.height - litPanel.widthPt) <= AGENT_DEVICE_PANEL_SURFACE_TOLERANCE;
25+
return upright || turned;
26+
}
27+
28+
static CGFloat RunnerPanelWindowFrameArea(CGRect frame) {
29+
CGFloat width = frame.size.width > 1 ? frame.size.width : 1;
30+
CGFloat height = frame.size.height > 1 ? frame.size.height : 1;
31+
return width * height;
32+
}
33+
34+
RunnerPanelSurfaceSelection RunnerPanelSurfaceSelectWindow(
35+
const RunnerPanelWindowFact *windows,
36+
NSUInteger count,
37+
RunnerLitPanel litPanel,
38+
RunnerPanelWindowFact *selected,
39+
NSUInteger *selectedIndex
40+
) {
41+
if (selected != NULL) {
42+
*selected = (RunnerPanelWindowFact){.displayID = 0, .frame = CGRectNull};
43+
}
44+
if (selectedIndex != NULL) {
45+
*selectedIndex = NSUIntegerMax;
46+
}
47+
if (windows == NULL || count == 0) {
48+
return RunnerPanelSurfaceSelectionNoWindow;
49+
}
50+
if (selected != NULL) {
51+
*selected = windows[0];
52+
}
53+
if (selectedIndex != NULL) {
54+
*selectedIndex = 0;
55+
}
56+
57+
RunnerPanelWindowFact largestPlaced = windows[0];
58+
NSUInteger largestPlacedIndex = 0;
59+
BOOL sawPlaced = NO;
60+
CGFloat largestPlacedArea = 0;
61+
for (NSUInteger index = 0; index < count; index += 1) {
62+
RunnerPanelWindowFact candidate = windows[index];
63+
if (candidate.displayID != litPanel.displayID) {
64+
continue;
65+
}
66+
if (RunnerPanelFrameIsLitPanelSurface(candidate.frame, litPanel)) {
67+
if (selected != NULL) {
68+
*selected = candidate;
69+
}
70+
if (selectedIndex != NULL) {
71+
*selectedIndex = index;
72+
}
73+
return RunnerPanelSurfaceSelectionLitPanelWindow;
74+
}
75+
CGFloat area = RunnerPanelWindowFrameArea(candidate.frame);
76+
if (!sawPlaced || area > largestPlacedArea) {
77+
sawPlaced = YES;
78+
largestPlacedArea = area;
79+
largestPlaced = candidate;
80+
largestPlacedIndex = index;
81+
}
82+
}
83+
if (sawPlaced) {
84+
if (selected != NULL) {
85+
*selected = largestPlaced;
86+
}
87+
if (selectedIndex != NULL) {
88+
*selectedIndex = largestPlacedIndex;
89+
}
90+
return RunnerPanelSurfaceSelectionLitDisplayWindow;
91+
}
92+
// The lit display named no window, so every window the app has is on the dark panel. Taking the
93+
// biggest of them by geometry would book a gesture or a capture on a screen nobody is looking at,
94+
// so this refuses and leaves the caller to report the panel it looked at.
95+
return RunnerPanelSurfaceSelectionOffLitPanel;
96+
}

0 commit comments

Comments
 (0)