Skip to content

Commit 0d5e5d6

Browse files
thymikeeclaude
andauthored
fix(ios): book every surface on the lit panel, not the first window (#4122)
* fix(ios): book every surface on the lit panel, not the first window On an iPhone Duo home screen the application's first window is SpringBoard's status-bar window: a 223x317 box on a panel that may be dark. Taps, the snapshot viewport, the capture display and the synthesized record all took that window as "the screen", so every one of them answered for a screen nobody was looking at. One rule now names the window a command books its surface against (RunnerPanelSurface): the window the host reports as the lit panel shows. The host reads panel power it already has and sends the lit panel as a typed fact on the requests that resolve a screen; a request that names no panel keeps the historic first-window walk, so single-panel iPhone, iPad and tvOS behavior is byte-for-byte the request and the read it always were. Geometry never guesses a panel: the dark panel's windows have perfectly good frames. A regular capture additionally applies the panel fact to its tree (RunnerPanelSurfaceTree): the root carries the surface box the host reads as the payload's viewport, surfaces off the panel are dropped with their subtrees, and when the application's subtree carries no window for the lit panel the resolved window is read once and grafted, which is what makes the home screen's icons addressable. Raw retains reported frames. A device whose every window sits on the dark panel refuses with APP_SCREEN_OFF_LIT_PANEL rather than capturing black. Device evidence (iPhone Duo 9542F4F5, iPhone 17 Pro BC54AC11): - folded: lit=1(466x678) selection=litPanelWindow frame=(0,0 466x678); home-screen snapshot lists icons; icon long-press opens the icon menu; Safari tap opens Safari; SYNTHESIZED_DISPATCH reference=(0,0 466x678), SYNTHESIZED_RECORD displayID=1; screenshot 466x678. - unfolded: lit=3(669x951) frame=(0,0 951x669), root republished from the dark panel's 466x678, 4 off-panel surfaces dropped, tree grafted to 67 nodes; snapshot lists icons; long-press opens the context menu; Settings tap opens Settings; reference=(0,0 951x669), record displayID=3; screenshot 951x669. - iPhone 17 Pro: tap and snapshot unchanged, no panel fact on the wire, no panel markers in runner.log, reference=(0,0 402x874). * refactor(ios): resolve one lit-panel window per request Reduce the foldable surface fix to one rule. The host reads the panel table once per surface command (the command traits name which ones) and sends `litPanel`. The runner walks at most eight windows, books the first full-panel window on the lit display, and uses it for dispatch, the coordinate fallback, the viewport, and capture. A regular snapshot keeps only the windows whose element snapshot names the lit display. Requests without `litPanel` keep main's first-window routes unchanged. The graft-and-prune tree pass, the panel-rule module, the APP_SCREEN_OFF_LIT_PANEL code, and the rotation retry option are gone. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: correct the lit-panel comments a review flagged Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): fail closed when the lit panel cannot be booked A request that names a lit panel now books that panel's window before it acts. A refusal comes back typed before any gesture or snapshot, and every later read in the request reuses the booked window, so nothing falls back to the application's own coordinates. - The window walk refuses after an unplaced window, even when a full-panel window follows it. - An app whose windows are all on the dark panel is APP_SCREEN_OFF_LIT_PANEL, and a capture no longer swaps it for SpringBoard. - A lit-panel snapshot runs only the tree tier and keeps only windows proven on the lit display. - A resolved panel-table read clears the probe back-off, and the probe never outlasts the request's timeoutMs. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): order the panel-table back-off behind newer resolved reads A failed read that finishes after a newer read resolved no longer sets the 5-second back-off: each read takes a sequence number, and a failure yields to any later read that resolved. The ADR now says the snapshot traverses the tree and keeps every window proven on the lit display. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(adr): count the off-lit-panel capture refusal ADR 0025's capture contract listed three fail-closed cases. The lit-panel amendment added a fourth, `offLitPanel` (APP_SCREEN_OFF_LIT_PANEL), and changed when capture asks the system surface on a multi-panel device. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): keep every lit-panel read on proven lit windows - Element reads (readText, the text-input lookup, selector queries) search only under the windows that report the lit display. One bridge owner, RunnerCopyLitPanelWindows, provides them. - The lit panel is written onto the main queue behind work already queued there, so work a previous command abandoned keeps its own panel. The booking is keyed to its app. - A snapshot books in command preparation, publishes the booked frame as its root, refuses a pin to a backend that cannot prove the panel, and filters --raw too. - A known multi-panel device whose panel table goes unread is refused with a typed panel-table-unreadable error. The runner gets the budget the panel read left. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): keep raw, unreadable-list and pin refusals on the lit panel - `--raw` publishes the booked window's frame as its root, so raw and regular coordinates share one space. - RunnerCopyLitPanelWindows returns nil for an unreadable window list. A lit-panel element read then refuses with APP_SCREEN_UNRESOLVED instead of finding nothing. - A backend pin on a lit-panel snapshot is refused in command preparation, so it leaves the cached target alone. - ADR 0025 says what a recording does after a fold: it keeps capturing the panel recordStart named. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): let element search own the lit-panel refusal Every element read on a lit-panel request searches the windows proven on the lit display, or `elementSearchRoots` throws APP_SCREEN_UNRESOLVED, which `executeOnMainPrepared` returns as the reply. This replaces the command-name precheck, so a coordinate `type` whose window list cannot be read refuses instead of tapping a point with no target. The snapshot path reads the booked window only from its capture target. `findText` and `querySelector` join the lit-panel-scoped commands, so the traits table is the one list. A `customActions` snapshot names `customActions` in its refusal. The client no longer short-circuits on a `litPanel` field no caller sets. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): share the lit-panel and penalized-clear test setups The new unreadable-list case and the pin case launched the app on its own panel the same way, and the added `try` made two penalized-clear calls a new clone. One helper each removes both clones. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(adr): qualify when customActions implies the private AX pin Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): refuse an unread lit-panel window list instead of falling back An unreadable window list on a lit-panel request resolved as "no window", which sends a capture to the system-surface fallback. It now refuses with APP_SCREEN_UNRESOLVED. A panel read that spends the whole request budget is refused before sending, instead of handing the runner a 1 ms timeout. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): contain the lit-panel window walk and keep its refusals closed The element search's window walk now runs inside an issue scope, so a window that vanishes mid-walk refuses with APP_SCREEN_UNRESOLVED instead of recording an XCTest failure. A text-entry re-read refused at its refresh point answers no input instead of the app-wide identifier query, and the Maestro-compat frame gate refuses a lit-panel request with no booked window. The snapshot pin refusal keeps one message that names its cause. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(ios): drop the single-caller textInputAt and trim lit-panel comments Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): route every lit-panel text-entry read through the element search The bound-input identifier query and the in-app back control searched the whole application, so on an unfolded Duo a same-identifier input or back button on the dark panel could answer. Both now search the element search's roots, and its refusal propagates: text-entry resolution throws instead of answering nil, and boundTextEntryInputIsGone answers "cannot prove gone". One owner returns the request's remaining budget, so the panel read and the runner timeout share one clock. The unreachable hasTappableFrame guard is removed. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): book the lit panel's whole box for a window that leaves a strip On the unfolded Duo, an app's window is 871x669 beside the inner panel's status strip, and the booking used that window frame as the synthesized dispatch reference. The orientation flip then mapped points against 871 instead of the 951-point screen, so taps landed 80 points off, and a fill's focus tap missed. The booked frame is now the panel's box turned like the window, at its origin, which is unchanged for full-panel windows. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(ios): tighten the lit-panel booking comment Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): share the bound-input fixture target Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): propagate lit-panel refusals from the commit poll and clear reads The synthesized commit wait's observe closure and the replacement clear's read swallowed a lit-panel refusal with `try?`, so the wait timed out or the clear reported TEXT_INPUT_CLEAR_NOT_OBSERVED. Both now throw, and the refusal answers the request. The clear read also keeps main's answer for an input that is simply gone. The bridge exports one lit-panel resolver that returns the window, frame and screen, and lit-panel comments are trimmed. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): book the lit panel's screen box at the screen origin The booked box was inferred from the window: its origin and whether it was wider than tall. A window that the inner panel's status strip pushes to a non-zero origin would then book a box 80 points past the screen, and every tap, reference and snapshot root would land off. The box is now the panel's native portrait box laid out in the app's interface orientation, at the screen origin, through the existing soleScreenReferenceFrame. The XCUICoordinate anchor keeps the window's own origin. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): turn lit-panel taps about the screen, keep the window's translation The previous commit booked the screen box, which broke the placed-window translation: XCTest reports a window at (0,80) from its own top-left, and the snapshot shifts those frames into screen space only when the viewport is that window's frame. The booking is the window frame again, as for a placed iPad window (#3899), and on a lit-panel request the synthesized turn is about the lit panel's screen in the interface orientation instead of the window's shape. That replaces the aspect guess and the window origin inference. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(ios): drive the lit-panel booking for a partial window; scope the walk cap The rotation-basis case now books the launched app's window under a lit panel taller than it and checks the synthesized reference and turn box, so a wiring regression fails it. The one-caller screen-frame helper is inlined. The window cap's doc names the booking walk it bounds; the element-search copy stays uncapped so every lit window is searched. Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(adr): map each capture refusal to its cases and scope the window cap Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): preserve screen points across lit-panel coordinate fallbacks * fix(ios): key lit-panel bookings by app and request identity * docs(adr): scope the panel-probe budget refusal to a named lit panel * fix(ios): refuse unbooked lit-panel reference frames * fix(ios): scope collapsed-tab candidates to the captured lit display * fix(ios): refuse unread lit-panel window frames --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent a2c169f commit 0d5e5d6

70 files changed

Lines changed: 2813 additions & 668 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerAppScreenCapture.swift‎

Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,24 @@ enum RunnerAppScreenCaptureFailure: String, Error {
2121
case unresolvedWindow = "APP_SCREEN_WINDOW_UNRESOLVED"
2222
case unresolvedScreen = "APP_SCREEN_UNRESOLVED"
2323
case unrenderableImage = "APP_SCREEN_CAPTURE_UNRENDERABLE"
24+
case offLitPanel = "APP_SCREEN_OFF_LIT_PANEL"
2425

2526
init(_ failure: RunnerApplicationScreenFailure) {
2627
switch failure {
2728
case .unresolvedWindow:
2829
self = .unresolvedWindow
30+
case .offLitPanel:
31+
self = .offLitPanel
2932
default:
3033
self = .unresolvedScreen
3134
}
3235
}
3336

37+
/// The refusal as a runner reply.
38+
var response: Response {
39+
Response(ok: false, error: ErrorPayload(code: rawValue, message: message, hint: hint))
40+
}
41+
3442
var message: String {
3543
switch self {
3644
case .unresolvedWindow:
@@ -39,6 +47,8 @@ enum RunnerAppScreenCaptureFailure: String, Error {
3947
return "The resolved app window did not report the display it is shown on."
4048
case .unrenderableImage:
4149
return "The resolved display handed back an image that could not be drawn upright."
50+
case .offLitPanel:
51+
return "Every window of the target app is on the panel the host reports as dark."
4252
}
4353
}
4454

@@ -50,6 +60,8 @@ enum RunnerAppScreenCaptureFailure: String, Error {
5060
return "Retry the capture. If it persists, the XCTest runtime is not reporting the window's display; runner.log names the window frame it resolved."
5161
case .unrenderableImage:
5262
return "Retry the capture. The display was resolved, so this is the capture itself failing, not a panel question."
63+
case .offLitPanel:
64+
return "Unfold or fold the device so the target app is on the lit panel, or bring it forward there; runner.log names the panel in AGENT_DEVICE_RUNNER_LIT_PANEL_WINDOW."
5365
}
5466
}
5567
}
@@ -82,14 +94,17 @@ extension RunnerTests {
8294
///
8395
/// Never activates `app`: an observation must not change which app is foregrounded just to get a
8496
/// screen, so the caller passes the app context it already intends to observe.
97+
///
98+
/// `litPanel` is the panel the request named; nil keeps the app's first window.
8599
func captureResolvedAppScreen(
86-
app: XCUIApplication
100+
app: XCUIApplication,
101+
litPanel: RunnerLitPanel?
87102
) -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> {
88103
var outcome: Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> = .failure(
89104
.unresolvedScreen
90105
)
91106
let capture = {
92-
outcome = Self.resolveCapturedAppScreen(app: app)
107+
outcome = Self.resolveCapturedAppScreen(app: app, litPanel: litPanel)
93108
}
94109
if Thread.isMainThread {
95110
capture()
@@ -109,22 +124,33 @@ extension RunnerTests {
109124
/// Only an unresolved *window* moves on to the system surface. A window that resolved and then
110125
/// refused to name its display is the failure the host has to see, and no second question about a
111126
/// different process should bury it.
112-
func captureObservedScreen(app: XCUIApplication) -> Result<
127+
func captureObservedScreen(app: XCUIApplication, litPanel: RunnerLitPanel?) -> Result<
113128
CapturedAppScreen, RunnerAppScreenCaptureFailure
114129
> {
115130
selectObservedScreenCapture(
116-
resolving: { captureResolvedAppScreen(app: app) },
117-
fallingBack: { captureResolvedAppScreen(app: springboard) }
131+
resolving: { captureResolvedAppScreen(app: app, litPanel: litPanel) },
132+
fallingBack: { captureResolvedAppScreen(app: springboard, litPanel: litPanel) }
118133
)
119134
}
120135

121136
private static func resolveCapturedAppScreen(
122-
app: XCUIApplication
137+
app: XCUIApplication,
138+
litPanel: RunnerLitPanel?
123139
) -> Result<CapturedAppScreen, RunnerAppScreenCaptureFailure> {
124140
var screen: AnyObject?
125141
var displayID: UInt = 0
126142
var resolution: RunnerApplicationScreenFailure = .unresolvedScreen
127-
guard RunnerResolveApplicationScreen(app, &screen, &displayID, &resolution),
143+
var resolved: Bool
144+
if let litPanel {
145+
var window: AnyObject?
146+
var frame = CGRect.null
147+
resolution = RunnerResolveLitPanelWindow(app, litPanel, &window, &frame, &screen)
148+
resolved = resolution == .none
149+
displayID = litPanel.displayID
150+
} else {
151+
resolved = RunnerResolveApplicationScreen(app, &screen, &displayID, &resolution)
152+
}
153+
guard resolved,
128154
let captureScreen = screen as? XCUIScreen
129155
else {
130156
return .failure(RunnerAppScreenCaptureFailure(resolution))
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
import XCTest
2+
3+
// The lit panel of the command in flight (ADR 0025), main-owned and written behind main work a
4+
// previous command left queued, so that work still reads its own command's panel.
5+
struct LitPanelBookingKey: Equatable {
6+
let bundleID: String
7+
let requestID: UUID
8+
}
9+
10+
struct LitPanelSurface {
11+
let key: LitPanelBookingKey
12+
let displayID: UInt
13+
let window: XCUIElement
14+
let frame: CGRect
15+
}
16+
17+
/// The command's lit panel and what it booked on it. Read and written only on the main thread.
18+
final class RunnerLitPanelState {
19+
var requestID: UUID?
20+
var panel: RunnerLitPanel?
21+
var surface: LitPanelSurface?
22+
}
23+
24+
extension RunnerTests {
25+
@discardableResult
26+
func setRequestLitPanel(from command: Command) -> UUID {
27+
let requestID = UUID()
28+
var panel: RunnerLitPanel?
29+
if let fact = command.litPanel, fact.displayId != 0, fact.widthPt.isFinite, fact.heightPt.isFinite,
30+
fact.widthPt > 0, fact.heightPt > 0
31+
{
32+
panel = RunnerLitPanel(displayID: fact.displayId, widthPt: fact.widthPt, heightPt: fact.heightPt)
33+
}
34+
let state = litPanelState
35+
let write = {
36+
state.requestID = requestID
37+
state.panel = panel
38+
state.surface = nil
39+
}
40+
if Thread.isMainThread {
41+
write()
42+
} else {
43+
DispatchQueue.main.async(execute: write)
44+
}
45+
return requestID
46+
}
47+
48+
func endRequestLitPanel(_ requestID: UUID) {
49+
let state = litPanelState
50+
let clear = {
51+
guard state.requestID == requestID else { return }
52+
state.requestID = nil
53+
state.panel = nil
54+
state.surface = nil
55+
}
56+
if Thread.isMainThread {
57+
clear()
58+
} else {
59+
DispatchQueue.main.async(execute: clear)
60+
}
61+
}
62+
63+
/// The lit panel the current request named, or nil for a single-panel request. Main only.
64+
var requestLitPanel: RunnerLitPanel? { litPanelState.panel }
65+
66+
/// Books `app`'s lit-panel window for the request, or returns why it cannot.
67+
func bookRequestLitPanelSurface(app: XCUIApplication) -> RunnerAppScreenCaptureFailure? {
68+
guard let litPanel = requestLitPanel else { return nil }
69+
guard let requestID = litPanelState.requestID,
70+
let bundleID = safely("LIT_PANEL_APP_IDENTITY", { app.value(forKey: "bundleID") as? String })?.trimmedNonEmpty
71+
else { return .unresolvedScreen }
72+
let key = LitPanelBookingKey(bundleID: bundleID, requestID: requestID)
73+
if litPanelState.surface?.key == key { return nil }
74+
litPanelState.surface = nil
75+
var window: AnyObject?
76+
var frame = CGRect.null
77+
var screen: AnyObject?
78+
let failure = RunnerResolveLitPanelWindow(app, litPanel, &window, &frame, &screen)
79+
guard failure == .none, let element = window as? XCUIElement else {
80+
return RunnerAppScreenCaptureFailure(failure == .none ? .unresolvedWindow : failure)
81+
}
82+
litPanelState.surface = LitPanelSurface(key: key, displayID: litPanel.displayID, window: element, frame: frame)
83+
return nil
84+
}
85+
86+
func requireRequestLitPanelSurface(app: XCUIApplication) throws -> LitPanelSurface {
87+
if let refusal = bookRequestLitPanelSurface(app: app) { throw refusal }
88+
guard let surface = litPanelState.surface else {
89+
throw RunnerAppScreenCaptureFailure.unresolvedScreen
90+
}
91+
return surface
92+
}
93+
94+
/// The roots of every element read: the app, or the windows proven on the lit display. A walk that
95+
/// cannot read the list, or that records an issue (#2995), throws `APP_SCREEN_UNRESOLVED`.
96+
func elementSearchRoots(root: XCUIElement, litDisplayID: UInt?) throws -> [XCUIElement] {
97+
guard let litDisplayID else { return [root] }
98+
var windows: [XCUIElement]?
99+
let walk = holdingRecordedIssues {
100+
windows = RunnerCopyLitPanelWindows(root, litDisplayID)?.compactMap { $0 as? XCUIElement }
101+
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
102+
if litPanelWindowWalkIssueForTesting {
103+
record(XCTIssue(type: .assertionFailure, compactDescription: "Window vanished during the lit-panel walk"))
104+
}
105+
#endif
106+
}
107+
guard walk.issues.isEmpty, walk.exceptionMessage == nil, let windows, !windows.isEmpty else {
108+
for issue in walk.issues {
109+
NSLog("AGENT_DEVICE_RUNNER_LIT_PANEL_WALK_ISSUE=%@", issue.compactDescription)
110+
}
111+
NSLog(
112+
"AGENT_DEVICE_RUNNER_LIT_PANEL_WALK_REFUSED issues=%ld exception=%@ windows=%ld",
113+
walk.issues.count, walk.exceptionMessage ?? "none", windows?.count ?? -1
114+
)
115+
throw RunnerAppScreenCaptureFailure.unresolvedScreen
116+
}
117+
return windows
118+
}
119+
120+
func elementQueries(app: XCUIApplication, matching type: XCUIElement.ElementType) throws -> [XCUIElementQuery] {
121+
try elementQueries(root: app, matching: type, litDisplayID: requestLitPanel?.displayID)
122+
}
123+
124+
func elementQueries(
125+
root: XCUIElement,
126+
matching type: XCUIElement.ElementType,
127+
litDisplayID: UInt?
128+
) throws -> [XCUIElementQuery] {
129+
try elementSearchRoots(root: root, litDisplayID: litDisplayID).map { $0.descendants(matching: type) }
130+
}
131+
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerScrollViewportPolicy.swift‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ extension RunnerTests {
190190
func resolvedScrollViewport(
191191
app: XCUIApplication,
192192
context: SynthesizedCoordinateContext
193-
) -> RunnerScrollViewport {
193+
) throws -> RunnerScrollViewport {
194194
#if os(iOS)
195195
// Every scroll reports its decision, including the two ways it avoids reading the keyboard at
196196
// all: unavailable accessibility health, and a probe that finds no keyboard.
@@ -223,7 +223,7 @@ extension RunnerTests {
223223
}
224224
return ScrollViewportPolicy.frames(referenceFrame: context.referenceFrame, clip: clip)
225225
#else
226-
let fallbackFrame = resolvedTouchReferenceFrame(app: app, appFrame: app.frame)
226+
let fallbackFrame = try resolvedTouchReferenceFrame(app: app, appFrame: app.frame)
227227
return ScrollViewportPolicy.frames(referenceFrame: fallbackFrame, clip: .unobstructed)
228228
#endif
229229
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+AXSnapshotFallback.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@ extension RunnerTests {
290290
timeout: 1,
291291
timeoutError: snapshotMainThreadTimeoutError("reading private AX viewport")
292292
) {
293-
self.safeSnapshotViewport(app: app, readingOrientation: true)
293+
try self.safeSnapshotViewport(app: app, readingOrientation: true)
294294
}
295295
if case .missing = reported {
296296
return fallback

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Alert.swift‎

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ extension RunnerTests {
9595
}
9696

9797
@MainActor
98-
func handleAlert(_ alert: RunnerAlert, action: String, deadline: Date) -> Response {
98+
func handleAlert(_ alert: RunnerAlert, action: String, deadline: Date) throws -> Response {
9999
if action == "accept" || action == "dismiss" {
100100
guard let button = chooseAlertButton(alert, action: action) else {
101101
// The choice refused before any activation, so this discloses dispatched: no like the
@@ -141,7 +141,7 @@ extension RunnerTests {
141141
// (#2546). The post-tap settle stays, because the verification below reads the alert this tap
142142
// replaces; an alert that dismisses and presents an identical replacement passes through a
143143
// window with no alert, and a first read landing there reports a dismissal nothing proved.
144-
guard let outcome = activateAlertButton(alert, button: button, action: action, frame: buttonFrame, deadline: deadline) else {
144+
guard let outcome = try activateAlertButton(alert, button: button, action: action, frame: buttonFrame, deadline: deadline) else {
145145
return alertVerificationResponse(.timedOut, action: action, activated: false)
146146
}
147147
if let response = unsupportedResponse(for: outcome) {
@@ -190,8 +190,9 @@ extension RunnerTests {
190190
action: String,
191191
frame: CGRect,
192192
deadline: Date
193-
) -> RunnerInteractionOutcome? {
193+
) throws -> RunnerInteractionOutcome? {
194194
var outcome: RunnerInteractionOutcome?
195+
var activationError: Error?
195196
withUIInterruptionHandlingDisabledIfSupported(alert.ownerApp) {
196197
withBoundedInteractionIdleTimeoutIfSupported(alert.ownerApp, waits: .preEventSkipped) {
197198
guard Date() < deadline else { return }
@@ -223,13 +224,20 @@ extension RunnerTests {
223224
}
224225
hostContext = context
225226
}
226-
switch synthesizedTapAt(
227-
app: alert.ownerApp,
228-
x: frame.midX,
229-
y: frame.midY,
230-
context: hostContext,
231-
deadline: deadline
232-
) {
227+
let attempt: RunnerInteractionOutcome?
228+
do {
229+
attempt = try synthesizedTapAt(
230+
app: alert.ownerApp,
231+
x: frame.midX,
232+
y: frame.midY,
233+
context: hostContext,
234+
deadline: deadline
235+
)
236+
} catch {
237+
activationError = error
238+
return
239+
}
240+
switch attempt {
233241
case .performed?:
234242
typealias WaitForQuiescence = @convention(c) (NSObject, Selector, Bool) -> Void
235243
let waitForQuiescence = unsafeBitCast(
@@ -247,15 +255,15 @@ extension RunnerTests {
247255
break
248256
}
249257
#else
250-
outcome = activateElement(
251-
app: alert.ownerApp,
252-
element: button,
253-
action: "alert \(action)",
254-
resolvedFrame: frame
255-
)
258+
do {
259+
outcome = try activateElement(app: alert.ownerApp, element: button, action: "alert \(action)", resolvedFrame: frame)
260+
} catch {
261+
activationError = error
262+
}
256263
#endif
257264
}
258265
}
266+
if let activationError { throw activationError }
259267
return outcome
260268
}
261269

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandDispatch.swift‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ extension RunnerTests {
77
func executeAccepted(command: Command) throws -> Response {
88
commandJournal.start(command: command)
99
pendingTargetActivation = nil
10+
let panelRequestID = setRequestLitPanel(from: command)
11+
defer { endRequestLitPanel(panelRequestID) }
1012
do {
1113
let response = try executeDispatched(command: command)
1214
commandJournal.finish(command: command, response: response)
@@ -432,9 +434,11 @@ extension RunnerTests {
432434
outputPath: resolvedOutPath,
433435
fps: command.fps.map { Int32($0) }
434436
)
437+
// The pump outlives this request, so the clip keeps the panel its `recordStart` named.
438+
let clipLitPanel = requestLitPanel
435439
try startRecording(recorder) { [weak self] in
436440
guard let self else { return .failure(.unresolvedScreen) }
437-
return self.captureRunnerFrameResult(app: activeApp)
441+
return self.captureRunnerFrameResult(app: activeApp, litPanel: clipLitPanel)
438442
}
439443
activeRecording = recorder
440444
return Response(ok: true, data: DataPayload(message: "recording started"))

0 commit comments

Comments
 (0)