You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 99c9b58
Browse filesBrowse the repository at this point in the historyBrowse files
docs(adr): record which runner capture surface sees the lit foldable panel (#2739)
* docs(adr): record which runner capture surface sees the lit foldable panel
#2727 measured every `XCUIScreenshotProviding` surface a runner can reach
against the `simctl` capture of the panel CoreDevice reports lit, closed and
open on an iPhone Duo with a 130-degree corroboration run and an iPhone 17
control. The resolved app window captures the display its window sits on and
needed no pose knowledge; `XCUIScreen.main` and `XCUIApplication.screenshot()`
stay pinned to the outer panel and read a dark one. `device-verification.md`
drops the matching promise about UDID-confirmed selection.
Records the four facts the capture path cannot read off the image — undeclared
identity keys and their resolution ordering, panel-native rotated geometry,
overlay compositing, and XCTest's refusal on an unresolved window — and turns
the runner fallback gap into a migration scope for #2728.
Two adversarial review passes ran against the first draft, whose findings are
folded in: the pose attribution follows the hinge read-back because two
sessions requested as open had settled at 130 degrees; captured sizes are PNG
header dimensions; the identity keys are KVC reads no XCTest header declares;
only the non-macOS screenshot branch was reached; the alert window's black crop
is explained by the dark panel it resolved to; and the unread consumers of
`captureRunnerFrame` are named rather than measured. The pre-rebase commits
8b3d56a and 97fc046 keep that review trail.
* docs(adr): wrap the capture section to the file's line budget
The added paragraphs ran past the hundred-column wrap the rest of the file
keeps. Text is unchanged; every token matches the previous commit.
@@ -19,7 +19,7 @@ device is in**. The first is answered by an official host API; the second only b
19
19
| Device has more than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`; never rely on the implicit default |
20
20
| Panel power is ambiguous (zero or several lit panels) | Still name a panel — the `primary` one — emit `apple_display_capture_ambiguous`, and report the pose as `unknown`|
21
21
| Device has one integrated panel | Keep the pre-panel behavior exactly: no display flag, no pose, unchanged scale probe |
22
-
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main`may be a different panel |
22
+
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main`names the outer panel even while the inner one is lit, which the Runner capture migration below would retire|
23
23
| Pose must be reported | From panel power alone, report `closed`, `fully-open`, or `unknown`, and never narrower; `fold` reports the exact pose because it reads the hinge angle |
24
24
| A pose change is requested |`agent-device fold <closed\|half-open\|open>`: press the pose control in the Device Hub window through macOS accessibility, then read the hinge angle back from CoreDevice until it agrees — for `half-open`, until two consecutive readings agree; refuse the pose if it never does |
25
25
| The hinge reaches `half-open` but keeps moving | Refuse it as `fold-pose-unsettled` with the observed and previous angles: an angle inside the open interval is an observed category, not a pose the hinge holds |
@@ -97,7 +97,7 @@ the read-back is. The pieces, each of which was checked on the shipping 27.1 too
97
97
| Who sets the pose | Device Hub's `CoreDevicePopDeviceKitExtension` (the V68 device view with its `poses` action bar) hands a "vendor defined" orientation-control payload to `CoreDevicePopCoreDeviceExtension`, which sends it through CoreDevice's private HID channel. No CLI, `simctl`, `devicectl`, or XCUITest surface reaches that channel |
98
98
| The public seam | The action bar's pose controls are ordinary `AXButton`s described `Closed`, `Book`, and `Open` in the Device Hub window; the simulated screen inside the same window is an `iOSContentGroup` with the app's own nodes. The earlier finding that the device surface exposes "zero accessibility nodes" was an artifact of System Events, which sees Device Hub with pid 0 because the app is launched through a trampoline; an `AXUIElement` built from the real pid works |
99
99
| Reading the pose |`xcrun devicectl device motion hinge-angle --device <udid>` streams the hinge angle for the Duo simulator (`Range:0-180°`): Closed 0°, Book 130°, Open 180°. The stream does not end when `--session-timeout` elapses, so one read is bounded by devicectl's own `--timeout`, whose smallest accepted value is 5 seconds; the sample it printed before aborting itself is the reading |
100
-
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier``TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so `fold` selects by UDID and only then presses|
100
+
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier``TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so the row is `fold`'s only device handle — but a missing row plus a same-named twin's matching title skips the selection before any press, and the confirmation is by title too (see Accepted evidence gaps)|
101
101
| No window | A simulator booted headlessly leaves Device Hub running with no window. LaunchServices cannot address the trampolined process by bundle id (`open -b`, `NSRunningApplication.activate` do nothing), but a `kAEReopenApplication` event sent to the pid restores the device window, the same event a Dock click sends |
102
102
103
103
The rule this yields: `closed` and `open` are the hinge's two end stops, so one read at the stop is
@@ -173,8 +173,7 @@ claims:
173
173
path, or invocation from this host was found, so it is an **unproven private lead** — not adopted,
174
174
and not a basis for any claim that pose control is possible.
175
175
- Device Hub's control could live in a private framework outside `Xcode.app`. That would not make it
176
-
usable: it is still not an API, and the Device Hub device surface exposes zero accessibility
177
-
nodes.
176
+
usable: it is still not an API.
178
177
179
178
Consequences that are now policy: no private per-guest XPC channel is driven, because an undocumented
180
179
hook is exactly the kind that breaks without notice; the one host control that exists is Device
@@ -255,24 +254,123 @@ rest inside the interval, so requiring a settled pair did not lose the Book pres
255
254
— the hinge is where the pair rule said it was. The run's session was closed and its daemon stopped
256
255
afterwards.
257
256
257
+
## Which XCTest capture surface sees the lit panel
258
+
259
+
#2727 measured every `XCUIScreenshotProviding` surface a runner can reach against the `simctl io
260
+
screenshot --display=<panel>` capture of the panel CoreDevice reports lit, on iPhone Duo (iOS 27.1,
261
+
Xcode 27.1 beta) closed at hinge 0° and open at 180°, with a 130° session on different app content
262
+
as the corroboration run and iPhone 17 (iOS 27.0) as the single-panel control. Poses came from the
263
+
hinge read-back, not from the pose that was requested — the amendment above is why that is the only
264
+
defensible source: the first two inner-panel sessions were booked as open and both read back 130°,
265
+
so they are recorded here as corroboration and the settled 180° session carries the table. A
266
+
different app screen was shown per session, so neither panel could be mistaken for the other by
267
+
content, and the app was left untouched between the oracle and the runner capture, so a luma gap is
268
+
a source difference rather than a content one. Mean luma comes from a 48x48 resample; panel
269
+
identity comes from CoreDevice's `displayId` and pixel agreement from comparing decoded pixels, so
270
+
no claim here rests on luma. Captured dimensions are PNG header dimensions, which is the panel's
271
+
native geometry — the capture's own orientation presents some of them turned.
|`simctl` capture of the lit panel |`1398x2034`, luma 221.86 |`2853x2007`, luma 192.02 | reference |
276
+
|`XCUIScreen.main.screenshot()`|`1398x2034`, luma 221.90 |`1398x2034`, **luma 0.00**| Names the outer panel in every pose, so it is correct only while that panel is lit |
277
+
|`XCUIApplication.screenshot()`|`1398x2034`, luma 221.90 |`2006x2852`, **luma 0.00**| Inner geometry, outer content: follows `app.screen`, which stays pinned to panel `1`|
278
+
|`XCUIScreen.screens[i].screenshot()`| its own panel | its own panel | Correct per object; naming the panel is the caller's job, never a default |
279
+
| Resolved `window.screenshot()`|`1398x2034`, luma 221.90, 0 of 2,843,532 pixels differing from that panel's own screen capture |`2006x2852`, luma 192.03, 142 of 5,721,112 pixels differing by at most one per channel |**Captures the display the window is on in every pose**|
280
+
|`window.screenshot()` with no prior `frame` read | not run | identical to the row above, 0 pixels differing | The element resolves its own snapshot; the frame-first rule belongs to the identity read below, not to the capture |
281
+
|`element.screenshot()`|`566x168` crop of a `188.67x56` pt cell |`168x1061` crop of a `353.67x56` pt cell | Correct; the crop is taken in the panel's native geometry, so it is turned relative to the element's frame |
282
+
283
+
The 130° corroboration session reproduced every verdict on the inner panel, including the black
284
+
`XCUIScreen.main`, the black application capture, and the window capture matching its panel's own
285
+
screen capture to within 187 pixels of 5,721,112 at content luma 236.91. So a window capture is the
286
+
same pixels as the screen capture of the panel its window sits on, and it is the only full-panel
287
+
surface measured that needed no pose knowledge. `element.screenshot()` followed the fold too, but
288
+
returns one element rather than a panel.
289
+
290
+
Four facts the capture path cannot read off the image:
291
+
292
+
-**Identity, and how little of it is declared.**`XCUIScreen.h` declares `screens` and
293
+
`mainScreen` alone, and documents `mainScreen` as "the primary screen of the device", so naming
294
+
the outer panel in every pose is the documented behavior and not a malfunction. `displayID`,
295
+
`scale`, and `bounds` answered KVC on every screen measured while `frame` and `name` raised
296
+
`NSUnknownKeyException`, but none of those keys appears in any XCTest header in this toolchain:
297
+
the identity read is as undocumented as the one #2724 already relies on for gesture routing, and
298
+
#2728 inherits that breakage risk whichever way it goes. It should reuse #2724's resolved-display
299
+
read rather than open a third route to the same private key. The ordering #2724 recorded held
300
+
here: `app.screen` and an unresolved window reported `displayID``1` while the app was visibly on
301
+
panel `3`, and the resolved window reported `3`. `XCUIScreen` carries no panel-power fact and no
302
+
`displayWithID:` lookup, so a panel is chosen by filtering `XCUIScreen.screens` and
303
+
`backlightState` stays CoreDevice's. - **Geometry.** The capture carries the panel's own scale
304
+
(measured `3`, equal to that panel's CoreDevice `pointScale`) in the panel's *native* geometry
305
+
rather than the interface orientation: on the `rot90` inner panel the PNG is `2006x2852` with the
306
+
landscape content rotated inside it and the `UIImage` carrying capture orientation raw value `3`,
307
+
whose display size transposes the panel, where `simctl` exports the same panel `2853x2007`
308
+
upright. The runner's production encoder and `XCUIScreenshot`'s produce 0 differing pixels for
309
+
one capture from `358,402` and `328,181` encoded bytes, so the turn belongs to the capture and
310
+
not to an encoder. Rotating is not enough to compare the two: the runner's canvas is `2006x2852`
311
+
and `simctl`'s `2853x2007`, a pixel apart on each axis, so #2729's normalization owes a rotation
312
+
and a crop. - **Overlays.** A screen capture and a window capture are both display captures — the
313
+
second cropped to the window — so the software keyboard, a SpringBoard-hosted permission alert,
314
+
and the status bar all appear in either. It cuts the other way as well: that alert's own window
315
+
resolved to `displayID``3`, the dark panel in the closed pose, and capturing it yielded a black
316
+
`668x950` crop indistinguishable from any capture of a dark panel. Whether a system window on the
317
+
*lit* panel captures was not measured, so a system surface is verified on the app's panel, not
318
+
through its window. - **Unresolved window.** Asking an element that does not exist for screenshot
319
+
data raises `Element Window (First Match) cannot request screenshot data because it does not
320
+
exist` instead of capturing anything, but that is the element route only: `XCUIAutomation` also
321
+
contains `Could not resolve displayID for snapshot %@; falling back to main display (id %lld)`,
322
+
so an internal snapshot-to-display resolution failure can reach the main display silently. A
323
+
helper therefore resolves the window itself and reports its own typed reason, and does not treat
324
+
that message text as the contract. Defaulting to `XCUIScreen.main` on that path would be this
325
+
ADR's original bug, renamed.
326
+
327
+
On the single-panel control every surface agreed: one screen, `1206x2622`, `rot0`, upright crops, no
328
+
rotation to repair. A window-resolved source changes nothing there except that its identity becomes
329
+
explicit.
330
+
258
331
## Accepted evidence gaps
259
332
260
-
-**Runner screenshot fallback.**`XCUIScreen.main` is hardcoded in the runner's `screenshot`
261
-
command and in the synthesized-gesture reference frame. The `simctl` path is the default for iOS
262
-
simulators, so the fix above covers the proven failure, but the runner fallback and the
263
-
physical-device path are unverified on an open Duo. `XCUIScreen` exposes no geometry, identity, or
264
-
`active` flag — only `screens` and `mainScreen` — so choosing the lit panel there cannot be done
265
-
officially without pairing screenshots against panel geometry, which is the pixel heuristic this
266
-
ADR rejects. Left unresolved rather than guessed.
267
-
-**Quarter-turn detection.** Both Duo panels report `currentOrientation: rot90`, and no available
268
-
path rotates a foldable, so the orientation half of the inventory is carried but never exercised
269
-
against a changed value.
270
-
-**Pose control on a second Device Hub instance.**`fold` drives the first `DeviceHub` process in
271
-
the process table. Two Xcodes each running a Device Hub is not a state this was verified in.
272
-
-**A hinge that settles slowly.** Four reads is twenty seconds of streams, and a Duo that needs
273
-
longer to come to rest inside `half-open` is now refused where the superseded rule would have
274
-
reported a pose. Every Duo run observed for this change settled inside the budget; no simulator
275
-
that needs longer was seen, so the budget stays as it is rather than growing on a hypothesis.
276
-
-**Physical foldables.** Device Hub poses simulators only; the leaf fact refuses a physical device,
277
-
and the hinge stream on one was not exercised.
333
+
-**Runner capture migration.**`XCUIScreen.main` is six call expressions in four files, and the
334
+
iOS-reachable ones capture a dark panel once an unfolded Duo moves the app to the inner one: the
335
+
`screenshot` command's non-macOS branch in `RunnerTests+CommandExecution.swift`; its two `#if
336
+
os(macOS)` siblings are `XCUIScreen.main` captures too and only the branch between them captures
337
+
the app's own root, so the desktop's behavior stays unmeasured here and the desktop also reaches
338
+
the screen through Screen Capture Kit in `AgentDeviceMacOSHelper`. Then `captureRunnerFrame` in
339
+
`RunnerTests+Lifecycle.swift`, which feeds both the recorder and the keyboard stability
340
+
fingerprint in `RunnerTests+Keyboard.swift`; the in-app-back visual-state probe in
341
+
`RunnerTests+Navigation.swift`; and the synthesized-gesture reference frame in
342
+
`RunnerTests+Interaction.swift`, which only the unmerged #2724 replaces. All of them call the
343
+
expression the measurement above ran, so they follow by construction and not by measurement, and
344
+
two fail less visibly than a black screenshot: identical black samples satisfy
345
+
`runnerScreenshotStabilitySettled` on its first interval whatever the keyboard is doing, and an
346
+
unchanged black pair logs `AGENT_DEVICE_RUNNER_IN_APP_BACK_FALLBACK_NO_STATE_CHANGE` and reports
347
+
a back navigation that worked as one that failed. Host-side, `screenshot.ts` passes no scale
348
+
precisely because the runner captures `XCUIScreen.main`, `display-inventory.test.ts` asserts
349
+
that, and `commands.md` tells users the physical capture is built from
350
+
`XCUIScreen.main.screenshot()` frames, so all three move with the runner; #2727's comment carries
351
+
the line-numbered inventory. CoreDevice still names the panel, since `XCUIScreen` has no
352
+
panel-power fact to trade on. Physical devices compile the same source and no physical foldable
353
+
was measured, so that path stays unverified. - **Quarter-turn detection.** Both Duo panels report
354
+
`currentOrientation: rot90`, and no available path rotates a foldable, so the orientation half of
355
+
the inventory is carried but never exercised against a changed value. - **Pose control on a
356
+
second Device Hub instance.**`fold` drives the first `DeviceHub` process in the process table.
357
+
Two Xcodes each running a Device Hub is not a state this was verified in. - **A hinge that
358
+
settles slowly.** Four reads is twenty seconds of streams, and a Duo that needs longer to come to
359
+
rest inside `half-open` is now refused where the superseded rule would have reported a pose.
360
+
Every Duo run observed for this change settled inside the budget; no simulator that needs longer
361
+
was seen, so the budget stays as it is rather than growing on a hypothesis.
362
+
363
+
-**Pose control when two simulators share a name.**`fold` picks the device by the sidebar row's
364
+
`AXIdentifier`, but `selectDevice` in `DeviceHubPose.swift` confirms the switch through
365
+
`windowShows(deviceName:)`, which reads the window title, and it skips the selection altogether
366
+
when no row was found and that title already matches — and two simulators named `iPhone Duo` are
367
+
both titled `iPhone Duo – iOS 27.1`. Observed while pressing the poses for the captures above,
368
+
and consistent with that path, a `fold open` aimed at one Duo pressed the pose control of the
369
+
other, which unfolded while the intended device stayed closed; the `apple_fold_pose_pressed`
370
+
diagnostic reports `selected` and nothing gates the press on it. The hinge read-back is what
371
+
stopped the command reporting a pose it had not achieved, yet the untargeted device had already
372
+
moved. Selecting that device's row before pressing routed every later pose correctly, so the row
373
+
is the only device identity Device Hub offers and an unconfirmed selection leaves the press aimed
374
+
at whatever it happens to display. - **Physical foldables.** Device Hub poses simulators only;
375
+
the leaf fact refuses a physical device, and the hinge stream on one was not exercised.
0 commit comments