Skip to content

Commit 8b3d56a

Browse files
committed
docs(adr): state the capture measurement at the strength of its evidence
Adversarial review of the first pass found five claims stated more strongly than the probes support: the identity keys are undeclared KVC reads rather than API, two macOS-only screenshot branches were never reached by an iOS run, captureRunnerFrame also feeds the keyboard stability fingerprint, the alert window's black crop is explained by the dark panel it resolved to, and pixel agreement was reported in PNG bytes. Also corrects the Device identity row, which promised a UDID-confirmed selection that Device Hub only confirms by title.
1 parent c5b2ccc commit 8b3d56a

1 file changed

Lines changed: 84 additions & 55 deletions

File tree

‎docs/adr/0025-foldable-apple-panels.md‎

Lines changed: 84 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ device is in**. The first is answered by an official host API; the second only b
1818
| Device has more than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`; never rely on the implicit default |
1919
| Panel power is ambiguous (zero or several lit panels) | Still name a panel — the `primary` one — emit `apple_display_capture_ambiguous`, and report the pose as `unknown` |
2020
| Device has one integrated panel | Keep the pre-panel behavior exactly: no display flag, no pose, unchanged scale probe |
21-
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` names the outer panel even while the inner one is lit |
21+
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` names the outer panel even while the inner one is lit, which the Runner capture migration below would retire |
2222
| Pose must be reported | From panel power alone, report `closed`, `fully-open`, or `unknown`, and never narrower; `fold` reports the exact pose because it reads the hinge angle |
2323
| A pose change is requested | `agent-device fold <closed\|half-open\|open>`: press the pose control in the Device Hub window through macOS accessibility, then read the hinge angle back from CoreDevice until it agrees; refuse the pose if it never does |
2424
| An external display is attached | It is not a panel: it never makes the device multi-screen and never produces a pose |
@@ -95,7 +95,7 @@ the read-back is. The pieces, each of which was checked on the shipping 27.1 too
9595
| Who sets the pose | Device Hub's `CoreDevicePopDeviceKitExtension` (the V68 device view with its `poses` action bar) hands a "vendor defined" orientation-control payload to `CoreDevicePopCoreDeviceExtension`, which sends it through CoreDevice's private HID channel. No CLI, `simctl`, `devicectl`, or XCUITest surface reaches that channel |
9696
| The public seam | The action bar's pose controls are ordinary `AXButton`s described `Closed`, `Book`, and `Open` in the Device Hub window; the simulated screen inside the same window is an `iOSContentGroup` with the app's own nodes. The earlier finding that the device surface exposes "zero accessibility nodes" was an artifact of System Events, which sees Device Hub with pid 0 because the app is launched through a trampoline; an `AXUIElement` built from the real pid works |
9797
| Reading the pose | `xcrun devicectl device motion hinge-angle --device <udid>` streams the hinge angle for the Duo simulator (`Range:0-180°`): Closed 0°, Book 130°, Open 180°. The stream does not end when `--session-timeout` elapses, so one read is bounded by devicectl's own `--timeout`, whose smallest accepted value is 5 seconds; the sample it printed before aborting itself is the reading |
98-
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier` `TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so `fold` selects by UDID and only then presses |
98+
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier` `TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so `fold` selects by UDID before pressing — though it confirms that switch by title, which is where twins still defeat it (see Accepted evidence gaps) |
9999
| No window | A simulator booted headlessly leaves Device Hub running with no window. LaunchServices cannot address the trampolined process by bundle id (`open -b`, `NSRunningApplication.activate` do nothing), but a `kAEReopenApplication` event sent to the pid restores the device window, the same event a Dock click sends |
100100

101101
The rule this yields: `closed` and `open` are the hinge's end stops, so one read at the stop is
@@ -223,75 +223,104 @@ replayed at the new point size, which is the pose-change rule working as designe
223223
#2727 measured every `XCUIScreenshotProviding` surface a runner can reach against the
224224
`simctl io screenshot --display=<panel>` capture of the panel CoreDevice reports lit, on iPhone Duo
225225
(iOS 27.1, `iPhone19,4`, Xcode 27.1 beta) in the closed, half-open (130°), and open (180°) poses,
226-
with iPhone 17 (iOS 27.0) as the single-panel control. Mean luma is sampled over the whole frame;
227-
the app was left untouched between the oracle and the runner capture, so a luma gap is a source
228-
difference and not a content difference. A different app screen was shown per pose, so no panel
229-
could be mistaken for the other by content.
226+
with iPhone 17 (iOS 27.0) as the single-panel control. A different app screen was shown per pose, so
227+
neither panel could be mistaken for the other by content, and the app was left untouched between the
228+
oracle and the runner capture, so a luma gap is a source difference rather than a content one. Mean
229+
luma comes from a 48x48 resample; panel identity comes from CoreDevice's `displayId` and pixel
230+
agreement from comparing decoded pixels, so no claim here rests on luma.
230231

231232
| Capture surface | Outer panel lit (closed) | Inner panel lit (half-open, open) | Verdict |
232233
| --- | --- | --- | --- |
233-
| `XCUIScreen.main.screenshot()` | `1398x2034`, luma 221.90 (oracle 221.86) | `2034x1398`, **luma 0.00** (oracle 236.91) | Names the outer panel in every pose, so it is correct only while that panel is lit |
234-
| `XCUIApplication.screenshot()` | luma 221.90 | `2006x2852`, **luma 0.00** | Follows `app.screen`, which stays pinned to the outer panel |
234+
| `XCUIScreen.main.screenshot()` | `1398x2034`, luma 221.90 (oracle 221.86) | `2034x1398`, **luma 0.00** (oracle 236.89) | Names the outer panel in every pose, so it is correct only while that panel is lit |
235+
| `XCUIApplication.screenshot()` | luma 221.90 | `2006x2852` canvas, **luma 0.00** | Follows `app.screen`, which stays pinned to the outer panel |
235236
| `XCUIScreen.screens[i].screenshot()` | its own panel | its own panel | Correct per object; naming the panel is the caller's job, never a default |
236-
| Resolved `window.screenshot()` | `1398x2034`, luma 221.90, 0 bytes differing from that panel's own screen capture | `2006x2852`, luma 236.91 (oracle 236.89) | **Captures the display the window is on in every pose** |
237-
| `element.screenshot()` | cropped to the element | cropped, from the lit panel | Correct; the crop is taken in the panel's native geometry |
238-
239-
The window-resolved capture is the same pixels as the screen capture of the panel the window sits
240-
on: 0 differing bytes closed, and 530 differing bytes of ~23 million with a maximum channel delta of
241-
1 on the open inner panel. It is the only surface measured that needed no pose knowledge.
242-
243-
Identity, geometry, and overlays are three separate facts, and none of them is readable off the
244-
image alone:
245-
246-
- **Identity.** `window.screen.displayID` equals the CoreDevice `displayId` of the lit panel (`1` is
247-
`LCD`, `3` is `LCD-1` here) — once the window snapshot is resolved. Before that read, and for
248-
`app.screen`, it reports `1` while the app is visibly on panel `3`, the same resolution ordering
249-
gesture routing depends on. `XCUIScreen` answers `displayID`, `scale`, and `bounds` and refuses
250-
`frame` and `name`, and no `displayWithID:` lookup exists, so a panel is chosen by filtering
251-
`XCUIScreen.screens`; no panel-power fact exists there, so `backlightState` stays CoreDevice's.
237+
| Resolved `window.screenshot()` | `1398x2034`, luma 221.90, 0 of 2,843,532 pixels differing from that panel's own screen capture | `2006x2852`, luma 236.91, 187 of 5,721,112 pixels differing by at most one per channel | **Captures the display the window is on in every pose** |
238+
| `element.screenshot()` | `566x168` crop of a `188.67x56` pt button | `168x1061` crop of a `353.67x56` pt drag source | Correct; the crop is taken in the panel's native geometry, so it is rotated relative to the element's frame |
239+
240+
A window capture is therefore the same pixels as the screen capture of the panel its window sits on,
241+
and it is the only full-panel surface measured that needed no pose knowledge. `element.screenshot()`
242+
followed the fold too, but returns one element rather than a panel.
243+
244+
Four facts the capture path cannot read off the image:
245+
246+
- **Identity, and how little of it is declared.** `XCUIScreen.h` declares `screens` and `mainScreen`
247+
alone, and documents `mainScreen` as "the primary screen of the device", so naming the outer panel
248+
in every pose is the documented behavior and not a malfunction. `displayID`, `scale`, and `bounds`
249+
answered KVC on every screen measured while `frame` and `name` raised `NSUnknownKeyException`, but
250+
none of those keys appears in any XCTest header in this toolchain: the identity read is as
251+
undocumented as the one #2724 already relies on for gesture routing, and #2728 inherits that
252+
breakage risk whichever way it goes. It should reuse #2724's resolved-display read rather than open a
253+
third route to the same private key. The ordering #2724 recorded held here: `app.screen` and an
254+
unresolved window reported `displayID` `1` while the app was visibly on panel `3`, and the resolved
255+
window reported `3`. `XCUIScreen` carries no panel-power fact and no `displayWithID:` lookup, so a
256+
panel is chosen by filtering `XCUIScreen.screens` and `backlightState` stays CoreDevice's.
252257
- **Geometry.** The capture carries the panel's own scale (measured `3`, equal to that panel's
253-
CoreDevice `pointScale`) in the panel's *native* geometry rather than the interface orientation:
254-
on the `rot90` inner panel the PNG is `2006x2852` with the landscape content rotated inside it,
255-
where `simctl` exports the same panel `2853x2007` upright. `XCUIScreenshot.PNGRepresentation` and
256-
`UIImage.pngData()` agree, so a runner capture is not comparable with a `simctl` capture of the
257-
same panel until one of them is rotated.
258+
CoreDevice `pointScale`) in the panel's *native* geometry rather than the interface orientation: on
259+
the `rot90` inner panel the PNG is `2006x2852` with the landscape content rotated inside it and the
260+
`UIImage` oriented with raw value `3`, where `simctl` exports the same panel `2853x2007` upright.
261+
The runner's production encoder and `XCUIScreenshot`'s produce 0 differing pixels for one capture
262+
from `358,402` and `328,181` encoded bytes, so the rotation belongs to the capture and not to an
263+
encoder, and a runner capture is not comparable with a `simctl` capture of the same panel until one
264+
of them is rotated.
258265
- **Overlays.** A screen capture and a window capture are both display captures — the second cropped
259266
to the window — so the software keyboard, a SpringBoard-hosted permission alert, and the status bar
260-
all appear in either. Capturing the alert's own `com.apple.springboard` window yields a black crop,
261-
so a system surface is verified on the app's panel, not through its window.
262-
- **Unresolved window.** XCTest refuses instead of guessing: asking an element that does not exist
263-
for screenshot data raises `Element <query> cannot request screenshot data because it does not
264-
exist`, with no observable fallback to `XCUIScreen.main`. A helper therefore has to resolve the
265-
window first and report its own failure when it cannot; selecting `XCUIScreen.main` on that path
266-
would be this ADR's original bug, renamed.
267-
268-
On the single-panel control every surface agreed: one screen, `1206x2622`, `rot0`, no rotation to
269-
repair. A window-resolved source changes nothing there except that its identity becomes explicit.
267+
all appear in either. It cuts the other way as well: that alert's own window resolved to
268+
`displayID` `3`, the dark panel in the closed pose, and capturing it yielded a black `668x950` crop
269+
indistinguishable from any capture of a dark panel. Whether a system window on the *lit* panel
270+
captures was not measured, so a system surface is verified on the app's panel, not through its
271+
window.
272+
- **Unresolved window.** Asking an element that does not exist for screenshot data raises
273+
`Element Window (First Match) cannot request screenshot data because it does not exist` instead of
274+
capturing anything, but that is the element route only: `XCUIAutomation` also contains `Could not
275+
resolve displayID for snapshot %@; falling back to main display (id %lld)`, so an internal
276+
snapshot-to-display resolution failure can reach the main display silently. A helper therefore
277+
resolves the window itself and reports its own typed reason, and does not treat that message text as
278+
the contract. Defaulting to `XCUIScreen.main` on that path would be this ADR's original bug, renamed.
279+
280+
On the single-panel control every surface agreed: one screen, `1206x2622`, `rot0`, upright crops, no
281+
rotation to repair. A window-resolved source changes nothing there except that its identity becomes
282+
explicit.
270283

271284
## Accepted evidence gaps
272285

273-
- **Runner capture migration.** `XCUIScreen.main` is hardcoded in the runner's `screenshot` command
274-
(`RunnerTests+CommandExecution.swift`), its recording frame source (`captureRunnerFrame`), its
275-
navigation visual-state fallback, and — until #2724 lands — the synthesized-gesture reference
276-
frame. The measurement above proves the first three capture nothing but black once an unfolded Duo
277-
moves the app to the inner panel, and proves the resolved-window capture does not. `XCUIScreen`
278-
does carry `displayID`, `scale`, and `bounds`, so the migration needs no pixel heuristic; it cannot
279-
read panel power, so CoreDevice still names the panel and the resolved window supplies the capture.
280-
`simctl` remains the default for iOS simulators, so this covers the runner fallback and the
281-
physical-device path, which was not re-measured on a physical foldable.
286+
- **Runner capture migration.** `XCUIScreen.main` is six call expressions in four files, and the
287+
iOS-reachable ones capture a dark panel once an unfolded Duo moves the app to the inner one: the
288+
`screenshot` command's non-macOS branch (`RunnerTests+CommandExecution.swift:2075`; `:2068` and
289+
`:2072` sit under `#if os(macOS)`, keep app-targeted capture, and stay unmeasured here; the desktop
290+
also captures through Screen Capture Kit in `AgentDeviceMacOSHelper/main.swift:567`), the frame
291+
source behind both the recorder
292+
(`RunnerTests+CommandExecution.swift:1468`) and the keyboard stability fingerprint
293+
(`RunnerTests+Keyboard.swift:259`) through `captureRunnerFrame`
294+
(`RunnerTests+Lifecycle.swift:45-48`), the in-app-back visual-state probe
295+
(`RunnerTests+Navigation.swift:141`), and the synthesized-gesture reference frame
296+
(`RunnerTests+Interaction.swift:841`), which only the unmerged #2724 replaces. The last three call
297+
the same API the measurement above ran, so they follow by construction and not by measurement, and
298+
two fail less visibly than a black screenshot: identical black samples settle the keyboard wait
299+
immediately, and an unchanged black pair reports a successful back navigation as
300+
`AGENT_DEVICE_RUNNER_IN_APP_BACK_FALLBACK_NO_STATE_CHANGE`. Host-side, `screenshot.ts:179` sends no
301+
scale precisely because the runner captures `XCUIScreen.main`, `display-inventory.test.ts:468`
302+
asserts that, and `commands.md:1122` tells users the physical capture is built from
303+
`XCUIScreen.main.screenshot()` frames, so all three move with the runner. CoreDevice still names the
304+
panel, since `XCUIScreen` has no panel-power fact to trade on. Physical devices compile the same
305+
source and no physical foldable was measured, so that path stays unverified. This measurement also
306+
answers the question #2724 leaves open in the same gap list, so whichever of the two PRs lands
307+
second rebases onto this text.
282308
- **Quarter-turn detection.** Both Duo panels report `currentOrientation: rot90`, and no available
283309
path rotates a foldable, so the orientation half of the inventory is carried but never exercised
284310
against a changed value.
285311
- **Pose control on a second Device Hub instance.** `fold` drives the first `DeviceHub` process in
286312
the process table. Two Xcodes each running a Device Hub is not a state this was verified in.
287-
- **Pose control when two simulators share a name.** A sidebar row is the only device identity
288-
Device Hub exposes, and the window title cannot confirm it: two Duos are both titled
313+
- **Pose control when two simulators share a name.** `fold` picks the device by the sidebar row's
314+
`AXIdentifier`, but it confirms the switch by window title (`DeviceHubPose.swift:217-229`) and skips
315+
the selection outright when no row was found and the title already matches
316+
(`DeviceHubPose.swift:201-204`) — and two simulators named `iPhone Duo` are both titled
289317
`iPhone Duo – iOS 27.1`. Measured while taking the captures above, a `fold open` aimed at one Duo
290-
pressed the pose control of the other, which unfolded while the intended device stayed closed.
291-
The hinge read-back is what stopped the command reporting a pose it had not achieved, yet the
292-
untargeted device had already moved. Selecting that device's row before pressing routed the next
293-
pose correctly, so an unconfirmed selection leaves the press aimed at whatever Device Hub happens
294-
to display.
318+
pressed the pose control of the other, which unfolded while the intended device stayed closed;
319+
`pressed.selected` is logged (`pose.ts:69`) and does not gate the press. The hinge read-back is what
320+
stopped the command reporting a pose it had not achieved, yet the untargeted device had already
321+
moved. Selecting that device's row before pressing routed every later pose correctly, so the row is
322+
the only device identity Device Hub offers and an unconfirmed selection leaves the press aimed at
323+
whatever it happens to display.
295324
- **Physical foldables.** Device Hub poses simulators only; the leaf fact refuses a physical device,
296325
and the hinge stream on one was not exercised.
297326

0 commit comments

Comments
 (0)