|
| 1 | +# ADR 0025: Foldable Apple Panels — Capture the Lit Panel |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +Accepted (2026-09-20). Covers iPhone Duo (iOS 27.1, `iPhone19,4`) and any Apple device that |
| 6 | +reports more than one integrated CoreDevice display. |
| 7 | + |
| 8 | +An iPhone Duo carries two integrated panels — Apple's **outer display** and **inner display** — |
| 9 | +and lights one of them at a time. Which one is lit is the device pose. Two independent facts |
| 10 | +follow, and this ADR keeps them separate: **which panel a capture must name**, and **what pose the |
| 11 | +device is in**. The first is answered by an official host API; the second only by inference. |
| 12 | + |
| 13 | +## Rules at a glance |
| 14 | + |
| 15 | +| Situation | Behavior | |
| 16 | +| --- | --- | |
| 17 | +| Any iOS simulator capture | Resolve the CoreDevice display table first; capture the panel the system currently lights | |
| 18 | +| Device has more than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`; never rely on the implicit default | |
| 19 | +| Panel power is ambiguous (zero or several lit panels) | Still name a panel — the `primary` one — emit `apple_display_capture_ambiguous`, and report the pose as `unknown` | |
| 20 | +| Device has one integrated panel | Keep the pre-panel behavior exactly: no display flag, no pose, unchanged scale probe | |
| 21 | +| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` may be a different panel | |
| 22 | +| Pose must be reported | Report `closed`, `fully-open`, or `unknown` from panel power, and never narrower | |
| 23 | +| A pose change is requested | Refuse in guidance, not by inventing a command: no official host API sets pose | |
| 24 | +| An external display is attached | It is not a panel: it never makes the device multi-screen and never produces a pose | |
| 25 | +| CoreDevice cannot answer | Return an unresolved inventory and keep the single-panel capture path; a missing host feature is not a capture failure | |
| 26 | + |
| 27 | +## Contracts |
| 28 | + |
| 29 | +`packages/platform-apple/src/core/display-inventory.ts` owns the panel model and is the only |
| 30 | +reader of CoreDevice display info. |
| 31 | + |
| 32 | +- **Authority.** `xcrun devicectl device info displays --json-output` (`jsonVersion` 5) is the only |
| 33 | + screen authority. It reports per panel: `name` (which `simctl io --display` also accepts), |
| 34 | + `displayId`, `nativeSize`, `pointScale`, `currentOrientation`, `type`, `primary`, `active`, and |
| 35 | + `backlightState`. It works for simulators (`reality: simulated`) and physical devices alike. |
| 36 | +- **Panel power.** `backlightState` is the authority and `active` is only a fallback, because |
| 37 | + `active` is genuinely absent from real payloads: a single-panel iPhone 17 reports |
| 38 | + `backlightState: activeOn` and no `active` key at all, so a parser keyed on `active` would call |
| 39 | + that panel dark. The `BacklightState` union is `activeOn`, `inactiveOn`, `activeDimmed`, `off`, |
| 40 | + `unknown`; the first three are lit. An unrecognized future case is `unknown`, not `dark`, because |
| 41 | + guessing dark would aim a capture at a panel that may be showing nothing. |
| 42 | +- **Panel identity.** A device is multi-screen when it reports more than one `type: integrated` |
| 43 | + panel. CoreDevice marks one panel `primary`; that panel is the outer display, because it is the |
| 44 | + panel that stays lit while the device is closed. `primary` and the lit count are the whole model: |
| 45 | + the inventory labels panels `outer`/`inner` nowhere, because no caller consumes a label — every |
| 46 | + decision is made from `primary` plus panel power, and a name derived from array order or from |
| 47 | + geometry would be a second source of truth ready to invert silently. When the primary panel is |
| 48 | + also the largest, the `primary` attribute still decides and the conflict is emitted as |
| 49 | + `apple_display_primary_geometry_conflict` rather than silently repaired by a geometry rule. |
| 50 | +- **Capture target.** `resolveAppleCaptureDisplay` returns a panel for every multi-panel device and |
| 51 | + `undefined` only for a single-panel device or an unresolved inventory. When panel power identifies |
| 52 | + exactly one lit panel that panel is named; otherwise the `primary` panel is named anyway, |
| 53 | + `ambiguous` is set, and `apple_display_capture_ambiguous` is emitted. A multi-panel device never |
| 54 | + receives a display-less capture, because the implicit default is precisely the black-and-exit-0 |
| 55 | + failure being fixed; naming a possibly-dark panel is a visible, diagnosable miss instead. |
| 56 | +- **No pose field.** Panel power cannot separate Apple's `UIHinge.Status.fullyOpen` from |
| 57 | + `.partiallyOpen` — both leave the inner panel lit and the outer panel dark — so a derived pose |
| 58 | + could only ever mean "not closed" while looking like a fact. The inventory therefore reports what |
| 59 | + it measured (`primary`, panel power, geometry, orientation) and leaves pose to the app under test, |
| 60 | + which can read `UIHinge.status` exactly. A caller that needs the pose asks the operator and |
| 61 | + re-snapshots. |
| 62 | +- **No cache.** The probe runs per capture. Which panel is lit is precisely what an operator |
| 63 | + changes by folding or opening the device, so a cached inventory would resume capturing the dark |
| 64 | + panel — the exact failure being fixed. The probe runs under |
| 65 | + `IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` (5s), deliberately below the 20s screenshot deadline it |
| 66 | + precedes, so a wedged CoreDevice cannot spend the capture's own budget; measured real cost is |
| 67 | + 0.16–0.26s against 4.93s for the screenshot. |
| 68 | + |
| 69 | +## Why the default capture had to stop being implicit |
| 70 | + |
| 71 | +`simctl io screenshot` without `--display` picks the **highest** screen ID, not the primary panel. |
| 72 | +On a closed Duo that is the inner panel, which is dark. The command exits 0 and writes a valid PNG |
| 73 | +whose content is black, so every downstream consumer trusts a capture of nothing: |
| 74 | + |
| 75 | +| Capture path | Before | After | |
| 76 | +| --- | --- | --- | |
| 77 | +| `screenshot` on a closed Duo | 669x951 @1x, mean luma 0.09 (black) | 466x678 @1x, mean luma 65.8 (the lit outer panel) | |
| 78 | +| `screenshot` on an open Duo | 951x669 @1x, mean luma 241 (correct by luck: the highest screen ID is the lit inner panel) | 951x669 @1x naming `LCD-1` explicitly | |
| 79 | +| `screenshot` on iPhone 17 | 402x874 @1x, mean luma 104.9 | unchanged | |
| 80 | + |
| 81 | +This is the failure behind reports that the tool "picks the wrong screen" on iPhone Duo. It was |
| 82 | +never a selection problem: only the lit panel is capturable, so naming the lit panel is the whole |
| 83 | +fix, and no screen-selection flag is warranted. |
| 84 | + |
| 85 | +`SIMULATOR_MAINSCREEN_SCALE` is likewise fixed to one panel while the captured panel can be the |
| 86 | +other, so density normalization now takes `pointScale` from the panel that was captured. |
| 87 | + |
| 88 | +## Pose is derived, and official control does not exist |
| 89 | + |
| 90 | +Apple ships fold state as an **app-side, read-only** API: `UIHinge.status` |
| 91 | +(`.closed`/`.partiallyOpen`/`.fullyOpen`) observed through `UIHingeInteraction`, and SwiftUI |
| 92 | +`DeviceHinge` / `.onHingeChange`, both `ios(27.1)`. There is no `UITraitCollection` trait for it. |
| 93 | + |
| 94 | +No **official** host-side control channel exists. Each candidate was checked against the shipping |
| 95 | +toolchain: |
| 96 | + |
| 97 | +| Candidate | Result | |
| 98 | +| --- | --- | |
| 99 | +| `simctl` subcommands | no `hinge`/`fold`/`pose` surface; no such token in the `simctl` binary or in `CoreSimulator.framework` | |
| 100 | +| `simctl io` | `screenshot`/`recordVideo` take `--display`; `screenConfig` sets `power` and `geometry` per screen — neither changes hinge state | |
| 101 | +| XCUITest | no hinge token in any `XCUIAutomation` header; `XCUIDeviceButton` is Home/VolumeUp/VolumeDown/Action/Camera only | |
| 102 | +| CoreDevice | `com.apple.coredevice.action.streamhingeangle`, `HingeAngleManager`, `DeviceHingeAngleSnapshot` and `HingeAngleStreamConfig` exist in `/Library/Developer/PrivateFrameworks/CoreDeviceUtilities.framework` (and `dtdeviceinfod`), but they **stream** an angle; the Duo reports no such feature, and `devicectl` exposes no CLI for it | |
| 103 | +| `devicectl` | `device info displays` reports panel state only | |
| 104 | +| `Xcode.app` frameworks | no hinge reference in `Xcode.app/Contents/{SharedFrameworks,Frameworks}`, so Device Hub's own control is not a hinge API shipped inside Xcode | |
| 105 | + |
| 106 | +Two searches came back empty rather than negative, and are recorded as limits rather than refuted |
| 107 | +claims: |
| 108 | + |
| 109 | +- The **simulator guest runtime** was not enumerated for a host-reachable control channel. Symbol |
| 110 | + archaeology turned up a private SpringBoard service (`SBDisplayToolService`, |
| 111 | + `com.apple.springboard.sbdisplay.service`) with `setPrimary:displayUUID:`, |
| 112 | + `setBacklightState:displayUUID:` and `replayHingeSamplesWithOptions:path:`, plus |
| 113 | + `SBContinuousFoldController` and a `com.apple.springboard.fold` notification. No client, launch |
| 114 | + path, or invocation from this host was found, so it is an **unproven private lead** — not adopted, |
| 115 | + and not a basis for any claim that pose control is possible. |
| 116 | +- Device Hub's control could live in a private framework outside `Xcode.app`. That would not make it |
| 117 | + usable: it is still not an API, and the Device Hub device surface exposes zero accessibility |
| 118 | + nodes. |
| 119 | + |
| 120 | +Consequences that are now policy: no `fold`/`unfold`/`half-unfold` command, because a command that |
| 121 | +cannot do the thing is worse than no command, and because a private per-guest XPC channel is exactly |
| 122 | +the kind of undocumented hook that breaks without notice. Guidance in `agent-device help foldable` |
| 123 | +tells agents pose is operator-controlled, and a derived `fully-open` verdict is documented to cover |
| 124 | +Apple's `fullyOpen` **and** `partiallyOpen` — panel power cannot separate them, so only an in-app |
| 125 | +`UIHinge.status` read can. |
| 126 | + |
| 127 | +## Refuted alternatives |
| 128 | + |
| 129 | +- **Pose commands backed by `simctl io screenConfig power`.** Rejected: it does not move |
| 130 | + `UIHinge.status`, so the app under test would not behave as folded. It would produce green tests |
| 131 | + of a state the device is not in. |
| 132 | +- **Pose commands backed by macOS UI automation of Device Hub.** Rejected: the Device Hub device |
| 133 | + surface reports zero accessibility nodes, so it is coordinate-only and permission-bound |
| 134 | + (Screen Recording), and no Xcode framework exposes the control it would press. Kept as a |
| 135 | + documented possibility in help, not as shipped automation. |
| 136 | +- **Luma or content heuristics to pick the lit panel.** Rejected: a black screenshot is legitimate |
| 137 | + content elsewhere, and the repo already forbids deciding on pixels when a typed fact exists. |
| 138 | + CoreDevice reports `active`/`backlightState` directly. |
| 139 | +- **A `--screen outer|inner` flag.** Rejected as unnecessary: the inactive panel renders black, so |
| 140 | + a non-active selection can only ever capture nothing. Revisit only when a device shows content on |
| 141 | + two panels at once. |
| 142 | +- **Reading the device type's `capabilities.plist` for the panel table.** Rejected: `devicectl` |
| 143 | + reports live panel state, which the plist cannot, and works identically on physical devices. |
| 144 | + |
| 145 | +## Consequences for agents |
| 146 | + |
| 147 | +A pose change moves the app to a different panel with different point size, so refs and |
| 148 | +coordinates do not survive it. `agent-device help foldable` states this and states that agents must |
| 149 | +report which poses remain unverified rather than assume a pose was set. |
| 150 | + |
| 151 | +`simctl io recordVideo` has the same implicit-display default as `screenshot`, so recording names |
| 152 | +the lit panel through the same resolver. On an open Duo the 27.1 toolchain accepts the panel name |
| 153 | +and honors it per panel; sampled mean luma over the whole frame: |
| 154 | + |
| 155 | +| `recordVideo` argv | exported size | mean luma | |
| 156 | +| --- | --- | --- | |
| 157 | +| `--display=LCD-1` (lit inner) | 2006x2852 | 241.42 | |
| 158 | +| `--display=LCD` (dark outer) | 1398x2034 | 0.00 | |
| 159 | +| no `--display` | 2006x2852 | 241.42 | |
| 160 | + |
| 161 | +`record start`/`record stop` exit 0 in both poses, and with `--hide-touches` the export keeps the |
| 162 | +captured geometry (`2006x2852`, mean luma 241.42). Without it, the touch-overlay exporter is broken |
| 163 | +for the inner panel's `rot90` track: it returns a `480x336` black video, and feeding an untouched |
| 164 | +raw `simctl` capture straight into `recording-overlay.swift` reproduces a `0x0` zero-duration |
| 165 | +output. That exporter defect is independent of panel selection — the same pipeline exports a |
| 166 | +non-rotated iPhone 17 recording intact (`1206x2622`, mean luma 228.30) — and is tracked separately. |
| 167 | + |
| 168 | +## Verified on a booted Duo |
| 169 | + |
| 170 | +Closed pose: `screenshot` moved from `669x951` luma 0.09 to `466x678` luma 65.8; a tap on Safari's |
| 171 | +address field at `(191, 620)` opened the keyboard; and on the `examples/test-app` dev build a |
| 172 | +41-node `snapshot -i`, a tap that dismissed the dev-menu sheet at `(345, 301)`, and a Catalog-tab tap |
| 173 | +at `(128, 626)` that settled `+19 -15`. |
| 174 | + |
| 175 | +Open pose, after an operator opened the device: the capture names `LCD-1` at `951x669 @1x`, |
| 176 | +`snapshot -i` returns Safari's nodes on that surface, `tap @e4` resolves to `(590, 478)` inside it, |
| 177 | +and text sent with `type` is found again by `find text` — hit testing and read-back both follow the |
| 178 | +lit panel. A ref issued before the fold is refused afterwards as an expired frame rather than |
| 179 | +replayed at the new point size, which is the pose-change rule working as designed. |
| 180 | + |
| 181 | +## Accepted evidence gaps |
| 182 | + |
| 183 | +- **Runner screenshot fallback.** `XCUIScreen.main` is hardcoded in the runner's `screenshot` |
| 184 | + command and in the synthesized-gesture reference frame. The `simctl` path is the default for iOS |
| 185 | + simulators, so the fix above covers the proven failure, but the runner fallback and the |
| 186 | + physical-device path are unverified on an open Duo. `XCUIScreen` exposes no geometry, identity, or |
| 187 | + `active` flag — only `screens` and `mainScreen` — so choosing the lit panel there cannot be done |
| 188 | + officially without pairing screenshots against panel geometry, which is the pixel heuristic this |
| 189 | + ADR rejects. Left unresolved rather than guessed. |
| 190 | +- **Quarter-turn detection.** Both Duo panels report `currentOrientation: rot90`, and no available |
| 191 | + path rotates a foldable, so the orientation half of the inventory is carried but never exercised |
| 192 | + against a changed value. |
| 193 | +- **Pose control.** No official host API sets the hinge angle, so both poses above depend on an |
| 194 | + operator opening or closing the device in Device Hub. |
| 195 | + |
0 commit comments