Skip to content

Commit 057dfb7

Browse files
authored
fix(ios): capture the lit panel on foldable Apple devices (#2703)
* fix(ios): capture the lit panel on foldable Apple devices simctl io screenshot picks the highest screen ID when no display is named, which on a closed iPhone Duo is the dark inner panel: the capture exits 0 with an all-black PNG. Resolve the CoreDevice display table per capture, name the panel the device actually lights, and normalize density with that panel's own point scale. Panel power comes from backlightState because CoreDevice omits the active key on real payloads, and a multi-panel device always names a display so an ambiguous read degrades to a diagnostic instead of a black screenshot. * fix(ios): record the lit panel of a foldable Apple device simctl io recordVideo shares screenshot's implicit display default, so a recording of a closed foldable is also black. The Apple package now builds the recordVideo argv and reaches the display inventory through a function-scoped import, which keeps the simctl facade's eager closure at its budget and puts the panel decision where the display model lives. * docs(ios): document foldable panels, pose limits, and Duo verification Adds ADR 0025, the `help foldable` topic with its benchmark case, and the verification traps that cost time on a Duo: the pinned 27.1 toolchain, the dark panel that captures black while exiting 0, and the UIScene lifecycle requirement that makes a legacy app trap at launch. Pose stays operator-controlled: no official host API sets it, so no command pretends. * chore(gates): suppress the simctl facade symbols fallow cannot reach The display symbols leave the facade through a function-scoped import read by a dynamic platform-runtime consumer, so fallow cannot connect the member read to the re-export. Scoped to the two facade symbols it names. * fix(ios): route the display probe through the owning devicectl helper `queryAppleDisplayInventory` copied the devicectl JSON protocol — temp path, `--json-output`, `runXcrun` with allowFailure, payload read, cleanup — and had already dropped the hint mapping the owning helper applies. Export `runIosDevicectlJsonRequest` as that single protocol implementation with a timeout and signal, and let the throwing command and the probe both build on it. Classify the probe by what actually happened: an exec kill stays a timeout via `isCommandTimeoutError`, `unreadable-json` now means devicectl answered with a payload that did not parse, and only those two reasons claim the toolchain lacks the subcommand. The old mapping labelled an AppError — i.e. the timeout — as an unreadable payload, which is inverted. Drop the panel labels nothing read. `outer`/`inner` and the derived pose had no consumer outside the module and its tests, while `primary` plus panel power is what every decision already used; the geometry conflict still reports `apple_display_primary_geometry_conflict`. Also stub `resolveCaptureDisplay` in the screenshot suite so it stops paying for a real CoreDevice probe, assert the caller signal actually reaches `runXcrun` instead of asserting it was never aborted, and drop the dynamic import of a module the facade already imports statically. * docs(ios): record the open-pose Duo evidence and the overlay exporter defect The operator opened the Duo mid-verification, so the open pose is now evidence rather than a gap: the capture names `LCD-1` at 951x669, taps resolve inside that surface, typed text reads back, and a pre-fold ref is refused instead of replayed at the new point size. What stays open is the quarter turn and the runner fallback, both restated as gaps. `recordVideo` gets the per-panel table that proves `--display` is honored (2006x2852 luma 241.42 for the lit panel against 1398x2034 luma 0.00 for the dark one), plus the separate finding that the touch-overlay exporter returns a 480x336 black video for the inner panel's rot90 track — reproducible by feeding a raw simctl capture straight into the untouched `recording-overlay.swift`, and fine on a non-rotated iPhone 17 export. Also drop the pose and role wording from the decision table, since the inventory no longer carries either.
1 parent 1f79374 commit 057dfb7

18 files changed

Lines changed: 1507 additions & 54 deletions

‎.fallowrc.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,9 +147,9 @@
147147
"exports": ["buildSimctlArgsForDevice"]
148148
},
149149
{
150-
"comment": "The device-aware simctl argument builder is a named package entrypoint reached through dynamic platform-runtime imports. Fallow cannot connect those member reads to the re-export; keep the suppression limited to this one facade symbol.",
150+
"comment": "The simctl façade symbols are named package entrypoints reached through dynamic platform-runtime imports. Fallow cannot connect those member reads to the re-exports; keep the suppression limited to these facade symbols.",
151151
"file": "packages/platform-apple/src/simctl-facade.ts",
152-
"exports": ["buildSimctlArgsForDevice"]
152+
"exports": ["buildSimctlArgsForDevice", "buildAppleSimulatorRecordVideoArgs"]
153153
},
154154
{
155155
"comment": "Apple runner mechanics are reached through the named runner operations façade. Fallow cannot connect workspace package exports to these source exports; keep this list limited to the façade and its cleanup composition.",
Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
# ADR 0025: Foldable Apple Panels — Capture the Lit Panel
2+
3+
## Status
4+
5+
Accepted (2026-09-20). Covers iPhone Duo (iOS 27.1, `iPhone19,4`) and any Apple device that
6+
reports more than one integrated CoreDevice display.
7+
8+
An iPhone Duo carries two integrated panels — Apple's **outer display** and **inner display** —
9+
and lights one of them at a time. Which one is lit is the device pose. Two independent facts
10+
follow, and this ADR keeps them separate: **which panel a capture must name**, and **what pose the
11+
device is in**. The first is answered by an official host API; the second only by inference.
12+
13+
## Rules at a glance
14+
15+
| Situation | Behavior |
16+
| --- | --- |
17+
| Any iOS simulator capture | Resolve the CoreDevice display table first; capture the panel the system currently lights |
18+
| Device has more than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`; never rely on the implicit default |
19+
| Panel power is ambiguous (zero or several lit panels) | Still name a panel — the `primary` one — emit `apple_display_capture_ambiguous`, and report the pose as `unknown` |
20+
| Device has one integrated panel | Keep the pre-panel behavior exactly: no display flag, no pose, unchanged scale probe |
21+
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` may be a different panel |
22+
| Pose must be reported | Report `closed`, `fully-open`, or `unknown` from panel power, and never narrower |
23+
| A pose change is requested | Refuse in guidance, not by inventing a command: no official host API sets pose |
24+
| An external display is attached | It is not a panel: it never makes the device multi-screen and never produces a pose |
25+
| CoreDevice cannot answer | Return an unresolved inventory and keep the single-panel capture path; a missing host feature is not a capture failure |
26+
27+
## Contracts
28+
29+
`packages/platform-apple/src/core/display-inventory.ts` owns the panel model and is the only
30+
reader of CoreDevice display info.
31+
32+
- **Authority.** `xcrun devicectl device info displays --json-output` (`jsonVersion` 5) is the only
33+
screen authority. It reports per panel: `name` (which `simctl io --display` also accepts),
34+
`displayId`, `nativeSize`, `pointScale`, `currentOrientation`, `type`, `primary`, `active`, and
35+
`backlightState`. It works for simulators (`reality: simulated`) and physical devices alike.
36+
- **Panel power.** `backlightState` is the authority and `active` is only a fallback, because
37+
`active` is genuinely absent from real payloads: a single-panel iPhone 17 reports
38+
`backlightState: activeOn` and no `active` key at all, so a parser keyed on `active` would call
39+
that panel dark. The `BacklightState` union is `activeOn`, `inactiveOn`, `activeDimmed`, `off`,
40+
`unknown`; the first three are lit. An unrecognized future case is `unknown`, not `dark`, because
41+
guessing dark would aim a capture at a panel that may be showing nothing.
42+
- **Panel identity.** A device is multi-screen when it reports more than one `type: integrated`
43+
panel. CoreDevice marks one panel `primary`; that panel is the outer display, because it is the
44+
panel that stays lit while the device is closed. `primary` and the lit count are the whole model:
45+
the inventory labels panels `outer`/`inner` nowhere, because no caller consumes a label — every
46+
decision is made from `primary` plus panel power, and a name derived from array order or from
47+
geometry would be a second source of truth ready to invert silently. When the primary panel is
48+
also the largest, the `primary` attribute still decides and the conflict is emitted as
49+
`apple_display_primary_geometry_conflict` rather than silently repaired by a geometry rule.
50+
- **Capture target.** `resolveAppleCaptureDisplay` returns a panel for every multi-panel device and
51+
`undefined` only for a single-panel device or an unresolved inventory. When panel power identifies
52+
exactly one lit panel that panel is named; otherwise the `primary` panel is named anyway,
53+
`ambiguous` is set, and `apple_display_capture_ambiguous` is emitted. A multi-panel device never
54+
receives a display-less capture, because the implicit default is precisely the black-and-exit-0
55+
failure being fixed; naming a possibly-dark panel is a visible, diagnosable miss instead.
56+
- **No pose field.** Panel power cannot separate Apple's `UIHinge.Status.fullyOpen` from
57+
`.partiallyOpen` — both leave the inner panel lit and the outer panel dark — so a derived pose
58+
could only ever mean "not closed" while looking like a fact. The inventory therefore reports what
59+
it measured (`primary`, panel power, geometry, orientation) and leaves pose to the app under test,
60+
which can read `UIHinge.status` exactly. A caller that needs the pose asks the operator and
61+
re-snapshots.
62+
- **No cache.** The probe runs per capture. Which panel is lit is precisely what an operator
63+
changes by folding or opening the device, so a cached inventory would resume capturing the dark
64+
panel — the exact failure being fixed. The probe runs under
65+
`IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` (5s), deliberately below the 20s screenshot deadline it
66+
precedes, so a wedged CoreDevice cannot spend the capture's own budget; measured real cost is
67+
0.16–0.26s against 4.93s for the screenshot.
68+
69+
## Why the default capture had to stop being implicit
70+
71+
`simctl io screenshot` without `--display` picks the **highest** screen ID, not the primary panel.
72+
On a closed Duo that is the inner panel, which is dark. The command exits 0 and writes a valid PNG
73+
whose content is black, so every downstream consumer trusts a capture of nothing:
74+
75+
| Capture path | Before | After |
76+
| --- | --- | --- |
77+
| `screenshot` on a closed Duo | 669x951 @1x, mean luma 0.09 (black) | 466x678 @1x, mean luma 65.8 (the lit outer panel) |
78+
| `screenshot` on an open Duo | 951x669 @1x, mean luma 241 (correct by luck: the highest screen ID is the lit inner panel) | 951x669 @1x naming `LCD-1` explicitly |
79+
| `screenshot` on iPhone 17 | 402x874 @1x, mean luma 104.9 | unchanged |
80+
81+
This is the failure behind reports that the tool "picks the wrong screen" on iPhone Duo. It was
82+
never a selection problem: only the lit panel is capturable, so naming the lit panel is the whole
83+
fix, and no screen-selection flag is warranted.
84+
85+
`SIMULATOR_MAINSCREEN_SCALE` is likewise fixed to one panel while the captured panel can be the
86+
other, so density normalization now takes `pointScale` from the panel that was captured.
87+
88+
## Pose is derived, and official control does not exist
89+
90+
Apple ships fold state as an **app-side, read-only** API: `UIHinge.status`
91+
(`.closed`/`.partiallyOpen`/`.fullyOpen`) observed through `UIHingeInteraction`, and SwiftUI
92+
`DeviceHinge` / `.onHingeChange`, both `ios(27.1)`. There is no `UITraitCollection` trait for it.
93+
94+
No **official** host-side control channel exists. Each candidate was checked against the shipping
95+
toolchain:
96+
97+
| Candidate | Result |
98+
| --- | --- |
99+
| `simctl` subcommands | no `hinge`/`fold`/`pose` surface; no such token in the `simctl` binary or in `CoreSimulator.framework` |
100+
| `simctl io` | `screenshot`/`recordVideo` take `--display`; `screenConfig` sets `power` and `geometry` per screen — neither changes hinge state |
101+
| XCUITest | no hinge token in any `XCUIAutomation` header; `XCUIDeviceButton` is Home/VolumeUp/VolumeDown/Action/Camera only |
102+
| CoreDevice | `com.apple.coredevice.action.streamhingeangle`, `HingeAngleManager`, `DeviceHingeAngleSnapshot` and `HingeAngleStreamConfig` exist in `/Library/Developer/PrivateFrameworks/CoreDeviceUtilities.framework` (and `dtdeviceinfod`), but they **stream** an angle; the Duo reports no such feature, and `devicectl` exposes no CLI for it |
103+
| `devicectl` | `device info displays` reports panel state only |
104+
| `Xcode.app` frameworks | no hinge reference in `Xcode.app/Contents/{SharedFrameworks,Frameworks}`, so Device Hub's own control is not a hinge API shipped inside Xcode |
105+
106+
Two searches came back empty rather than negative, and are recorded as limits rather than refuted
107+
claims:
108+
109+
- The **simulator guest runtime** was not enumerated for a host-reachable control channel. Symbol
110+
archaeology turned up a private SpringBoard service (`SBDisplayToolService`,
111+
`com.apple.springboard.sbdisplay.service`) with `setPrimary:displayUUID:`,
112+
`setBacklightState:displayUUID:` and `replayHingeSamplesWithOptions:path:`, plus
113+
`SBContinuousFoldController` and a `com.apple.springboard.fold` notification. No client, launch
114+
path, or invocation from this host was found, so it is an **unproven private lead** — not adopted,
115+
and not a basis for any claim that pose control is possible.
116+
- Device Hub's control could live in a private framework outside `Xcode.app`. That would not make it
117+
usable: it is still not an API, and the Device Hub device surface exposes zero accessibility
118+
nodes.
119+
120+
Consequences that are now policy: no `fold`/`unfold`/`half-unfold` command, because a command that
121+
cannot do the thing is worse than no command, and because a private per-guest XPC channel is exactly
122+
the kind of undocumented hook that breaks without notice. Guidance in `agent-device help foldable`
123+
tells agents pose is operator-controlled, and a derived `fully-open` verdict is documented to cover
124+
Apple's `fullyOpen` **and** `partiallyOpen` — panel power cannot separate them, so only an in-app
125+
`UIHinge.status` read can.
126+
127+
## Refuted alternatives
128+
129+
- **Pose commands backed by `simctl io screenConfig power`.** Rejected: it does not move
130+
`UIHinge.status`, so the app under test would not behave as folded. It would produce green tests
131+
of a state the device is not in.
132+
- **Pose commands backed by macOS UI automation of Device Hub.** Rejected: the Device Hub device
133+
surface reports zero accessibility nodes, so it is coordinate-only and permission-bound
134+
(Screen Recording), and no Xcode framework exposes the control it would press. Kept as a
135+
documented possibility in help, not as shipped automation.
136+
- **Luma or content heuristics to pick the lit panel.** Rejected: a black screenshot is legitimate
137+
content elsewhere, and the repo already forbids deciding on pixels when a typed fact exists.
138+
CoreDevice reports `active`/`backlightState` directly.
139+
- **A `--screen outer|inner` flag.** Rejected as unnecessary: the inactive panel renders black, so
140+
a non-active selection can only ever capture nothing. Revisit only when a device shows content on
141+
two panels at once.
142+
- **Reading the device type's `capabilities.plist` for the panel table.** Rejected: `devicectl`
143+
reports live panel state, which the plist cannot, and works identically on physical devices.
144+
145+
## Consequences for agents
146+
147+
A pose change moves the app to a different panel with different point size, so refs and
148+
coordinates do not survive it. `agent-device help foldable` states this and states that agents must
149+
report which poses remain unverified rather than assume a pose was set.
150+
151+
`simctl io recordVideo` has the same implicit-display default as `screenshot`, so recording names
152+
the lit panel through the same resolver. On an open Duo the 27.1 toolchain accepts the panel name
153+
and honors it per panel; sampled mean luma over the whole frame:
154+
155+
| `recordVideo` argv | exported size | mean luma |
156+
| --- | --- | --- |
157+
| `--display=LCD-1` (lit inner) | 2006x2852 | 241.42 |
158+
| `--display=LCD` (dark outer) | 1398x2034 | 0.00 |
159+
| no `--display` | 2006x2852 | 241.42 |
160+
161+
`record start`/`record stop` exit 0 in both poses, and with `--hide-touches` the export keeps the
162+
captured geometry (`2006x2852`, mean luma 241.42). Without it, the touch-overlay exporter is broken
163+
for the inner panel's `rot90` track: it returns a `480x336` black video, and feeding an untouched
164+
raw `simctl` capture straight into `recording-overlay.swift` reproduces a `0x0` zero-duration
165+
output. That exporter defect is independent of panel selection — the same pipeline exports a
166+
non-rotated iPhone 17 recording intact (`1206x2622`, mean luma 228.30) — and is tracked separately.
167+
168+
## Verified on a booted Duo
169+
170+
Closed pose: `screenshot` moved from `669x951` luma 0.09 to `466x678` luma 65.8; a tap on Safari's
171+
address field at `(191, 620)` opened the keyboard; and on the `examples/test-app` dev build a
172+
41-node `snapshot -i`, a tap that dismissed the dev-menu sheet at `(345, 301)`, and a Catalog-tab tap
173+
at `(128, 626)` that settled `+19 -15`.
174+
175+
Open pose, after an operator opened the device: the capture names `LCD-1` at `951x669 @1x`,
176+
`snapshot -i` returns Safari's nodes on that surface, `tap @e4` resolves to `(590, 478)` inside it,
177+
and text sent with `type` is found again by `find text` — hit testing and read-back both follow the
178+
lit panel. A ref issued before the fold is refused afterwards as an expired frame rather than
179+
replayed at the new point size, which is the pose-change rule working as designed.
180+
181+
## Accepted evidence gaps
182+
183+
- **Runner screenshot fallback.** `XCUIScreen.main` is hardcoded in the runner's `screenshot`
184+
command and in the synthesized-gesture reference frame. The `simctl` path is the default for iOS
185+
simulators, so the fix above covers the proven failure, but the runner fallback and the
186+
physical-device path are unverified on an open Duo. `XCUIScreen` exposes no geometry, identity, or
187+
`active` flag — only `screens` and `mainScreen` — so choosing the lit panel there cannot be done
188+
officially without pairing screenshots against panel geometry, which is the pixel heuristic this
189+
ADR rejects. Left unresolved rather than guessed.
190+
- **Quarter-turn detection.** Both Duo panels report `currentOrientation: rot90`, and no available
191+
path rotates a foldable, so the orientation half of the inventory is carried but never exercised
192+
against a changed value.
193+
- **Pose control.** No official host API sets the hinge angle, so both poses above depend on an
194+
operator opening or closing the device in Device Hub.
195+

‎docs/adr/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
| [0022 Daemon — Platform Runtime Coupling Audit and Ownership Ratchets](0022-daemon-platform-runtime-coupling.md) | daemon imports of root `platform-runtime-*` modules, the R76 edge classification inventory, R75 session-authority ratchet, entry-to-platform hop routes and roles |
2727
| [0023 End-State Entry-to-Platform Hop Trace](0023-end-state-hop-trace.md) | the hop counting definition, the ordered `press`/Android and dual-arm `snapshot`/iOS chains, per-hop roles, and the deletion test behind the numbers ADR 0019 and ADR 0022 quote |
2828
| [0024 Screen Recording — One Coordinator, Two Facts (Proposed)](0024-screen-recording-provable-signal.md) | `record start`/`stop` lifecycle, the export-exists / recorder-stopped split, backend stop-and-collect contract, recovery through the coordinator, the ADR 0019 §5 amendment and the failed-finish cleanup fix |
29+
| [0025 Foldable Apple Panels — Capture the Lit Panel](0025-foldable-apple-panels.md) | iPhone Duo outer/inner panels, CoreDevice display info as the only screen authority, why `simctl`'s implicit display default captures a black panel, derived (never asserted) pose, and why no pose-control command exists |
2930

3031
ADRs record *why*; the registries and gates they describe are the living source of truth — when
3132
prose and a registry disagree, the registry wins and the ADR needs a follow-up.

‎docs/agents/device-verification.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,27 @@ behavior, and xctestrun build/cache logic stays outside request execution.
5959
- If cleanup cannot be completed, report the remaining session name, state dir, and the
6060
`device status --stale` output as a blocker.
6161

62+
## Foldable Apple devices
63+
64+
Read ADR 0025 before changing capture behavior on a multi-panel device. The iOS 27.1 runtime ships
65+
only with the Xcode that carries it, and `xcode-select` may point at an older one, so pin the
66+
toolchain per command:
67+
`DEVELOPER_DIR=<Xcode-27.1>/Contents/Developer xcrun devicectl device info displays --device <udid>`.
68+
69+
- Panels: that command lists each integrated panel with `backlightState`. Only the lit panel is
70+
capturable — a capture of the dark panel exits 0 and writes an all-black PNG.
71+
- Pose is not scriptable. Ask the operator to fold or open the device in Device Hub, then
72+
re-snapshot; refs and coordinates do not survive the pose change.
73+
- Record a foldable with `record start --hide-touches`. The touch-overlay exporter returns a small
74+
all-black video for the inner panel's `rot90` track, which reads as a capture failure but is not
75+
one; the raw `simctl` capture behind it is correct. See ADR 0025.
76+
- An app must adopt the UIScene lifecycle to launch on iOS 27.1 at all: a legacy
77+
`UIApplicationDelegate` app traps at launch inside
78+
`___UIApplicationEvaluateRuntimeIssueForNoSceneLifecycleAdoption`, which reads like a broken
79+
device but is not one.
80+
- The 27.1 runtime in this beta accepts only the `iPhone Duo` device type, so a second non-foldable
81+
27.1 simulator cannot be created as a control.
82+
6283
## Sandboxed environments
6384

6485
The daemon binds localhost. If the sandbox rejects the listener with `listen EPERM`, rerun with

0 commit comments

Comments
 (0)