Skip to content

Commit 99c9b58

Browse files
authored
docs(adr): record which runner capture surface sees the lit foldable panel (#2739)
* docs(adr): record which runner capture surface sees the lit foldable panel #2727 measured every `XCUIScreenshotProviding` surface a runner can reach against the `simctl` capture of the panel CoreDevice reports lit, closed and open on an iPhone Duo with a 130-degree corroboration run and an iPhone 17 control. The resolved app window captures the display its window sits on and needed no pose knowledge; `XCUIScreen.main` and `XCUIApplication.screenshot()` stay pinned to the outer panel and read a dark one. `device-verification.md` drops the matching promise about UDID-confirmed selection. Records the four facts the capture path cannot read off the image — undeclared identity keys and their resolution ordering, panel-native rotated geometry, overlay compositing, and XCTest's refusal on an unresolved window — and turns the runner fallback gap into a migration scope for #2728. Two adversarial review passes ran against the first draft, whose findings are folded in: the pose attribution follows the hinge read-back because two sessions requested as open had settled at 130 degrees; captured sizes are PNG header dimensions; the identity keys are KVC reads no XCTest header declares; only the non-macOS screenshot branch was reached; the alert window's black crop is explained by the dark panel it resolved to; and the unread consumers of `captureRunnerFrame` are named rather than measured. The pre-rebase commits 8b3d56a and 97fc046 keep that review trail. * docs(adr): wrap the capture section to the file's line budget The added paragraphs ran past the hundred-column wrap the rest of the file keeps. Text is unchanged; every token matches the previous commit.
1 parent 916b82d commit 99c9b58

2 files changed

Lines changed: 122 additions & 23 deletions

File tree

‎docs/adr/0025-foldable-apple-panels.md‎

Lines changed: 120 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ device is in**. The first is answered by an official host API; the second only b
1919
| Device has more than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`; never rely on the implicit default |
2020
| Panel power is ambiguous (zero or several lit panels) | Still name a panel — the `primary` one — emit `apple_display_capture_ambiguous`, and report the pose as `unknown` |
2121
| Device has one integrated panel | Keep the pre-panel behavior exactly: no display flag, no pose, unchanged scale probe |
22-
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` may be a different panel |
22+
| Density normalization | Use the captured panel's own `pointScale`; a runner-fallback capture keeps the scale probe because `XCUIScreen.main` names the outer panel even while the inner one is lit, which the Runner capture migration below would retire |
2323
| Pose must be reported | From panel power alone, report `closed`, `fully-open`, or `unknown`, and never narrower; `fold` reports the exact pose because it reads the hinge angle |
2424
| A pose change is requested | `agent-device fold <closed\|half-open\|open>`: press the pose control in the Device Hub window through macOS accessibility, then read the hinge angle back from CoreDevice until it agrees — for `half-open`, until two consecutive readings agree; refuse the pose if it never does |
2525
| The hinge reaches `half-open` but keeps moving | Refuse it as `fold-pose-unsettled` with the observed and previous angles: an angle inside the open interval is an observed category, not a pose the hinge holds |
@@ -97,7 +97,7 @@ the read-back is. The pieces, each of which was checked on the shipping 27.1 too
9797
| Who sets the pose | Device Hub's `CoreDevicePopDeviceKitExtension` (the V68 device view with its `poses` action bar) hands a "vendor defined" orientation-control payload to `CoreDevicePopCoreDeviceExtension`, which sends it through CoreDevice's private HID channel. No CLI, `simctl`, `devicectl`, or XCUITest surface reaches that channel |
9898
| The public seam | The action bar's pose controls are ordinary `AXButton`s described `Closed`, `Book`, and `Open` in the Device Hub window; the simulated screen inside the same window is an `iOSContentGroup` with the app's own nodes. The earlier finding that the device surface exposes "zero accessibility nodes" was an artifact of System Events, which sees Device Hub with pid 0 because the app is launched through a trampoline; an `AXUIElement` built from the real pid works |
9999
| Reading the pose | `xcrun devicectl device motion hinge-angle --device <udid>` streams the hinge angle for the Duo simulator (`Range:0-180°`): Closed 0°, Book 130°, Open 180°. The stream does not end when `--session-timeout` elapses, so one read is bounded by devicectl's own `--timeout`, whose smallest accepted value is 5 seconds; the sample it printed before aborting itself is the reading |
100-
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier` `TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so `fold` selects by UDID and only then presses |
100+
| Device identity | Device Hub titles the window `<name> – iOS 27.1`, which two simulators sharing a name cannot distinguish. Its sidebar rows carry `AXIdentifier` `TableRow.Device.<UDID>`, and setting `AXSelected` on a row switches the window to that device, so the row is `fold`'s only device handle — but a missing row plus a same-named twin's matching title skips the selection before any press, and the confirmation is by title too (see Accepted evidence gaps) |
101101
| No window | A simulator booted headlessly leaves Device Hub running with no window. LaunchServices cannot address the trampolined process by bundle id (`open -b`, `NSRunningApplication.activate` do nothing), but a `kAEReopenApplication` event sent to the pid restores the device window, the same event a Dock click sends |
102102

103103
The rule this yields: `closed` and `open` are the hinge's two end stops, so one read at the stop is
@@ -173,8 +173,7 @@ claims:
173173
path, or invocation from this host was found, so it is an **unproven private lead** — not adopted,
174174
and not a basis for any claim that pose control is possible.
175175
- Device Hub's control could live in a private framework outside `Xcode.app`. That would not make it
176-
usable: it is still not an API, and the Device Hub device surface exposes zero accessibility
177-
nodes.
176+
usable: it is still not an API.
178177

179178
Consequences that are now policy: no private per-guest XPC channel is driven, because an undocumented
180179
hook is exactly the kind that breaks without notice; the one host control that exists is Device
@@ -255,24 +254,123 @@ rest inside the interval, so requiring a settled pair did not lose the Book pres
255254
— the hinge is where the pair rule said it was. The run's session was closed and its daemon stopped
256255
afterwards.
257256

257+
## Which XCTest capture surface sees the lit panel
258+
259+
#2727 measured every `XCUIScreenshotProviding` surface a runner can reach against the `simctl io
260+
screenshot --display=<panel>` capture of the panel CoreDevice reports lit, on iPhone Duo (iOS 27.1,
261+
Xcode 27.1 beta) closed at hinge 0° and open at 180°, with a 130° session on different app content
262+
as the corroboration run and iPhone 17 (iOS 27.0) as the single-panel control. Poses came from the
263+
hinge read-back, not from the pose that was requested — the amendment above is why that is the only
264+
defensible source: the first two inner-panel sessions were booked as open and both read back 130°,
265+
so they are recorded here as corroboration and the settled 180° session carries the table. A
266+
different app screen was shown per session, so neither panel could be mistaken for the other by
267+
content, and the app was left untouched between the oracle and the runner capture, so a luma gap is
268+
a source difference rather than a content one. Mean luma comes from a 48x48 resample; panel
269+
identity comes from CoreDevice's `displayId` and pixel agreement from comparing decoded pixels, so
270+
no claim here rests on luma. Captured dimensions are PNG header dimensions, which is the panel's
271+
native geometry — the capture's own orientation presents some of them turned.
272+
273+
| Capture surface | Outer panel lit (closed) | Inner panel lit (open, 180°) | Verdict |
274+
| --- | --- | --- | --- |
275+
| `simctl` capture of the lit panel | `1398x2034`, luma 221.86 | `2853x2007`, luma 192.02 | reference |
276+
| `XCUIScreen.main.screenshot()` | `1398x2034`, luma 221.90 | `1398x2034`, **luma 0.00** | Names the outer panel in every pose, so it is correct only while that panel is lit |
277+
| `XCUIApplication.screenshot()` | `1398x2034`, luma 221.90 | `2006x2852`, **luma 0.00** | Inner geometry, outer content: follows `app.screen`, which stays pinned to panel `1` |
278+
| `XCUIScreen.screens[i].screenshot()` | its own panel | its own panel | Correct per object; naming the panel is the caller's job, never a default |
279+
| Resolved `window.screenshot()` | `1398x2034`, luma 221.90, 0 of 2,843,532 pixels differing from that panel's own screen capture | `2006x2852`, luma 192.03, 142 of 5,721,112 pixels differing by at most one per channel | **Captures the display the window is on in every pose** |
280+
| `window.screenshot()` with no prior `frame` read | not run | identical to the row above, 0 pixels differing | The element resolves its own snapshot; the frame-first rule belongs to the identity read below, not to the capture |
281+
| `element.screenshot()` | `566x168` crop of a `188.67x56` pt cell | `168x1061` crop of a `353.67x56` pt cell | Correct; the crop is taken in the panel's native geometry, so it is turned relative to the element's frame |
282+
283+
The 130° corroboration session reproduced every verdict on the inner panel, including the black
284+
`XCUIScreen.main`, the black application capture, and the window capture matching its panel's own
285+
screen capture to within 187 pixels of 5,721,112 at content luma 236.91. So a window capture is the
286+
same pixels as the screen capture of the panel its window sits on, and it is the only full-panel
287+
surface measured that needed no pose knowledge. `element.screenshot()` followed the fold too, but
288+
returns one element rather than a panel.
289+
290+
Four facts the capture path cannot read off the image:
291+
292+
- **Identity, and how little of it is declared.** `XCUIScreen.h` declares `screens` and
293+
`mainScreen` alone, and documents `mainScreen` as "the primary screen of the device", so naming
294+
the outer panel in every pose is the documented behavior and not a malfunction. `displayID`,
295+
`scale`, and `bounds` answered KVC on every screen measured while `frame` and `name` raised
296+
`NSUnknownKeyException`, but none of those keys appears in any XCTest header in this toolchain:
297+
the identity read is as undocumented as the one #2724 already relies on for gesture routing, and
298+
#2728 inherits that breakage risk whichever way it goes. It should reuse #2724's resolved-display
299+
read rather than open a third route to the same private key. The ordering #2724 recorded held
300+
here: `app.screen` and an unresolved window reported `displayID` `1` while the app was visibly on
301+
panel `3`, and the resolved window reported `3`. `XCUIScreen` carries no panel-power fact and no
302+
`displayWithID:` lookup, so a panel is chosen by filtering `XCUIScreen.screens` and
303+
`backlightState` stays CoreDevice's. - **Geometry.** The capture carries the panel's own scale
304+
(measured `3`, equal to that panel's CoreDevice `pointScale`) in the panel's *native* geometry
305+
rather than the interface orientation: on the `rot90` inner panel the PNG is `2006x2852` with the
306+
landscape content rotated inside it and the `UIImage` carrying capture orientation raw value `3`,
307+
whose display size transposes the panel, where `simctl` exports the same panel `2853x2007`
308+
upright. The runner's production encoder and `XCUIScreenshot`'s produce 0 differing pixels for
309+
one capture from `358,402` and `328,181` encoded bytes, so the turn belongs to the capture and
310+
not to an encoder. Rotating is not enough to compare the two: the runner's canvas is `2006x2852`
311+
and `simctl`'s `2853x2007`, a pixel apart on each axis, so #2729's normalization owes a rotation
312+
and a crop. - **Overlays.** A screen capture and a window capture are both display captures — the
313+
second cropped to the window — so the software keyboard, a SpringBoard-hosted permission alert,
314+
and the status bar all appear in either. It cuts the other way as well: that alert's own window
315+
resolved to `displayID` `3`, the dark panel in the closed pose, and capturing it yielded a black
316+
`668x950` crop indistinguishable from any capture of a dark panel. Whether a system window on the
317+
*lit* panel captures was not measured, so a system surface is verified on the app's panel, not
318+
through its window. - **Unresolved window.** Asking an element that does not exist for screenshot
319+
data raises `Element Window (First Match) cannot request screenshot data because it does not
320+
exist` instead of capturing anything, but that is the element route only: `XCUIAutomation` also
321+
contains `Could not resolve displayID for snapshot %@; falling back to main display (id %lld)`,
322+
so an internal snapshot-to-display resolution failure can reach the main display silently. A
323+
helper therefore resolves the window itself and reports its own typed reason, and does not treat
324+
that message text as the contract. Defaulting to `XCUIScreen.main` on that path would be this
325+
ADR's original bug, renamed.
326+
327+
On the single-panel control every surface agreed: one screen, `1206x2622`, `rot0`, upright crops, no
328+
rotation to repair. A window-resolved source changes nothing there except that its identity becomes
329+
explicit.
330+
258331
## Accepted evidence gaps
259332

260-
- **Runner screenshot fallback.** `XCUIScreen.main` is hardcoded in the runner's `screenshot`
261-
command and in the synthesized-gesture reference frame. The `simctl` path is the default for iOS
262-
simulators, so the fix above covers the proven failure, but the runner fallback and the
263-
physical-device path are unverified on an open Duo. `XCUIScreen` exposes no geometry, identity, or
264-
`active` flag — only `screens` and `mainScreen` — so choosing the lit panel there cannot be done
265-
officially without pairing screenshots against panel geometry, which is the pixel heuristic this
266-
ADR rejects. Left unresolved rather than guessed.
267-
- **Quarter-turn detection.** Both Duo panels report `currentOrientation: rot90`, and no available
268-
path rotates a foldable, so the orientation half of the inventory is carried but never exercised
269-
against a changed value.
270-
- **Pose control on a second Device Hub instance.** `fold` drives the first `DeviceHub` process in
271-
the process table. Two Xcodes each running a Device Hub is not a state this was verified in.
272-
- **A hinge that settles slowly.** Four reads is twenty seconds of streams, and a Duo that needs
273-
longer to come to rest inside `half-open` is now refused where the superseded rule would have
274-
reported a pose. Every Duo run observed for this change settled inside the budget; no simulator
275-
that needs longer was seen, so the budget stays as it is rather than growing on a hypothesis.
276-
- **Physical foldables.** Device Hub poses simulators only; the leaf fact refuses a physical device,
277-
and the hinge stream on one was not exercised.
333+
- **Runner capture migration.** `XCUIScreen.main` is six call expressions in four files, and the
334+
iOS-reachable ones capture a dark panel once an unfolded Duo moves the app to the inner one: the
335+
`screenshot` command's non-macOS branch in `RunnerTests+CommandExecution.swift`; its two `#if
336+
os(macOS)` siblings are `XCUIScreen.main` captures too and only the branch between them captures
337+
the app's own root, so the desktop's behavior stays unmeasured here and the desktop also reaches
338+
the screen through Screen Capture Kit in `AgentDeviceMacOSHelper`. Then `captureRunnerFrame` in
339+
`RunnerTests+Lifecycle.swift`, which feeds both the recorder and the keyboard stability
340+
fingerprint in `RunnerTests+Keyboard.swift`; the in-app-back visual-state probe in
341+
`RunnerTests+Navigation.swift`; and the synthesized-gesture reference frame in
342+
`RunnerTests+Interaction.swift`, which only the unmerged #2724 replaces. All of them call the
343+
expression the measurement above ran, so they follow by construction and not by measurement, and
344+
two fail less visibly than a black screenshot: identical black samples satisfy
345+
`runnerScreenshotStabilitySettled` on its first interval whatever the keyboard is doing, and an
346+
unchanged black pair logs `AGENT_DEVICE_RUNNER_IN_APP_BACK_FALLBACK_NO_STATE_CHANGE` and reports
347+
a back navigation that worked as one that failed. Host-side, `screenshot.ts` passes no scale
348+
precisely because the runner captures `XCUIScreen.main`, `display-inventory.test.ts` asserts
349+
that, and `commands.md` tells users the physical capture is built from
350+
`XCUIScreen.main.screenshot()` frames, so all three move with the runner; #2727's comment carries
351+
the line-numbered inventory. CoreDevice still names the panel, since `XCUIScreen` has no
352+
panel-power fact to trade on. Physical devices compile the same source and no physical foldable
353+
was measured, so that path stays unverified. - **Quarter-turn detection.** Both Duo panels report
354+
`currentOrientation: rot90`, and no available path rotates a foldable, so the orientation half of
355+
the inventory is carried but never exercised against a changed value. - **Pose control on a
356+
second Device Hub instance.** `fold` drives the first `DeviceHub` process in the process table.
357+
Two Xcodes each running a Device Hub is not a state this was verified in. - **A hinge that
358+
settles slowly.** Four reads is twenty seconds of streams, and a Duo that needs longer to come to
359+
rest inside `half-open` is now refused where the superseded rule would have reported a pose.
360+
Every Duo run observed for this change settled inside the budget; no simulator that needs longer
361+
was seen, so the budget stays as it is rather than growing on a hypothesis.
362+
363+
- **Pose control when two simulators share a name.** `fold` picks the device by the sidebar row's
364+
`AXIdentifier`, but `selectDevice` in `DeviceHubPose.swift` confirms the switch through
365+
`windowShows(deviceName:)`, which reads the window title, and it skips the selection altogether
366+
when no row was found and that title already matches — and two simulators named `iPhone Duo` are
367+
both titled `iPhone Duo – iOS 27.1`. Observed while pressing the poses for the captures above,
368+
and consistent with that path, a `fold open` aimed at one Duo pressed the pose control of the
369+
other, which unfolded while the intended device stayed closed; the `apple_fold_pose_pressed`
370+
diagnostic reports `selected` and nothing gates the press on it. The hinge read-back is what
371+
stopped the command reporting a pose it had not achieved, yet the untargeted device had already
372+
moved. Selecting that device's row before pressing routed every later pose correctly, so the row
373+
is the only device identity Device Hub offers and an unconfirmed selection leaves the press aimed
374+
at whatever it happens to display. - **Physical foldables.** Device Hub poses simulators only;
375+
the leaf fact refuses a physical device, and the hinge stream on one was not exercised.
278376

‎docs/agents/device-verification.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ toolchain per command:
7575
hinge back through `devicectl device motion hinge-angle`; re-snapshot afterwards, because refs
7676
and coordinates do not survive the pose change. Expect 10-16s per fold. The host needs
7777
Accessibility permission, and the command reopens Device Hub's window and selects the simulator
78-
by UDID itself. To read the angle by hand:
78+
by UDID itself — unless the sidebar row is missing and the window title already matches, which two
79+
same-named simulators defeat (ADR 0025 records it). To read the angle by hand:
7980
`xcrun devicectl device motion hinge-angle --device <udid> --session-timeout 1 --timeout 5`
8081
(the stream never ends on its own; 5 is the smallest timeout devicectl accepts).
8182
- When a recording must show touches, assume it cannot. The touch-overlay exporter loses the track

0 commit comments

Comments
 (0)