investigate(ios): establish a measured screen-capture source for the active window

Part of #2725

Purpose

Before replacing remaining XCUIScreen.main reads, establish which native screen-capture operation actually captures the active app window's display. PR #2724 established that resolving a window frame before reading its screen yields the correct gesture display identity. That does not yet prove screenshot behavior.

Inputs and narrow scope

Read PR #2724, ADR 0025, RunnerXCTestEventBridge.m, RunnerTests+Lifecycle.swift, and the screenshot case in RunnerTests+CommandExecution.swift. This task is a bounded investigation, not a runtime migration.

Use a dedicated session and iPhone Duo/iOS 27.1-capable Xcode. Read the fixture app setup instructions. Test closed, half-open, and open; use visibly different app content to distinguish panels. Also check an ordinary single-panel iPhone. Resolve the window first, inspect its screen identity, and test the screen's screenshot operation. Include orientation, logical window size, pixel size/scale, and whether system overlays are included. Do not infer success from nonempty PNG bytes alone.

Deliverable / stop condition

Post a remote-accessible evidence table on this issue, with tested commit, toolchain, commands, images or artifact links, and an explicit supported/unsupported verdict for each case. Propose one narrow native helper signature that returns the captured image with its actual display/geometry metadata. State how an unresolved window is handled without silently selecting main.

No permanent runtime code changes or new fallback chains in this task. Remove probes and clean up the session. If a required simulator is unavailable, record that blocker; do not substitute mocks for the measurement.

Dependency context: #2724 is open at issue creation; use its measured findings as evidence and check its latest status. Existing input bugs #2711/#2719 stay owned by that work, not this investigation.

Working rules

Read AGENTS.md and its relevant task procedures. Keep this to one task/PR; do not absorb sibling issues. Before implementation, check current main and prerequisite PRs: the review baseline is v0.21.8 and findings may already be fixed. Do not change the active handoff worktree or use its simulator without checking ownership. Follow repository validation and manual-session cleanup requirements. Report an unavailable runtime or device as a blocker, not a passing validation.

Investigation completed — 2026-09-21

Base commit a2f0e2068985f8c02bb69520305b78665d02ce67 plus a temporary native XCTest probe (source pattern below; not a production change). Xcode 27.1 beta, Duo iOS 27.1 build 24A94401; single-panel control iPhone 18 Pro/iOS 27.0. Fixture com.callstack.agentdevicelab, Metro readiness checked and visible Agent Device Tester surface verified. Every successful test below executed, rather than only building/discovering.

Case	Resolved display	App window points	Actual screen raster after upright draw	Scale	Visible result
Duo open, hinge 180	3	951x669	2852x2006	3	Fixture content and status UI
Duo half-open, hinge 130	3	951x669	2852x2006	3	Fixture content and status UI
Duo closed, hinge 0	1	466x678	1398x2034	3	Fixture content and status UI
Single-panel iPhone 18 Pro	1	402x874	1206x2622	3	Fixture content; matches main-screen capture
Duo open, native fixture alert	3	951x669	2852x2006	3	Native alert included over fixture

The resolved-screen PNGs were visually inspected, not accepted merely for having bytes. Open/half-open XCUIScreen.main instead captured a black outer panel. On the closed Duo both sources selected display 1.

Critical encoding result

On open/half-open Duo the returned UIImage has size approximately 950.667x668.667 points, scale 3, and .right orientation (raw 3). Its CGImage and image.pngData() IHDR remain 2006x2852. Drawing that image at its oriented logical size/actual scale produces upright 2852x2006 pixels. This is one pixel smaller per axis than CoreDevice's nominal 2853x2007; preserve actual capture dimensions, do not stretch or infer scale from nominal panel size. The existing raw PNG-size consumer cannot safely assume EXIF orientation has already been applied.

Minimal reproducible probe

Run inside a runner XCTest on each selected pose (activate is test setup, not proposed capture-helper behavior):

let target = XCUIApplication(bundleIdentifier: "com.callstack.agentdevicelab")
target.activate()
let window = target.windows.firstMatch
let frame = window.frame // resolves the window snapshot before asking for its screen
let screen = try XCTUnwrap(window.value(forKey: "screen") as? XCUIScreen)
let displayID = screen.value(forKey: "displayID")
let screenshot = screen.screenshot()
let image = screenshot.image
let format = UIGraphicsImageRendererFormat()
format.scale = image.scale
let upright = UIGraphicsImageRenderer(size: image.size, format: format).image { _ in
  image.draw(in: CGRect(origin: .zero, size: image.size))
}
// Attach screenshot, image.pngData(), and upright.pngData() with keepAlways.
// Log frame, displayID, image.size, image.scale, image.imageOrientation,
// CGImage dimensions and encoded PNG IHDR dimensions; compare XCUIScreen.main.

Build with xcodebuild build-for-testing against AgentDeviceRunner.xcodeproj, scheme AgentDeviceRunner, an explicit iOS Simulator UDID and dedicated derived data; run test-without-building -only-testing:AgentDeviceRunnerUITests/RunnerTests/testDisplayCaptureInvestigation. Extract images using xcresulttool export attachments. The alert probe navigated Settings -> Open automation lab -> Open automation alert, asserted the alert existed, then used the same capture path. An initial alert attempt relying on a deep link failed setup and was not counted as evidence; visible navigation fixed it.

Decision and limits

Use one resolved-window screen-capture helper with upright raster plus actual display/scale/dimensions; #2728 now specifies the production boundary and failure semantics. No guessed screen selection or main-screen fallback. Native in-app alert and status UI inclusion are proven; this does not establish every cross-process system surface or physical-device configuration. Consumer-level fallback/navigation/keyboard tests remain implementation acceptance in #2728.

Local tooling observation

The fold helper initially reported device-hub-window-missing while the user confirmed the window existed on a second host display. After the user moved it to the main display, half-open/closed/open controls succeeded. Record this as a workaround and possible helper limitation, not proof that the window was absent. No source workaround was added. Duo was restored open, the temporary probe was removed, and the single-panel control was returned to shutdown.

Measured: the resolved app window is the capture source that follows the fold

Recorded in ADR 0025 by #2739, which replaces the Runner screenshot fallback gap that asked this question.

Every XCUIScreenshotProviding surface a runner can reach, measured against the simctl io screenshot --display=<panel> capture of the panel CoreDevice reports lit. Supported: window.screenshot() on a resolved window. Unsupported: XCUIScreen.main.screenshot() and XCUIApplication.screenshot() once the inner panel is lit — the first is the runner's iOS screenshot branch, its recording frame source, and its navigation probe.

Erratum (2026-09-21, second review pass). Six statements in the first version of this comment were wrong or misattributed, and the numbers here now match the artifacts: (1) the table formerly headed "Open pose, hinge 180°" carried numbers from a session whose hinge read back 130° — the settled 180° session's values are now the primary column and the 130° runs are labelled corroboration; (2) "half-open reproduced every open-pose row" compared two 130° sessions; (3) the main-screen capture is 1398x2034 in the PNG header, and 2034x1398 was an NSImage orientation artifact; (4) the UIImage orientation case was named without proof — only rawValue: 3 is recorded; (5) RunnerTests+CommandExecution.swift:2068 and :2072 are themselves XCUIScreen.main captures, and only the :2070 branch between them captures the app root; (6) iPhone19,4 appears in no retained artifact and is dropped. See #2739 for the same corrections in ADR 0025.

Corrected after adversarial review: the identity keys are KVC-only and declared in no XCTest header, the #if os(macOS) branches were never reached by these runs, one open-table oracle figure was the runner capture's own luma, and pixel agreement is now stated in pixels rather than PNG bytes. #2739 carries the same corrections.

Test bed

Item	Value
Device	iPhone Duo simulator `9542F4F5-51D1-4F5D-B3C9-02D78A69F0D9`, iOS 27.1 (24A94401)
Control	iPhone 17 simulator `0E0FCBDC-88E2-4CB6-A1DA-E148F77743F4`, iOS 27.0 (24A434)
Toolchain	`DEVELOPER_DIR=/Applications/Xcode-27.1.0-Beta.app/Contents/Developer` for every `xcrun`/`xcodebuild` call
App under test	`examples/test-app` repacked dev build (`com.callstack.agentdevicelab`, JS embedded — no Metro), a different screen per pose so no panel could be confused for the other by content
Commit	`45e4c594a` (v0.21.8) runner sources, plus a test-only probe file in `AgentDeviceRunnerUITests/UnitTests/` compiled under `-D AGENT_DEVICE_RUNNER_UNIT_TESTS`. The probe calls only XCTest capture APIs. `git grep -n XCUIScreen.main 45e4c594a origin/main -- apple` returns the same six call expressions in four files on both commits, and `git diff --stat 45e4c594a origin/main -- apple` is empty, so what was built is the capture path on `origin/main`. Disclosure: the host checkout was shared and later moved to another branch, so the later runs are not pinned to a named commit
Poses	Device Hub pose control pressed through accessibility; pose read back with `xcrun devicectl device motion hinge-angle`. Read-back is the record, not the request: closed 0°, and two sessions requested as `open` read back 130°, so only a later session reached a settled 180°
Oracle	`xcrun devicectl device info displays` (jsonVersion 5) for lit panel + `nativeSize` + `pointScale`, then `simctl io screenshot --display=LCD` and `--display=LCD-1`

Probe method: one XCTest function activated the app, then captured from XCUIScreen.main, every XCUIScreen.screens element, app.windows.firstMatch (frame read first, and again with no frame read), app, one button element, and the SpringBoard alert window. Per capture it recorded UIImage size/scale/orientation, CGImage pixel size, PNG byte counts from both PNGRepresentation and UIImage.pngData(), mean/max luma and non-black fraction over a 48×48 resample, and wrote every PNG. Luma never carries an equality claim here: panel identity comes from CoreDevice's displayId and pixel agreement from decoding both PNGs and comparing every pixel. Panel identity was confirmed against CoreDevice displayId, never inferred from pixels; mean luma only corroborated.

Evidence

Closed pose, hinge 0°, CoreDevice: LCD displayId 1 primary activeOn 1398x2034 @3 rot0; LCD-1 displayId 3 off 2007x2853 @3 rot90.

Capture surface	Result	Verdict
`simctl --display=LCD` (oracle)	1398x2034, mean luma 221.86	reference
`XCUIScreen.main.screenshot()`	1398x2034, luma 221.90	✅ supported
`XCUIScreen.screens[0]` (`displayID 1`)	1398x2034, luma 221.90	✅ supported
`XCUIScreen.screens[1]` (`displayID 3`, dark)	2006x2852, luma 0.00	correctly black — the dark panel is not capturable
resolved `window.screenshot()`	1398x2034, luma 221.90 — 0 of 2,843,532 pixels differ vs `screens[0]`	✅ supported
`app.screenshot()`	1398x2034, luma 221.90	✅ (only because main is lit)
`element.screenshot()` ("Drag source" cell, 188.67x56 pt)	566x168 crop, luma 240.01	✅ supported

Proof images, resized for upload — the pixel dimensions cited in the tables are the originals.

Inner panel lit. The settled 180° session (hinge 180.0°, CoreDevice LCD off / LCD-1 activeOn) is the primary run; two 130° sessions on other app content are corroboration. Every verdict below holds in all three.

Capture surface	Result	Verdict
`simctl --display=LCD-1` (oracle)	2853x2007, luma 192.02 (130° runs: 236.89 and 189.94)	reference
`simctl --display=LCD` (now dark)	2034x1398, luma 0.00	reference (dark)
`XCUIScreen.main.screenshot()`	1398x2034 header, presented turned, luma 0.00	❌ unsupported — still names panel 1, the dark outer panel; `XCUIScreen.h` documents `mainScreen` as "the primary screen of the device", so that is the documented meaning, not a glitch
`XCUIScreen.screens[0]` / `[1]`	luma 0.00 / 192.03 (130° runs: 236.91 and 189.94)	✅ each names its own panel — the choice is the caller's
resolved `window.screenshot()`	2006x2852, luma 192.03 — 142 of 5,721,112 pixels differ, each by at most 1 per channel (delta sum 400) vs `screens[1]`; the 130° run: luma 236.91, 187 differing, sum 530	✅ supported in every pose
`window.screenshot()` with no prior `frame` read	2006x2852, luma 192.03, 0 pixels differing from the row above	✅ `XCUIElement.screenshot()` resolves its own snapshot; the frame-first rule is needed for the `displayID` read, not for the capture
`app.screenshot()`	2006x2852 canvas, luma 0.00 (all three sessions)	❌ unsupported — inner geometry, dark content; `app.screen` is pinned to panel 1
`element.screenshot()`	168x1061 crop of the 353.67x56 pt "Drag source" cell the probe recorded, luma 193.41 (130° run: 241.81)	✅ supported; the crop is taken in panel-native geometry, so it is rotated relative to the element's frame

Single-panel control, iPhone 17 / iOS 27.0, one integrated panel (LCD displayId 1, 1206x2622 @3, activeOn, no active key — as ADR 0025 records): every surface agreed at 1206x2622, luma 224.48 vs oracle 224.44, rot0, no rotation, and both element crops came back upright. Nothing here changes that path except that panel identity becomes explicit.

Facts the capture path cannot read off the image

Identity. window.screen is an XCUIScreen whose displayID equals CoreDevice's displayId (1 = LCD, 3 = LCD-1) — but only after the window snapshot is resolved. Measured in open pose: app.screen.displayID = 1, unresolved window.screen.displayID = 1, resolved window.screen.displayID = 3. Same ordering fix(ios): target the active window for foldable interactions #2724 measured for gesture routing. XCUIScreen answers displayID, scale, bounds (KVC/responds(to:)) and raises NSUnknownKeyException for frame and name; XCUIScreen.self responds only to screens/mainScreen — no displayWithID:/screenWithDisplayID:orError: — so a panel is selected by filtering XCUIScreen.screens. No backlight or active fact exists there, so panel power stays CoreDevice's. ADR 0025's sentence was right about the declared surface and understated the runtime: XCUIAutomation.framework/Headers/XCUIScreen.h in this toolchain declares mainScreen and screens and nothing else, and no XCTest header mentions displayID. Every identity read above is KVC against an undeclared key — the same category of read fix(ios): target the active window for foldable interactions #2724 already depends on for gesture routing — so picking the lit panel was never impossible, it was undocumented, and taking that on is fix(ios): use the resolved display for screenshots and visual verification #2728's decision to make, not this measurement's.
Geometry. XCUIScreen.scale = 3 = that panel's CoreDevice pointScale. Captures arrive in the panel's native geometry, not the interface orientation: on the rot90 inner panel the PNG header is 2006x2852 with the landscape content turned inside it (UIImageOrientation(rawValue: 3), whose display size transposes the panel — naming the case was not verified and a 180-degree case is ruled out by that transposition), while simctl exports that panel 2853x2007 upright. Rotating alone does not make them comparable: the canvases are a pixel apart on each axis (2006x2852 against 2853x2007). The runner's production encoder and PNGRepresentation agree on the pixels — 0 differing pixels from 358,402 and 328,181 encoded bytes for the same capture — so only the compression differs, and the rotation is the capture's own. A runner capture and a simctl capture of the same panel are not comparable until one is rotated.
Overlays included. Screen and window captures are display captures (the window one cropped to the window), so the software keyboard, a SpringBoard-hosted microphone-permission alert, and the status bar all appear in both — verified by viewing the PNGs and by luma agreement with the oracle. A capture of the alert's own com.apple.springboard window is a black 668x950 crop — and that window's resolved screen.displayID was 3, the dark panel in the closed pose, so panel power explains the crop without needing any system-window rule. Whether a system window on the lit panel captures was not measured, so verifying a system surface on the app's panel remains the safe rule. Proof, a closed-pose app-window capture with the keyboard composited in:

Unresolved window. XCTest refuses rather than defaulting: asking screenshot data of a non-existent element raises Element Window (First Match) cannot request screenshot data because it does not exist — no silent XCUIScreen.main fallback is observable on that path. XCUIAutomation does contain Could not resolve displayID for snapshot %@; falling back to main display (id %lld), so an internal snapshot→display resolution failure can fall back to main silently; that is a reason to resolve explicitly, not to rely on the element path's own lookup.

Proposed helper (not implemented here)

struct RunnerDisplayCapture {
  let pngData: Data                  // exactly the bytes XCTest produced — no silent rotation
  let displayID: UInt64              // window.screen.displayID == CoreDevice displayId
  let panelScale: CGFloat            // XCUIScreen.scale (measured == CoreDevice pointScale)
  let panelNativeSize: CGSize        // XCUIScreen.bounds.size, panel-native pixels
  let windowFrame: CGRect            // resolved window frame, window point space
  let imageOrientation: UIImage.Orientation   // measured .downMirrored on the rot90 panel
}

enum RunnerCaptureFailure: Error {
  case windowUnresolved(bundleId: String)   // no window, or a null/empty frame after resolution
  case displayUnresolved(bundleId: String)  // window resolved, screen/displayID missing
}

func captureApplicationDisplay(of app: XCUIApplication) throws -> RunnerDisplayCapture

Order is the one #2724 proved and ships: read windows.firstMatch.frame to resolve the snapshot, then read screen for displayID/scale/bounds, then window.screenshot(). #2724 already owns that resolution in RunnerResolveApplicationDisplayID (RunnerXCTestEventBridge.m) with these same two failure cases and unit tests, so the capture helper should extend it rather than open a third route to the same private key — and a fourth path has to converge with it: onScreenWindowFrame (RunnerTests+Interaction.swift:222-231) resolves the same window through windows.element(boundBy: 0) and silently falls back to app.frame, the never-refuse shape this issue warns about. An unresolved window throws .windowUnresolved — it never captures XCUIScreen.main and never pairs an image with a panel nobody measured. That matters because XCUIScreen.main is exactly what today's fallback captures, so a main-screen default would reintroduce the black-panel bug under a new name. displayID is the seam to the existing resolver, with the caveat that it is an undeclared KVC key, so #2728 inherits the breakage risk #2724 accepted for gesture routing. #2728 can require capture.displayID == litPanel.displayId from resolveAppleCaptureDisplay and report a typed mismatch instead of trusting either side, which keeps CoreDevice as the panel authority and gives density normalization the captured panel's own pointScale instead of a scale probe. The rotation column is the input #2729 needs before runner and simctl captures are compared.

Consequence for #2728

XCUIScreen.main is six call expressions in four files, not one site. iOS-reachable: the screenshot command's #else branch (RunnerTests+CommandExecution.swift:2075) — :2068 and :2072 sit under #if os(macOS) and are themselves XCUIScreen.main captures — only the :2070 branch between them captures the app's root — so these runs prove nothing about the desktop behavior, and the desktop also has a Screen Capture Kit route (AgentDeviceMacOSHelper/main.swift:567); captureRunnerFrame (RunnerTests+Lifecycle.swift:45-48), which feeds both the recorder (RunnerTests+CommandExecution.swift:1468) and the keyboard stability fingerprint (RunnerTests+Keyboard.swift:259); the in-app-back visual-state probe (RunnerTests+Navigation.swift:141); and the synthesized-gesture reference frame (RunnerTests+Interaction.swift:841), which only the unmerged #2724 replaces.

Two of those fail less visibly than a black screenshot: identical black samples satisfy runnerScreenshotStabilitySettled on the first interval whatever the keyboard is doing, and an unchanged black pair logs AGENT_DEVICE_RUNNER_IN_APP_BACK_FALLBACK_NO_STATE_CHANGE and reports a back navigation that worked as one that failed. They call the API this probe ran, so that follows by construction and not by measurement.

The host side moves with them: packages/platform-apple/src/core/screenshot.ts:179 passes undefined scale because "The runner captures XCUIScreen.main", display-inventory.test.ts:468 asserts exactly that, and website/docs/docs/commands.md:1122 tells users physical capture is built from repeated XCUIScreen.main.screenshot() frames.

Collision to resolve before review: #2724 edits this same gap paragraph and inserts a section at the same point in ADR 0025. #2739 supersedes the fallback bullet's question, and whichever PR lands second rebases onto this measurement.

Method limits, and one incidental defect

Poses were set by pressing Device Hub's accessibility controls; agent-device fold was tried first and is where the incidental defect below surfaced. ADR 0025's Device identity row said fold selects by UDID and only then presses; DeviceHubPose.swift:201-204 skips selection when the sidebar row is absent and the window title already matches, and :217-229 confirms by title too, so that row is corrected in docs(adr): record which runner capture surface sees the lit foldable panel #2739.
Half-open and the SpringBoard-alert overlay set were measured in the same session, and half-open was reproduced on a second boot for the open-pose rows.
A rot90 outer panel and a quarter-turned foldable were not reachable: the pose presets are Closed/Book/Open and no available path rotates a foldable, so the rotation finding covers the inner panel only.
No permanent runtime code changed. The probe file was deleted, its adprobe containers removed, the session closed, the daemon stopped with --clean, both simulators I booted shut down, and my Duo left at the default closed pose.
Incidental, recorded in the ADR gaps by docs(adr): record which runner capture surface sees the lit foldable panel #2739: with two simulators named iPhone Duo, fold aimed at one pressed the pose control of the other. It correctly refused (iPhone Duo did not reach the open pose: CoreDevice still reports a hinge angle of 0°) because the hinge read-back is the verdict — but the other simulator physically unfolded. The Device Hub window title (iPhone Duo – iOS 27.1) cannot distinguish twins, and pressed.selected is only emitted as the apple_fold_pose_pressed diagnostic (packages/platform-apple/src/foldable/pose.ts:65, selected at :72) while the refusal at :163 keys on the hinge angle, so an unconfirmed selection still presses. Explicitly selecting the target's sidebar row before pressing routed every later pose correctly. fix(fold): distinguish an observed half-open angle from a settled pose #2730 turned out to be the settle rule rather than this, so the selection hazard is still open and is recorded as an ADR 0025 evidence gap by docs(adr): record which runner capture surface sees the lit foldable panel #2739.

Erratum to my evidence comment above

A second review pass checked the comment against the retained artifacts instead of against my notes, and six statements in the first version did not survive that. The comment has been rewritten in place; this is the record of what changed.

The open-pose table was not from a 180° session. Its numbers came from a run whose hinge read back Angle:130,0°, as did an earlier session I had also booked as open. The settled 180° session is a different one, and its values are now the primary column: simctl --display=LCD-1 oracle luma 192.02, XCUIScreen.main 0.00, resolved window 192.03 with 142 of 5,721,112 pixels differing from screens[1] at a maximum of one per channel, app.screenshot() 0.00, element crop 193.41. The requested pose is not the pose; the hinge-angle read-back is. That is a method error, not a typo, and it is the same mistake ADR 0025 warns about for fold.
"Half-open reproduced every open-pose row" compared two 130° sessions (189.94 and 236.91). Both sessions are now labelled corroboration of the 180° verdicts, which is what they actually show.
2034x1398 for the XCUIScreen.main capture is not its PNG header size, which is 1398x2034. The transposed figure came from NSImage-based dimension reporting — the same tool whose dimensions this comment already distrusts for another reason.
The UIImage orientation case was named without evidence. Only UIImageOrientation(rawValue: 3) is in the probe log. The case name was wrong enough to be load-bearing: a 180° orientation would not transpose the display size, and the recorded size does transpose.
RunnerTests+CommandExecution.swift:2068 and :2072 are themselves XCUIScreen.main.screenshot() calls, both under #if os(macOS); the app-root capture is the :2070 branch between them. The first version said those two "keep app-targeted capture", which would have sent fix(ios): use the resolved display for screenshots and visual verification #2728's implementer to the wrong branches.
iPhone19,4 is dropped. No retained artifact records a model identifier for the Duo simulator.

Two smaller fixes rode along: the closed-pose element is the "Drag source" cell rather than a button, and the element-crop luma is 241.81 from the PNG rather than 241.82 from the probe's own resample.

One addition worth the reviewers' attention: rotating a runner capture is not enough to compare it with a simctl capture of the same panel. The runner's canvas is 2006x2852 and simctl's is 2853x2007, so they are a pixel apart on each axis as well as turned — #2729 owes a rotation and a crop.

ADR 0025 carries the same corrections in #2739, now 45e29845b after rebasing onto main; the two review-pass commits (8b3d56a0e, 97fc04631) stay readable in the PR's timeline. The verdict this issue asked for is unchanged: the resolved app window is the capture surface that follows the fold, and XCUIScreen.main and XCUIApplication.screenshot() are not — reproduced in all three inner-panel sessions.

docs(adr): record which runner capture surface sees the lit foldable …

Navigation Menu

Repository navigation

investigate(ios): establish a measured screen-capture source for the active window #2727

Purpose

Inputs and narrow scope

Deliverable / stop condition

Working rules