Repository navigation
Commit 0d5e5d6
fix(ios): book every surface on the lit panel, not the first window (#4122)
* fix(ios): book every surface on the lit panel, not the first window
On an iPhone Duo home screen the application's first window is SpringBoard's
status-bar window: a 223x317 box on a panel that may be dark. Taps, the
snapshot viewport, the capture display and the synthesized record all took
that window as "the screen", so every one of them answered for a screen
nobody was looking at.
One rule now names the window a command books its surface against
(RunnerPanelSurface): the window the host reports as the lit panel shows.
The host reads panel power it already has and sends the lit panel as a typed
fact on the requests that resolve a screen; a request that names no panel
keeps the historic first-window walk, so single-panel iPhone, iPad and tvOS
behavior is byte-for-byte the request and the read it always were. Geometry
never guesses a panel: the dark panel's windows have perfectly good frames.
A regular capture additionally applies the panel fact to its tree
(RunnerPanelSurfaceTree): the root carries the surface box the host reads as
the payload's viewport, surfaces off the panel are dropped with their
subtrees, and when the application's subtree carries no window for the lit
panel the resolved window is read once and grafted, which is what makes the
home screen's icons addressable. Raw retains reported frames. A device whose
every window sits on the dark panel refuses with APP_SCREEN_OFF_LIT_PANEL
rather than capturing black.
Device evidence (iPhone Duo 9542F4F5, iPhone 17 Pro BC54AC11):
- folded: lit=1(466x678) selection=litPanelWindow frame=(0,0 466x678);
home-screen snapshot lists icons; icon long-press opens the icon menu;
Safari tap opens Safari; SYNTHESIZED_DISPATCH reference=(0,0 466x678),
SYNTHESIZED_RECORD displayID=1; screenshot 466x678.
- unfolded: lit=3(669x951) frame=(0,0 951x669), root republished from the
dark panel's 466x678, 4 off-panel surfaces dropped, tree grafted to 67
nodes; snapshot lists icons; long-press opens the context menu; Settings
tap opens Settings; reference=(0,0 951x669), record displayID=3;
screenshot 951x669.
- iPhone 17 Pro: tap and snapshot unchanged, no panel fact on the wire, no
panel markers in runner.log, reference=(0,0 402x874).
* refactor(ios): resolve one lit-panel window per request
Reduce the foldable surface fix to one rule. The host reads the panel
table once per surface command (the command traits name which ones) and
sends `litPanel`. The runner walks at most eight windows, books the
first full-panel window on the lit display, and uses it for dispatch,
the coordinate fallback, the viewport, and capture. A regular snapshot
keeps only the windows whose element snapshot names the lit display.
Requests without `litPanel` keep main's first-window routes unchanged.
The graft-and-prune tree pass, the panel-rule module, the
APP_SCREEN_OFF_LIT_PANEL code, and the rotation retry option are gone.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs: correct the lit-panel comments a review flagged
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): fail closed when the lit panel cannot be booked
A request that names a lit panel now books that panel's window before it
acts. A refusal comes back typed before any gesture or snapshot, and every
later read in the request reuses the booked window, so nothing falls back
to the application's own coordinates.
- The window walk refuses after an unplaced window, even when a full-panel
window follows it.
- An app whose windows are all on the dark panel is APP_SCREEN_OFF_LIT_PANEL,
and a capture no longer swaps it for SpringBoard.
- A lit-panel snapshot runs only the tree tier and keeps only windows
proven on the lit display.
- A resolved panel-table read clears the probe back-off, and the probe
never outlasts the request's timeoutMs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): order the panel-table back-off behind newer resolved reads
A failed read that finishes after a newer read resolved no longer sets
the 5-second back-off: each read takes a sequence number, and a failure
yields to any later read that resolved. The ADR now says the snapshot
traverses the tree and keeps every window proven on the lit display.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(adr): count the off-lit-panel capture refusal
ADR 0025's capture contract listed three fail-closed cases. The lit-panel
amendment added a fourth, `offLitPanel` (APP_SCREEN_OFF_LIT_PANEL), and
changed when capture asks the system surface on a multi-panel device.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): keep every lit-panel read on proven lit windows
- Element reads (readText, the text-input lookup, selector queries) search
only under the windows that report the lit display. One bridge owner,
RunnerCopyLitPanelWindows, provides them.
- The lit panel is written onto the main queue behind work already
queued there, so work a previous command abandoned keeps its own panel.
The booking is keyed to its app.
- A snapshot books in command preparation, publishes the booked frame as
its root, refuses a pin to a backend that cannot prove the panel, and
filters --raw too.
- A known multi-panel device whose panel table goes unread is refused
with a typed panel-table-unreadable error. The runner gets the budget
the panel read left.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): keep raw, unreadable-list and pin refusals on the lit panel
- `--raw` publishes the booked window's frame as its root, so raw and
regular coordinates share one space.
- RunnerCopyLitPanelWindows returns nil for an unreadable window list.
A lit-panel element read then refuses with APP_SCREEN_UNRESOLVED
instead of finding nothing.
- A backend pin on a lit-panel snapshot is refused in command
preparation, so it leaves the cached target alone.
- ADR 0025 says what a recording does after a fold: it keeps
capturing the panel recordStart named.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): let element search own the lit-panel refusal
Every element read on a lit-panel request searches the windows proven on
the lit display, or `elementSearchRoots` throws APP_SCREEN_UNRESOLVED,
which `executeOnMainPrepared` returns as the reply. This replaces the
command-name precheck, so a coordinate `type` whose window list cannot be
read refuses instead of tapping a point with no target.
The snapshot path reads the booked window only from its capture target.
`findText` and `querySelector` join the lit-panel-scoped commands, so the
traits table is the one list. A `customActions` snapshot names
`customActions` in its refusal. The client no longer short-circuits on a
`litPanel` field no caller sets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(ios): share the lit-panel and penalized-clear test setups
The new unreadable-list case and the pin case launched the app on its own
panel the same way, and the added `try` made two penalized-clear calls a
new clone. One helper each removes both clones.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(adr): qualify when customActions implies the private AX pin
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): refuse an unread lit-panel window list instead of falling back
An unreadable window list on a lit-panel request resolved as "no window",
which sends a capture to the system-surface fallback. It now refuses with
APP_SCREEN_UNRESOLVED. A panel read that spends the whole request budget is
refused before sending, instead of handing the runner a 1 ms timeout.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): contain the lit-panel window walk and keep its refusals closed
The element search's window walk now runs inside an issue scope, so a
window that vanishes mid-walk refuses with APP_SCREEN_UNRESOLVED instead
of recording an XCTest failure. A text-entry re-read refused at its
refresh point answers no input instead of the app-wide identifier query,
and the Maestro-compat frame gate refuses a lit-panel request with no
booked window. The snapshot pin refusal keeps one message that names its
cause.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor(ios): drop the single-caller textInputAt and trim lit-panel comments
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): route every lit-panel text-entry read through the element search
The bound-input identifier query and the in-app back control searched the
whole application, so on an unfolded Duo a same-identifier input or back
button on the dark panel could answer. Both now search the element
search's roots, and its refusal propagates: text-entry resolution throws
instead of answering nil, and boundTextEntryInputIsGone answers "cannot
prove gone". One owner returns the request's remaining budget, so the
panel read and the runner timeout share one clock. The unreachable
hasTappableFrame guard is removed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): book the lit panel's whole box for a window that leaves a strip
On the unfolded Duo, an app's window is 871x669 beside the inner panel's
status strip, and the booking used that window frame as the synthesized
dispatch reference. The orientation flip then mapped points against 871
instead of the 951-point screen, so taps landed 80 points off, and a
fill's focus tap missed. The booked frame is now the panel's box turned
like the window, at its origin, which is unchanged for full-panel
windows.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(ios): tighten the lit-panel booking comment
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(ios): share the bound-input fixture target
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): propagate lit-panel refusals from the commit poll and clear reads
The synthesized commit wait's observe closure and the replacement clear's
read swallowed a lit-panel refusal with `try?`, so the wait timed out or
the clear reported TEXT_INPUT_CLEAR_NOT_OBSERVED. Both now throw, and the
refusal answers the request. The clear read also keeps main's answer for
an input that is simply gone.
The bridge exports one lit-panel resolver that returns the window, frame
and screen, and lit-panel comments are trimmed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): book the lit panel's screen box at the screen origin
The booked box was inferred from the window: its origin and whether it
was wider than tall. A window that the inner panel's status strip pushes
to a non-zero origin would then book a box 80 points past the screen,
and every tap, reference and snapshot root would land off. The box is
now the panel's native portrait box laid out in the app's interface
orientation, at the screen origin, through the existing
soleScreenReferenceFrame. The XCUICoordinate anchor keeps the window's
own origin.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): turn lit-panel taps about the screen, keep the window's translation
The previous commit booked the screen box, which broke the placed-window
translation: XCTest reports a window at (0,80) from its own top-left, and
the snapshot shifts those frames into screen space only when the viewport
is that window's frame. The booking is the window frame again, as for a
placed iPad window (#3899), and on a lit-panel request the synthesized
turn is about the lit panel's screen in the interface orientation instead
of the window's shape. That replaces the aspect guess and the window
origin inference.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(ios): drive the lit-panel booking for a partial window; scope the walk cap
The rotation-basis case now books the launched app's window under a lit
panel taller than it and checks the synthesized reference and turn box,
so a wiring regression fails it. The one-caller screen-frame helper is
inlined. The window cap's doc names the booking walk it bounds; the
element-search copy stays uncapped so every lit window is searched.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(adr): map each capture refusal to its cases and scope the window cap
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(ios): preserve screen points across lit-panel coordinate fallbacks
* fix(ios): key lit-panel bookings by app and request identity
* docs(adr): scope the panel-probe budget refusal to a named lit panel
* fix(ios): refuse unbooked lit-panel reference frames
* fix(ios): scope collapsed-tab candidates to the captured lit display
* fix(ios): refuse unread lit-panel window frames
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent a2c169f commit 0d5e5d6
70 files changed
Lines changed: 2813 additions & 668 deletions
File tree
- apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests
- UnitTests
- contracts/fixtures
- docs/adr
- packages
- apple-runner/src
- __tests__
- platform-apple/src
- __tests__
- core
- foldable
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
| 31 | + | |
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
34 | 42 | | |
35 | 43 | | |
36 | 44 | | |
| |||
39 | 47 | | |
40 | 48 | | |
41 | 49 | | |
| 50 | + | |
| 51 | + | |
42 | 52 | | |
43 | 53 | | |
44 | 54 | | |
| |||
50 | 60 | | |
51 | 61 | | |
52 | 62 | | |
| 63 | + | |
| 64 | + | |
53 | 65 | | |
54 | 66 | | |
55 | 67 | | |
| |||
82 | 94 | | |
83 | 95 | | |
84 | 96 | | |
| 97 | + | |
| 98 | + | |
85 | 99 | | |
86 | | - | |
| 100 | + | |
| 101 | + | |
87 | 102 | | |
88 | 103 | | |
89 | 104 | | |
90 | 105 | | |
91 | 106 | | |
92 | | - | |
| 107 | + | |
93 | 108 | | |
94 | 109 | | |
95 | 110 | | |
| |||
109 | 124 | | |
110 | 125 | | |
111 | 126 | | |
112 | | - | |
| 127 | + | |
113 | 128 | | |
114 | 129 | | |
115 | 130 | | |
116 | | - | |
117 | | - | |
| 131 | + | |
| 132 | + | |
118 | 133 | | |
119 | 134 | | |
120 | 135 | | |
121 | 136 | | |
122 | | - | |
| 137 | + | |
| 138 | + | |
123 | 139 | | |
124 | 140 | | |
125 | 141 | | |
126 | 142 | | |
127 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
128 | 154 | | |
129 | 155 | | |
130 | 156 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
193 | | - | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
226 | | - | |
| 226 | + | |
227 | 227 | | |
228 | 228 | | |
229 | 229 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
290 | 290 | | |
291 | 291 | | |
292 | 292 | | |
293 | | - | |
| 293 | + | |
294 | 294 | | |
295 | 295 | | |
296 | 296 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | | - | |
| 98 | + | |
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| |||
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
144 | | - | |
| 144 | + | |
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
193 | | - | |
| 193 | + | |
194 | 194 | | |
| 195 | + | |
195 | 196 | | |
196 | 197 | | |
197 | 198 | | |
| |||
223 | 224 | | |
224 | 225 | | |
225 | 226 | | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
233 | 241 | | |
234 | 242 | | |
235 | 243 | | |
| |||
247 | 255 | | |
248 | 256 | | |
249 | 257 | | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
256 | 263 | | |
257 | 264 | | |
258 | 265 | | |
| 266 | + | |
259 | 267 | | |
260 | 268 | | |
261 | 269 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
432 | 434 | | |
433 | 435 | | |
434 | 436 | | |
| 437 | + | |
| 438 | + | |
435 | 439 | | |
436 | 440 | | |
437 | | - | |
| 441 | + | |
438 | 442 | | |
439 | 443 | | |
440 | 444 | | |
| |||
0 commit comments