Skip to content

Commit cdebec6

Browse files
thymikeeclaude
thymikee
and
claude
authored
fix(ios): publish a placed iPad window's capture in screen space (#3920)
* fix(ios): publish a windowed iPad app's capture in screen space (#3899) An iPad window that does not fill the screen reports its own screen box while its subtree arrives anchored at its own top-left. The capture published a screen-space viewport around window-relative rects: the fold clipped content to (w - x) x (h - y), a rect named no screenshot position, and a synthesized tap landed at the point minus the window origin. SnapshotContentTranslation reads the one vector from a carrier's (0,0)-anchored copy of the viewport box to the viewport itself, and SnapshotGeometrySpace.normalized translates the subtree it declares in the same pass that turns a quarter-turned subtree. CoordinateSpaceRotation now turns axes only; the digitizer origin stays where the display put it. Replayed from contracts/fixtures/window-coordinate-space.json in both languages. Refs #3899 * revert: keep the snapshot-route warning owned by #3914 The refusal wording belongs to the snapshot-route failure policy, not to this fix. Refs #3899 * fix(ios): read a placed window's snapshot frames at their screen place (#3899) XCUIElement.frame reports a placed window's content in screen space, but XCUIElementSnapshot.frame reports it in the window's own content space, so the text-input probe saw two spaces on an iPad Windowed Apps window and refused a fill whose point was on the screen. The probe, the focus tap, and the typing reader now read the same SnapshotContentTranslation vector the capture reads, from the two live boxes that name it. Refs #3899 * fix(ios): keep caret repositioning and keyboard evidence decision stable on a placed window [ #3899 ] * fix(ios): translate only snapshot frames and pin the unmeasured landscape case [ #3899 ] * refactor(ios): keep the window-content translation with the capture that runs it [ #3899 ] * refactor(ios): make ResolvedTextEntryInput own the probe-side screen frame (#3899) The readiness retap re-read the input as a bare XCUIElementSnapshot and tapped its raw center through tapAt, which takes screen points, so on a windowed iPad the retap landed outside the window. The live window lookup also ran whole-app queries four to six times per fill on every iOS device. Every snapshot frame that becomes a dispatched point now passes through the one content translation, held on ResolvedTextEntryInput.frameOnScreen. The vector is read once per command and stored on TextEntryTarget.contentTranslation; the probe matching, focus tap, refresh point, caret tap and readiness retap all read that one property. The threaded translation: parameters and the per-read lookups are gone, and the once-per-command keyboard read in the typing command is reverted to the base behavior with the retap fixed. * docs(ios): name the probe-side owner and the screen-space dispatch basis (#3899) Amends ADR 0004 to name ResolvedTextEntryInput as the probe-side owner of the content vector, cites the measured Add-button rect at (659,321), records that the runner republishes a windowed capture in screen space where the bridge-refusal warning said it was offset, and names screen space plus the app.frame origin assumption in the interactionCoordinate comment. * refactor(ios): cut the screen-space comments and the duplicated frame offset (#3899) The vector on the target replaces the offset arithmetic in textEntryRefreshPoint, which read the snapshot frame and added the vector itself. One owner now: ResolvedTextEntryInput.frameOnScreen. The comments that restated the capture's rule at every call site are cut to the constraint the code cannot show. SnapshotCoordinateSpace.swift ships in the npm package without comment stripping, so its prose lives in ADR 0004 and the golden table instead; the same rule text is removed from the TypeScript decoder's foot note. * refactor(runner): make the text-entry content vector a non-defaulted tri-state frameOnScreen becomes optional and carries the #3060 rule: it names no box without a snapshot or under an unknown content space, so the four consumer guards collapse into one nil check. liveContentTranslation returns TextEntryContentTranslation (known/none/unknown) instead of CGSize?, so a failed window read can no longer look like full screen and send the readiness retap to a raw window-local point. The vector now rides on TextInputAtPoint, cutting one parameter from the probe signatures. retapAt skips the point on unknown; probe matching and the focus tap fall back to element.frame. Adds the unknown-space retap unit test. * feat(runner): log the readiness retap point and its skips * fix(runner): take the probe vector from the resolved window box alone Reply to review 5642729478: - liveContentTranslation builds the zero-origin reported box from the resolved window's size, so an app.frame that unions transformed subtrees cannot silently cost a placed window its vector (P1 thread 1). - Synthesized dispatch turns about the display's oriented box, never a placed window's smaller box; landscape under Windowed Apps stays unmeasured and the ADR says so (P1 thread 2). - requireTextInputAtPoint passes the live vector (P2 thread 3). - ADR 0004 distinguishes the refusing bridge from the converting runner and states the focus-tap/retap refusal rule (P2 threads 4-5). - The initial focus tap dispatches the caller's requested point when the read names no screen box; the live element.frame is never dispatched, and focusTapPoint is pinned by a unit test (P2 thread 7). * refactor(runner): shrink the windowed-iPad screen-space change to its rule Keep the accepted design (one SnapshotContentTranslation rule in normalized, ResolvedTextEntryInput.frameOnScreen, a non-defaulted known/none/unknown vector read once per command, rotation about the sole display) and cut what did not serve it: cosmetic renames and reformatting, comments that restated code, the one-caller focusTapPoint helper, TextInputAtPoint's copy of the target's vector, the duplicated sole-display math, duplicate unit tests and fixture rows, and most of the ADR 0004 amendment prose. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(snapshot): pin translated hittability and the offset rule's empty-box guards Review follow-ups to the shrink: assert a translated node's hittability, add the two fixture rows that discriminate the positive-finite guards, keep windowContentCases names unique, read the display size from the fixed coordinate space, and correct ADR 0004's vector-read and gap wording. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(runner): restore the discriminating window-content rows and the retap skip reason Put back the moved-window, keyboard, and SpringBoard rows in windowContentCases, and log one skip line carrying its reason when the readiness retap names no screen point. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 7ee441f commit cdebec6

23 files changed

Lines changed: 437 additions & 115 deletions

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,8 @@ extension RunnerTests {
352352

353353
func readTextAt(app: XCUIApplication, x: Double, y: Double) -> String? {
354354
let point = CGPoint(x: x, y: y)
355-
let textInputCandidates = textInputCandidatesAt(app: app, point: point).map(\.element)
355+
let textInputCandidates = textInputCandidatesAt(app: app, point: point, contentTranslation: liveContentTranslation(app: app))
356+
.map(\.element)
356357
for element in textInputCandidates where prefersExpandedTextRead(element) {
357358
if let text = readableText(for: element) {
358359
return text
@@ -618,7 +619,8 @@ extension RunnerTests {
618619
private func interactionCoordinate(app: XCUIApplication, x: Double, y: Double) -> XCUICoordinate {
619620
#if os(iOS)
620621
// Coordinate taps, double taps, long presses, and drags anchor at the app origin, not the
621-
// resolved window: on iOS the snapshot hands back app-space points, and the first qualifying
622+
// resolved window: callers pass screen points, and `app.frame` starts at the screen origin even
623+
// for a placed iPad window (#3899). The first qualifying
622624
// window is not always the one under the finger — a SpringBoard `alert accept` can live in an
623625
// alert window while the wallpaper or status-bar window qualifies first, so a window-relative
624626
// anchor misses the button. Reference frames and the synthesized display ID still resolve

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -31,14 +31,15 @@ extension RunnerTests {
3131
)
3232
}
3333
let frame = context.referenceFrame
34+
let rotationFrame = synthesizedRotationFrame(fallback: frame, interfaceOrientation: orientation)
3435
let start = CoordinateSpaceRotation.native(
3536
point: CGPoint(x: x, y: y),
36-
in: frame,
37+
in: rotationFrame,
3738
interfaceOrientation: orientation
3839
)
3940
let end = CoordinateSpaceRotation.native(
4041
point: CGPoint(x: x2, y: y2),
41-
in: frame,
42+
in: rotationFrame,
4243
interfaceOrientation: orientation
4344
)
4445
logSynthesizedDispatch(
@@ -117,7 +118,7 @@ extension RunnerTests {
117118
}
118119
let point = CoordinateSpaceRotation.native(
119120
point: CGPoint(x: x, y: y),
120-
in: context.referenceFrame,
121+
in: synthesizedRotationFrame(fallback: context.referenceFrame, interfaceOrientation: orientation),
121122
interfaceOrientation: orientation
122123
)
123124
logSynthesizedDispatch(
@@ -191,7 +192,7 @@ extension RunnerTests {
191192
let orientation = Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
192193
let point = CoordinateSpaceRotation.native(
193194
point: CGPoint(x: x, y: y),
194-
in: context.referenceFrame,
195+
in: synthesizedRotationFrame(fallback: context.referenceFrame, interfaceOrientation: orientation),
195196
interfaceOrientation: orientation
196197
)
197198
logSynthesizedDispatch(
@@ -524,24 +525,38 @@ extension RunnerTests {
524525
/// that can show an app in part of a screen (iPad, more than one screen) keeps requiring a window.
525526
@MainActor
526527
func soleScreenReferenceFrame(app: XCUIApplication) -> CGRect? {
527-
guard UIDevice.current.userInterfaceIdiom == .phone, XCUIScreen.screens.count == 1 else {
528+
guard UIDevice.current.userInterfaceIdiom == .phone else {
528529
return nil
529530
}
531+
let frame = soleDisplayFrame(
532+
interfaceOrientation: Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
533+
)
534+
NSLog(
535+
"AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved=%@",
536+
frame.map { NSCoder.string(for: $0) } ?? "none"
537+
)
538+
return frame
539+
}
540+
541+
/// The box a synthesized turn swaps side lengths about. The digitizer turns about its display, so a
542+
/// placed iPad window's smaller box is not the basis (#3899); a multi-screen host keeps the
543+
/// resolved window, which names its panel (ADR 0025).
544+
func synthesizedRotationFrame(fallback: CGRect, interfaceOrientation: Int) -> CGRect {
545+
soleDisplayFrame(interfaceOrientation: interfaceOrientation) ?? fallback
546+
}
547+
548+
private func soleDisplayFrame(interfaceOrientation: Int) -> CGRect? {
549+
guard XCUIScreen.screens.count == 1 else { return nil }
530550
let screen = UIScreen.main
531551
let scale = screen.nativeScale
532552
guard scale > 0 else { return nil }
533-
let frame = Self.soleScreenReferenceFrame(
553+
return Self.soleScreenReferenceFrame(
534554
nativePortraitSize: CGSize(
535555
width: screen.nativeBounds.width / scale,
536556
height: screen.nativeBounds.height / scale
537557
),
538-
interfaceOrientation: Int(RunnerSynthesizedGesture.interfaceOrientation(forApplication: app))
539-
)
540-
NSLog(
541-
"AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE resolved=%@",
542-
frame.map { NSCoder.string(for: $0) } ?? "none"
558+
interfaceOrientation: interfaceOrientation
543559
)
544-
return frame
545560
}
546561
#endif
547562

@@ -732,11 +747,12 @@ extension RunnerTests {
732747
width: plan.viewport.width,
733748
height: plan.viewport.height
734749
)
750+
let rotationFrame = synthesizedRotationFrame(fallback: frame, interfaceOrientation: orientation)
735751
let pointerSamples: [[[String: NSNumber]]] = plan.pointers.map { pointer in
736752
pointer.samples.map { sample in
737753
let point = CoordinateSpaceRotation.native(
738754
point: CGPoint(x: sample.point.x, y: sample.point.y),
739-
in: frame,
755+
in: rotationFrame,
740756
interfaceOrientation: orientation
741757
)
742758
return [

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextEntry.swift‎

Lines changed: 48 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -145,10 +145,21 @@ extension RunnerTests {
145145
let identity: TextEntryInputIdentity
146146
}
147147

148+
/// The vector from a snapshot frame to the screen (#3899, ADR 0004), read once per command by
149+
/// `liveContentTranslation` and never defaulted: a vector nobody read must not pass for full screen.
150+
enum TextEntryContentTranslation {
151+
case known(CGSize)
152+
/// The app fills the screen, so a snapshot frame is already a screen frame.
153+
case none
154+
/// The window read could not answer, so no snapshot frame names a screen box.
155+
case unknown
156+
}
157+
148158
struct TextEntryTarget {
149159
let element: XCUIElement?
150160
let refreshPoint: CGPoint?
151161
let prefersFocusedElement: Bool
162+
let contentTranslation: TextEntryContentTranslation
152163
let fromTapWitness: Bool
153164
/// The input the first resolved element was. Once bound, resolution refuses any other input.
154165
let boundIdentity: TextEntryInputIdentity?
@@ -166,6 +177,7 @@ extension RunnerTests {
166177
element: XCUIElement?,
167178
refreshPoint: CGPoint?,
168179
prefersFocusedElement: Bool,
180+
contentTranslation: TextEntryContentTranslation,
169181
fromTapWitness: Bool = false,
170182
boundIdentity: TextEntryInputIdentity? = nil,
171183
inputAtRefreshPoint: TextInputAtPoint? = nil,
@@ -174,6 +186,7 @@ extension RunnerTests {
174186
self.element = element
175187
self.refreshPoint = refreshPoint
176188
self.prefersFocusedElement = prefersFocusedElement
189+
self.contentTranslation = contentTranslation
177190
self.fromTapWitness = fromTapWitness
178191
self.boundIdentity = boundIdentity ?? inputAtRefreshPoint?.identity
179192
self.inputAtRefreshPoint = inputAtRefreshPoint
@@ -186,11 +199,12 @@ extension RunnerTests {
186199
guard let input else {
187200
return self
188201
}
189-
let frame = input.snapshot?.frame ?? .zero
202+
let frame = input.frameOnScreen ?? .zero
190203
return TextEntryTarget(
191204
element: input.element,
192205
refreshPoint: frame.isEmpty ? refreshPoint : CGPoint(x: frame.midX, y: frame.midY),
193206
prefersFocusedElement: prefersFocusedElement,
207+
contentTranslation: contentTranslation,
194208
fromTapWitness: fromTapWitness,
195209
boundIdentity: boundIdentity ?? input.identity,
196210
inputAtRefreshPoint: inputAtRefreshPoint
@@ -208,16 +222,18 @@ extension RunnerTests {
208222
let element: XCUIElement
209223
/// Nil when the element exists but could not be snapshotted.
210224
let snapshot: XCUIElementSnapshot?
225+
let contentTranslation: TextEntryContentTranslation
211226

212-
private init(element: XCUIElement, snapshot: XCUIElementSnapshot?) {
227+
private init(element: XCUIElement, snapshot: XCUIElementSnapshot?, contentTranslation: TextEntryContentTranslation) {
213228
self.element = element
214229
self.snapshot = snapshot
230+
self.contentTranslation = contentTranslation
215231
}
216232

217233
/// Reads a handle that may name an input the app has removed; nil when it names none or the read
218234
/// raised. The snapshot of a missing element throws only after XCTest waits about two seconds for
219235
/// a match, so existence is read first.
220-
init?(reading element: XCUIElement) {
236+
init?(reading element: XCUIElement, contentTranslation: TextEntryContentTranslation) {
221237
var exists = false
222238
var snapshot: XCUIElementSnapshot?
223239
if let exception = RunnerObjCExceptionCatcher.catchException({
@@ -228,15 +244,15 @@ extension RunnerTests {
228244
return nil
229245
}
230246
guard exists else { return nil }
231-
self.init(element: element, snapshot: snapshot)
247+
self.init(element: element, snapshot: snapshot, contentTranslation: contentTranslation)
232248
}
233249

234250
/// Reads a handle its query has just listed, inside the caller's exception containment, so the
235251
/// snapshot is read first and existence only when the snapshot fails; nil when it names no element.
236-
init?(readingListed element: XCUIElement) {
252+
init?(readingListed element: XCUIElement, contentTranslation: TextEntryContentTranslation) {
237253
let snapshot = try? element.snapshot()
238254
guard snapshot != nil || element.exists else { return nil }
239-
self.init(element: element, snapshot: snapshot)
255+
self.init(element: element, snapshot: snapshot, contentTranslation: contentTranslation)
240256
}
241257

242258
var identity: TextEntryInputIdentity? {
@@ -249,6 +265,18 @@ extension RunnerTests {
249265
}
250266
return element
251267
}
268+
269+
/// The input's screen box. Nil without a snapshot, because the live `element.frame` of a handle
270+
/// that stopped answering records a failure (#3060), and nil for an `.unknown` vector, because a
271+
/// window-local point must not dispatch as a screen point (#3899). Point dispatch refuses nil.
272+
var frameOnScreen: CGRect? {
273+
guard let frame = snapshot?.frame else { return nil }
274+
switch contentTranslation {
275+
case .known(let offset): return frame.offsetBy(dx: offset.width, dy: offset.height)
276+
case .none: return frame
277+
case .unknown: return nil
278+
}
279+
}
252280
}
253281

254282
/// A text entry target and its input as last read. `input()` hands the last read out again until
@@ -494,7 +522,8 @@ extension RunnerTests {
494522
return resolveUnboundTextEntryInput(app: app, target: target)
495523
}
496524
func boundInput(_ candidate: XCUIElement) -> ResolvedTextEntryInput? {
497-
guard let input = ResolvedTextEntryInput(reading: candidate), input.identity == boundIdentity else { return nil }
525+
guard let input = ResolvedTextEntryInput(reading: candidate, contentTranslation: target.contentTranslation),
526+
input.identity == boundIdentity else { return nil }
498527
return input
499528
}
500529
if target.prefersFocusedElement, let focused = focusedTextInput(app: app), let input = boundInput(focused) {
@@ -503,12 +532,13 @@ extension RunnerTests {
503532
if let element = target.element, let input = boundInput(element) {
504533
return input
505534
}
506-
if let input = target.inputAtRefreshPoint, let read = textInputIfStillResolves(input) {
535+
if let input = target.inputAtRefreshPoint,
536+
let read = textInputIfStillResolves(input, contentTranslation: target.contentTranslation) {
507537
return read
508538
}
509539
if target.inputAtRefreshPoint == nil,
510540
let refreshPoint = target.refreshPoint,
511-
case .matches(let candidates) = probeTextInputs(app: app, point: refreshPoint),
541+
case .matches(let candidates) = probeTextInputs(app: app, point: refreshPoint, contentTranslation: target.contentTranslation),
512542
let match = candidates.first(where: { $0.identity == boundIdentity }) {
513543
return match
514544
}
@@ -543,7 +573,7 @@ extension RunnerTests {
543573
guard let refreshPoint = target.refreshPoint else {
544574
return true
545575
}
546-
switch probeTextInputs(app: app, point: refreshPoint) {
576+
switch probeTextInputs(app: app, point: refreshPoint, contentTranslation: target.contentTranslation) {
547577
case .absent:
548578
return true
549579
case .matches(let candidates):
@@ -577,17 +607,20 @@ extension RunnerTests {
577607

578608
/// Resolution for a target not bound to an input yet: the first element that exists.
579609
private func resolveUnboundTextEntryInput(app: XCUIApplication, target: TextEntryTarget) -> ResolvedTextEntryInput? {
580-
if target.prefersFocusedElement, let focused = focusedTextInput(app: app).flatMap(ResolvedTextEntryInput.init(reading:)) {
610+
let read = { (element: XCUIElement) in
611+
ResolvedTextEntryInput(reading: element, contentTranslation: target.contentTranslation)
612+
}
613+
if target.prefersFocusedElement, let focused = focusedTextInput(app: app).flatMap(read) {
581614
return focused
582615
}
583-
if let element = target.element, let input = ResolvedTextEntryInput(reading: element) {
616+
if let element = target.element, let input = read(element) {
584617
return input
585618
}
586619
if let refreshPoint = target.refreshPoint,
587-
let refreshed = textInputAt(app: app, x: refreshPoint.x, y: refreshPoint.y) {
620+
let refreshed = textInputAt(app: app, x: refreshPoint.x, y: refreshPoint.y, contentTranslation: target.contentTranslation) {
588621
return refreshed
589622
}
590-
return focusedTextInput(app: app).flatMap(ResolvedTextEntryInput.init(reading:))
623+
return focusedTextInput(app: app).flatMap(read)
591624
}
592625

593626
private func moveCaretToEnd(app: XCUIApplication, input: ResolvedTextEntryInput) {
@@ -606,7 +639,7 @@ extension RunnerTests {
606639
// zero-size one (how a departed handle reads), there is no point to dispatch — an edge-tap
607640
// from a frame the field moved after records a failure or navigates away, the shape
608641
// `clearTextInput`'s comment refuses — so degrade to the caller's point-free delete burst.
609-
guard let frame = input.snapshot?.frame, !frame.isEmpty else {
642+
guard let frame = input.frameOnScreen, !frame.isEmpty else {
610643
return
611644
}
612645
_ = tapAt(

0 commit comments

Comments
 (0)