Skip to content

Commit a3d69c2

Browse files
thymikeeclaude
andcommitted
fix(ios): send device back as a hold-free 200 ms fling, and select the General cell by id in the physical replay
The XCTest press-and-drag held at the edge for at least 0.1 s and moved at 500 px/s, which iOS can cancel as a pop; a synthesized 200 ms fling to 0.9 of the width popped 10 of 10 on an iPhone 17 Pro. The fling is resolved before the just-before sample and reads no tree. The device contact allowance drops to 0.4 s; the still span still grows with the slowest capture, so a 0.4 s device capture is covered on both sides of the gap check. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 63959d7 commit a3d69c2

7 files changed

Lines changed: 58 additions & 92 deletions

File tree

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift‎

Lines changed: 13 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -134,13 +134,13 @@ extension RunnerTests {
134134

135135
/// The left-edge back swipe across the context's screen route, resolved up to its contact. A
136136
/// simulator sends it as one edge-masked touch at `backSwipeEndpoints`; a device has no such touch
137-
/// and takes `prepareDeviceBackDrag`.
137+
/// and takes `prepareDeviceBackFling`.
138138
@MainActor
139139
func prepareBackGesture(app: XCUIApplication, context: SynthesizedCoordinateContext) throws -> PreparedTouch {
140140
#if targetEnvironment(simulator)
141141
return try prepareScreenEdgeBackSwipe(app: app, surface: context.geometry.windowOnScreen.rect)
142142
#else
143-
return try prepareDeviceBackDrag(app: app, context: context)
143+
return try prepareDeviceBackFling(app: app, context: context)
144144
#endif
145145
}
146146

@@ -172,31 +172,21 @@ extension RunnerTests {
172172
)
173173
}
174174

175-
/// The back drag a device takes, from 2 pt inside the leading edge to 0.8 of the width at
176-
/// mid-height. Driven on a simulator, that XCTest drag popped Settings > General and a React Native
177-
/// header-less screen 9 times in 10 each, where one starting 20 pt in popped 2 in 6. XCTest cannot
178-
/// anchor a coordinate on a tree it cannot serialize (#4221), so there the same path is synthesized
179-
/// on the context's booked route instead, without reading the tree. The window anchor and the plan
180-
/// are resolved here, so the returned dispatch runs only the drag.
175+
/// The back gesture a device takes: a hold-free 200 ms fling from 2 pt inside the leading edge to
176+
/// 0.9 of the width at mid-height, synthesized on the context's screen route without reading the
177+
/// tree (#4221). On an iPhone 17 Pro it popped Settings > General and a header-less React Native
178+
/// screen 5 times in 5 each, where XCTest's press-and-drag, which holds at the edge for at least
179+
/// 0.1 s and then moves at 500 px/s, missed 2 in 10. The plan is resolved here, so the returned
180+
/// dispatch runs only the fling.
181+
static let deviceBackFlingDurationMs = 200.0
182+
181183
@MainActor
182-
func prepareDeviceBackDrag(app: XCUIApplication, context: SynthesizedCoordinateContext) throws -> PreparedTouch {
183-
guard context.treeUnreadable else {
184-
let frame = try normalizedOffsetAnchor(app: app).frame
185-
let appOrigin = app.frame.origin
186-
// Offsets from the app's origin, read now, so the drag does not query the windows again.
187-
let origin = app.coordinate(withNormalizedOffset: .zero)
188-
let start = origin.withOffset(CGVector(dx: frame.minX - appOrigin.x + 2, dy: frame.midY - appOrigin.y))
189-
let end = origin.withOffset(CGVector(dx: frame.minX - appOrigin.x + frame.width * 0.8, dy: frame.midY - appOrigin.y))
190-
return .ready {
191-
start.press(forDuration: 0.05, thenDragTo: end)
192-
return .sent
193-
}
194-
}
184+
func prepareDeviceBackFling(app: XCUIApplication, context: SynthesizedCoordinateContext) throws -> PreparedTouch {
195185
let surface = context.geometry.windowOnScreen.rect
196186
let plan = Self.backFlingPlan(
197187
from: CGPoint(x: surface.minX + 2, y: surface.midY),
198-
to: CGPoint(x: surface.minX + surface.width * 0.8, y: surface.midY),
199-
durationMs: 300,
188+
to: CGPoint(x: surface.minX + surface.width * 0.9, y: surface.midY),
189+
durationMs: Self.deviceBackFlingDurationMs,
200190
across: surface,
201191
contactOrigin: nil
202192
)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@ import XCTest
44
extension RunnerTests {
55
static let backSwipeVerificationDelay: TimeInterval = 0.25
66
/// The time the contact itself is allowed in the gap from the sample taken just before the swipe to
7-
/// the one after it. A simulator's edge touch is resolved before that sample and runs a 100 ms
8-
/// contact; a device's XCTest drag resolves its coordinates as it drags and takes longer.
7+
/// the one after it. Both are resolved before that sample: a simulator's edge touch runs a 100 ms
8+
/// contact, and a device's synthesized fling runs 200 ms.
99
#if targetEnvironment(simulator)
1010
static let backGestureAllowanceMs = 200.0
1111
#else
12-
static let backGestureAllowanceMs = 650.0
12+
static let backGestureAllowanceMs = 400.0
1313
#endif
1414
static let backSwipeSettleSampleBudget = 8
1515

@@ -360,7 +360,8 @@ extension RunnerTests {
360360
/// Whether the screen had been still for at least as long as the swipe's gap: the time from the
361361
/// sample taken just before the swipe to the one after it must not exceed the time the screen had
362362
/// already agreed with itself, or a self-repainting screen slower than the window could read as the
363-
/// swipe's effect.
363+
/// swipe's effect. All three times are taken when a capture completed, so a slow capture counts on
364+
/// both sides.
364365
static func backSwipeGapIsCovered(stableSinceMs: Double, justBeforeMs: Double, afterMs: Double) -> Bool {
365366
afterMs - justBeforeMs <= justBeforeMs - stableSinceMs
366367
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+NavigationTests.swift‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,26 @@ extension RunnerTests {
128128
XCTAssertGreaterThanOrEqual(slowCaptures, 4, "settle, just-before, and after samples were all slowed")
129129
}
130130

131+
/// A device's sample cost: every capture takes 0.4 s and the contact runs 0.2 s, as a physical
132+
/// iPhone's capture and fling do. The still span grows with the capture, so the pop reads as
133+
/// performed.
134+
@MainActor
135+
func testInAppBackWithSlowCapturesAndAFlingLongContactIsPerformed() throws {
136+
let status = try launchBackFixture(["--agent-device-back-regression"])
137+
defer { endBackFixture() }
138+
let restoreCapture = try slowScreenshots(by: 0.4) {}
139+
defer { restoreCapture() }
140+
let restoreContact = try slowEdgeTouchContact(by: 0.1)
141+
defer { restoreContact() }
142+
143+
let outcome = try performInAppBack(app: app)
144+
145+
guard case .performed = outcome else {
146+
return XCTFail("0.4 s captures and a 0.2 s contact on a still screen must read as performed, got \(outcome)")
147+
}
148+
XCTAssertEqual(status.label, "Back performed")
149+
}
150+
131151
/// The guard still holds: a contact that outruns the time the screen had been still reports an
132152
/// unknown outcome even though the screen changed.
133153
@MainActor

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInteractionTests.swift‎

Lines changed: 8 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -235,11 +235,10 @@ extension RunnerTests {
235235
}
236236
}
237237

238-
/// A device's back drag presses an XCTest coordinate only where the tree can anchor it. On a tree
239-
/// XCTest cannot serialize it synthesizes the same path on the booked route: no tree anchor, no
240-
/// coordinate press, which would raise and end the runner (#4221).
238+
/// A device's back fling is synthesized on the booked route, so even on a tree XCTest cannot
239+
/// serialize it presses no XCTest coordinate, which would raise and end the runner (#4221).
241240
@MainActor
242-
func testDeviceBackDragOnAnUnserializableTreeIsSynthesizedWithoutAnXCTestPress() throws {
241+
func testDeviceBackFlingOnAnUnserializableTreeIsSynthesizedWithoutAnXCTestPress() throws {
243242
app.launch()
244243
mainOwned.app = app
245244
let restoreXCTest = try simulateUnserializableAppTree()
@@ -252,66 +251,20 @@ extension RunnerTests {
252251
try synthesizedCoordinateContext(app: app),
253252
"the sole-screen route books a context on an unserializable tree"
254253
)
255-
guard context.treeUnreadable else {
256-
return XCTFail("the drag would take the XCTest coordinate path on a readable tree")
257-
}
254+
XCTAssertTrue(context.treeUnreadable)
258255

259-
guard case .ready(let drag) = try prepareDeviceBackDrag(app: app, context: context) else {
260-
return XCTFail("the drag must resolve on the booked route")
256+
guard case .ready(let fling) = try prepareDeviceBackFling(app: app, context: context) else {
257+
return XCTFail("the fling must resolve on the booked route")
261258
}
262-
let dispatch = try drag()
259+
let dispatch = try fling()
263260

264261
guard case .sent = dispatch else {
265-
return XCTFail("the drag must be synthesized on the booked route: \(dispatch)")
262+
return XCTFail("the fling must be synthesized on the booked route: \(dispatch)")
266263
}
267264
XCTAssertEqual(RunnerCoordinatePressCounter.presses, 0)
268265
XCTAssertEqual(RunnerCoordinatePressCounter.drags, 0, "the XCTest press-and-drag never ran")
269266
}
270267

271-
/// On a readable tree the device back drag reads its window anchor while it is prepared, so the
272-
/// drag itself, which runs after the just-before sample, queries no windows.
273-
@MainActor
274-
func testDeviceBackDragOnAReadableTreeQueriesNoWindowsWhileItDrags() throws {
275-
app.launch()
276-
mainOwned.app = app
277-
defer {
278-
invalidateCachedTarget(reason: "unit_test_cleanup")
279-
app.terminate()
280-
}
281-
let context = try XCTUnwrap(try synthesizedCoordinateContext(app: app))
282-
guard !context.treeUnreadable else { return XCTFail("the fixture tree must be readable") }
283-
guard case .ready(let drag) = try prepareDeviceBackDrag(app: app, context: context) else {
284-
return XCTFail("a readable tree resolves the XCTest drag")
285-
}
286-
let dragMethod = try XCTUnwrap(class_getInstanceMethod(XCUICoordinate.self, #selector(XCUICoordinate.press(forDuration:thenDragTo:))))
287-
let countedDrag = try XCTUnwrap(class_getInstanceMethod(
288-
RunnerCoordinatePressCounter.self,
289-
#selector(RunnerCoordinatePressCounter.press(forDuration:thenDragTo:))
290-
))
291-
let originalDrag = method_setImplementation(dragMethod, method_getImplementation(countedDrag))
292-
defer { method_setImplementation(dragMethod, originalDrag) }
293-
RunnerCoordinatePressCounter.drags = 0
294-
let selector = NSSelectorFromString("allElementsBoundByIndex")
295-
let query = try XCTUnwrap(class_getInstanceMethod(XCUIElementQuery.self, selector))
296-
let originalQuery = method_getImplementation(query)
297-
var queries = 0
298-
let countedQuery = imp_implementationWithBlock({ (target: AnyObject) -> NSArray in
299-
queries += 1
300-
typealias Query = @convention(c) (AnyObject, Selector) -> NSArray
301-
return unsafeBitCast(originalQuery, to: Query.self)(target, selector)
302-
} as @convention(block) (AnyObject) -> NSArray)
303-
method_setImplementation(query, countedQuery)
304-
defer {
305-
method_setImplementation(query, originalQuery)
306-
imp_removeBlock(countedQuery)
307-
}
308-
309-
guard case .sent = try drag() else { return XCTFail("the drag must be sent") }
310-
311-
XCTAssertEqual(RunnerCoordinatePressCounter.drags, 1)
312-
XCTAssertEqual(queries, 0, "no windows query runs inside the gap")
313-
}
314-
315268
/// A synthesis that fails before its record is handed to XCTest touched nothing; one that fails
316269
/// after may have run.
317270
func testSynthesisFailureMapsByWhetherTheRecordWasDispatched() {

‎apple/runner/RUNNER_PROTOCOL.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -153,12 +153,12 @@ contact, and one more sample, taken right before the swipe, must still match the
153153
| Platform | Resolved before the last sample | Left in the gap | Contact allowance |
154154
| --- | --- | --- | --- |
155155
| Simulator | the edge-touch plan, its display, and the HID touchscreen sender | the 100 ms edge contact | 0.2 s |
156-
| Device | the window anchor of the XCTest drag, or on a tree XCTest cannot serialize, the synthesized drag plan | the drag (a 0.05 s press then the move, or a 300 ms synthesized drag) | 0.65 s |
156+
| Device | the synthesized fling plan (2 pt inside the leading edge to 0.9 of the width, no hold), which reads no tree | the 200 ms fling | 0.4 s |
157157

158158
Settling takes at most 8 samples, so the wait before the swipe is at most 1.75 s of delays plus
159-
eight captures, then the preparation and the last sample. With the 0.1 s captures measured on a
160-
local simulator that is about 2.6 s; a slower capture adds 8 times its extra time. Usually the
161-
screen settles sooner: a still simulator screen after 3 samples, a still device screen after 3 or 4.
159+
eight captures, then the preparation and the last sample: about 2.6 s with the 0.1 s captures of a
160+
local simulator, and about 5 s with the 0.4 s captures of a physical iPhone. Usually a still screen
161+
settles after 3 samples.
162162

163163
Only a change from the last sample to the sample after the swipe counts, and only when the time
164164
between those two does not exceed the time the screen had been still; otherwise the result is
Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
# Dogfood iOS physical device lifecycle through the replay suite runner.
2+
# The General cell is selected by its identifier: on iOS 27 a label match can land on the
3+
# floating Search bar that overlaps the list.
24
context platform=ios target=mobile
35
open com.apple.Preferences --relaunch
46
snapshot
5-
scroll down --until "role=cell label=General || role=button label=General" --max-passes 5
6-
click "role=cell label=General || role=button label=General"
7+
scroll down --until "id=com.apple.settings.general" --max-passes 5
8+
click "id=com.apple.settings.general"
79
wait "label=About || label=\"Software Update\"" 5000
810
back
9-
wait "role=cell label=General || role=button label=General" 5000
10-
is exists "role=cell label=General || role=button label=General"
11+
wait "id=com.apple.settings.general" 5000
12+
is exists "id=com.apple.settings.general"

‎website/docs/docs/commands.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,12 +124,12 @@ iOS Simulator launch behavior:
124124
Back, orientation, and hardware buttons:
125125

126126
- `back` uses app-owned back navigation: the back keyevent on Android, the visible back control on macOS, and the Siri Remote Menu button on tvOS. `--in-app` names this default explicitly.
127-
- On iOS, `back` is the left-edge back swipe, with or without `--in-app` or `--system`. It succeeds only when the app's screen, drawn as a 48×96 grayscale grid with its top 7 rows left out (in portrait, the status bar and Dynamic Island band, whether the status bar is shown or hidden; in landscape the iPhone status bar is hidden and the Dynamic Island sits at a side edge, outside those rows), differs in more than 1% of the remaining cells by more than 24 of 256 levels. A small spinner usually stays under that; a larger animation can still count as a change. The change counted is the one between the sample taken immediately before the swipe and the one taken about 0.25 s after it, so a screen that changes on its own inside that gap, after it looked still, can be misread as a back navigation; verify with `snapshot` when the screen has auto-advancing content. Before it swipes, it waits for the screen to stop changing: up to 8 samples 0.25 s apart, which is 1.75 s plus eight screen captures, about 2.6 s on a simulator and longer where captures are slower.
127+
- On iOS, `back` is the left-edge back swipe, with or without `--in-app` or `--system`. It succeeds only when the app's screen, drawn as a 48×96 grayscale grid with its top 7 rows left out (in portrait, the status bar and Dynamic Island band, whether the status bar is shown or hidden; in landscape the iPhone status bar is hidden and the Dynamic Island sits at a side edge, outside those rows), differs in more than 1% of the remaining cells by more than 24 of 256 levels. A small spinner usually stays under that; a larger animation can still count as a change. The change counted is the one between the sample taken immediately before the swipe and the one taken about 0.25 s after it, so a screen that changes on its own inside that gap, after it looked still, can be misread as a back navigation; verify with `snapshot` when the screen has auto-advancing content. Before it swipes, it waits for the screen to stop changing: up to 8 samples 0.25 s apart, which is 1.75 s plus eight screen captures: about 2.6 s on a simulator and about 5 s on a physical iPhone.
128128
- `reason: back_swipe_no_change`: the swipe was sent and the screen did not change. Press the visible back or close control.
129129
- `reason: back_swipe_unverified`: the swipe may or may not have navigated, because the screen kept changing before the swipe, the swipe took longer than the screen had been still, or the screen could not be sampled after it. Check with `snapshot`.
130130
- `reason: back_swipe_not_dispatched`: the swipe was refused before any touch went down, so nothing reached the device.
131131
- If the app's screen cannot be captured before the swipe, `back` does not swipe: it refuses with `COMMAND_FAILED` and `details.runnerErrorCode` set to `APP_SCREEN_WINDOW_UNRESOLVED`, `APP_SCREEN_UNRESOLVED`, `APP_SCREEN_CAPTURE_UNRENDERABLE`, or `APP_SCREEN_OFF_LIT_PANEL`. Nothing is sent.
132-
- A simulator sends the swipe as a screen-edge touch. A physical device has no such touch and drags from the leading edge instead; if that drag does not pop the screen, `back` refuses with `back_swipe_no_change`. visionOS has no back gesture and refuses before dispatch with `UNSUPPORTED_OPERATION`.
132+
- A simulator sends the swipe as a screen-edge touch. A physical device has no such touch and sends a quick fling from the leading edge instead; if that fling does not pop the screen, `back` refuses with `back_swipe_no_change`. visionOS has no back gesture and refuses before dispatch with `UNSUPPORTED_OPERATION`.
133133
- An app with a left-edge drawer may open the drawer instead of going back. When a back or close control is visible, press it by its ref.
134134
- `back --system` sends system back input: the Android back keyevent, the iOS back swipe, or the tvOS Siri Remote Menu button. macOS has no system back, so it refuses before dispatch.
135135
- `orientation portrait|portrait-upside-down|landscape-left|landscape-right` works on iOS and Android mobile targets only. On ADB-backed Android it turns auto-rotate off, and the lock persists after the session ends until something turns it back on. A foldable iPhone simulator (iPhone Duo) refuses `portrait-upside-down`, and refuses every rotation in a scoped simulator set (`--ios-simulator-device-set`).

0 commit comments

Comments
 (0)