Skip to content

Commit 1154091

Browse files
thymikeeclaude
andcommitted
fix(ios): read the bar once more after the back button confirmation budget
A snapshot of a navigation bar that vanished mid-pop spends about 2 s in XCTest's element-resolution retries before it raises a held issue. On a slow CI simulator that read used up the 1 s confirmation budget, so no read saw the settled screen and back answered back_swipe_unverified after a real pop. One read now always follows the budget, with no deadline, so the screen after the pop is read. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent f443d78 commit 1154091

5 files changed

Lines changed: 53 additions & 23 deletions

File tree

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift‎

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -384,24 +384,27 @@ extension RunnerTests {
384384
}
385385
}
386386

387-
/// Whether a later read of the bar differs from `before`. Reads start only until `deadlineMs`; a
388-
/// read in flight then finishes, or ends at XCTest's own query timeout, which is a hung query. A
389-
/// read that fails proves nothing, so it never confirms.
387+
/// Whether a later read of the bar differs from `before`. Reads start until `deadlineMs`, and one
388+
/// final read starts after it and runs to the end, so the screen after the pop is always read: a
389+
/// read of a bar that vanishes mid-pop can spend the whole budget in XCTest's retries. A read in
390+
/// flight ends at XCTest's own query timeout, which is a hung query. A read that fails proves
391+
/// nothing, so it never confirms.
390392
@MainActor
391393
private func navigationBarBackConfirmation(
392394
app: XCUIApplication,
393395
from before: NavigationBarBackState,
394396
deadlineMs: Double
395397
) -> NavigationBarBackConfirmation {
396398
while true {
397-
switch readNavigationBarBackState(app: app, phase: .confirmation, deadlineMs: deadlineMs) {
399+
let isFinalRead = currentUptimeMs() >= deadlineMs
400+
let readDeadlineMs = isFinalRead ? Double.infinity : deadlineMs
401+
switch readNavigationBarBackState(app: app, phase: .confirmation, deadlineMs: readDeadlineMs) {
398402
case .read(let after) where Self.navigationBarBackPopped(from: before, to: after): return .changed
399403
case .hung: return .hung
400404
case .read, .unreadable: break
401405
}
402-
let remainingMs = deadlineMs - currentUptimeMs()
403-
if remainingMs <= 0 { return .unchanged }
404-
sleepFor(min(Self.navigationBarBackReadInterval, remainingMs / 1000))
406+
if isFinalRead { return .unchanged }
407+
sleepFor(min(Self.navigationBarBackReadInterval, max(0, deadlineMs - currentUptimeMs()) / 1000))
405408
}
406409
}
407410

@@ -434,8 +437,8 @@ extension RunnerTests {
434437
let held = holdingRecordedIssues {
435438
value = read()
436439
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
437-
if let injected = navigationBarBackReadIssueForTesting, injected.phase == phase {
438-
record(XCTIssue(type: .assertionFailure, compactDescription: injected.description))
440+
if let description = navigationBarBackReadForTesting?(phase) {
441+
record(XCTIssue(type: .assertionFailure, compactDescription: description))
439442
}
440443
#endif
441444
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests.swift‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -219,9 +219,10 @@ final class RunnerTests: XCTestCase {
219219
var litPanelWindowWalkIssueForTesting = false
220220

221221
#if os(iOS)
222-
/// Records an XCTest issue with this description during every navigation bar read of this phase
223-
/// of iOS `back`, as a back button that vanishes mid-transition, or a hung query, does.
224-
var navigationBarBackReadIssueForTesting: (phase: NavigationBarBackReadPhase, description: String)?
222+
/// Runs at the end of every navigation bar read of iOS `back`, and records an XCTest issue with
223+
/// the description it returns, as a bar that vanishes mid-transition, or a hung query, does. It
224+
/// can also sleep, as a read on a slow host does.
225+
var navigationBarBackReadForTesting: ((NavigationBarBackReadPhase) -> String?)?
225226

226227
/// What the last iOS `back` saw of the navigation bar back button, so a test can tell the button
227228
/// path from the swipe when both navigate.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+NavigationTests.swift‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,8 @@ extension RunnerTests {
133133
func testInAppBackWithAnIssueInTheButtonLookupAnswersFromTheSwipe() throws {
134134
_ = try launchBackNavigationFixture(["--agent-device-back-navigation-bar"])
135135
defer { endBackFixture() }
136-
navigationBarBackReadIssueForTesting = (.lookup, Self.vanishedBackButtonIssue)
137-
defer { navigationBarBackReadIssueForTesting = nil }
136+
navigationBarBackReadForTesting = { $0 == .lookup ? Self.vanishedBackButtonIssue : nil }
137+
defer { navigationBarBackReadForTesting = nil }
138138

139139
let response = try execute(command: try runnerCommandFixture(
140140
#"{"command":"backInApp","commandId":"back-lookup-issue","appBundleId":"com.callstack.agentdevice.runner"}"#
@@ -152,18 +152,18 @@ extension RunnerTests {
152152
func testInAppBackWithAHungQueryInTheButtonLookupSendsNothing() throws {
153153
_ = try launchBackNavigationFixture(["--agent-device-back-navigation-bar"])
154154
defer { endBackFixture() }
155-
defer { navigationBarBackReadIssueForTesting = nil }
155+
defer { navigationBarBackReadForTesting = nil }
156156

157157
let outcome = try expectingHungQueryRecorded {
158-
navigationBarBackReadIssueForTesting = (.lookup, Self.hungQueryIssue)
158+
navigationBarBackReadForTesting = { $0 == .lookup ? Self.hungQueryIssue : nil }
159159
return try performInAppBack(app: app)
160160
}
161161

162162
guard case .refused(let payload) = outcome else { return XCTFail("a hung lookup must send nothing") }
163163
XCTAssertEqual(payload.code, RunnerWireErrorCode.backSwipeNotDispatched)
164164
XCTAssertEqual(lastNavigationBarBackTraceForTesting?.found, "hung")
165165
XCTAssertEqual(lastNavigationBarBackTraceForTesting?.pressed, false)
166-
navigationBarBackReadIssueForTesting = nil
166+
navigationBarBackReadForTesting = nil
167167
XCTAssertTrue(app.navigationBars["Detail"].exists, "nothing popped the detail screen")
168168
}
169169

@@ -173,10 +173,10 @@ extension RunnerTests {
173173
func testInAppBackWithAHungQueryInTheConfirmationAnswersUnverified() throws {
174174
_ = try launchBackNavigationFixture(["--agent-device-back-navigation-bar"])
175175
defer { endBackFixture() }
176-
defer { navigationBarBackReadIssueForTesting = nil }
176+
defer { navigationBarBackReadForTesting = nil }
177177

178178
let outcome = try expectingHungQueryRecorded {
179-
navigationBarBackReadIssueForTesting = (.confirmation, Self.hungQueryIssue)
179+
navigationBarBackReadForTesting = { $0 == .confirmation ? Self.hungQueryIssue : nil }
180180
return try performInAppBack(app: app)
181181
}
182182

@@ -223,20 +223,44 @@ extension RunnerTests {
223223
func testInAppBackWithAnIssueInTheConfirmationAnswersUnverified() throws {
224224
let status = try launchBackNavigationFixture(["--agent-device-back-navigation-bar"])
225225
defer { endBackFixture() }
226-
navigationBarBackReadIssueForTesting = (.confirmation, Self.vanishedBackButtonIssue)
227-
defer { navigationBarBackReadIssueForTesting = nil }
226+
navigationBarBackReadForTesting = { $0 == .confirmation ? Self.vanishedBackButtonIssue : nil }
227+
defer { navigationBarBackReadForTesting = nil }
228228

229229
let response = try execute(command: try runnerCommandFixture(
230230
#"{"command":"backInApp","commandId":"back-confirmation-issue","appBundleId":"com.callstack.agentdevice.runner"}"#
231231
))
232232

233233
XCTAssertEqual(response.error?.code, RunnerWireErrorCode.backSwipeUnverified)
234234
XCTAssertEqual(lastNavigationBarBackTraceForTesting?.pressed, true)
235-
navigationBarBackReadIssueForTesting = nil
235+
navigationBarBackReadForTesting = nil
236236
let popped = XCTNSPredicateExpectation(predicate: NSPredicate(format: "label == %@", "Back pressed"), object: status)
237237
XCTAssertEqual(XCTWaiter().wait(for: [popped], timeout: 5), .completed, "the tap itself was delivered")
238238
}
239239

240+
/// A slow host can spend the whole confirmation budget on reads of the bar that is vanishing mid-pop,
241+
/// each one an issue that proves nothing. The read after the budget sees the settled screen, so
242+
/// the pop is confirmed, not left unverified.
243+
@MainActor
244+
func testInAppBackConfirmsAPopWhoseReadsInTheBudgetAllFailOnASlowHost() throws {
245+
_ = try launchBackNavigationFixture(["--agent-device-back-navigation-bar"])
246+
defer { endBackFixture() }
247+
var confirmationReads = 0
248+
navigationBarBackReadForTesting = { phase in
249+
guard phase == .confirmation else { return nil }
250+
confirmationReads += 1
251+
guard confirmationReads <= 2 else { return nil }
252+
Thread.sleep(forTimeInterval: 0.6)
253+
return Self.vanishedBackButtonIssue
254+
}
255+
defer { navigationBarBackReadForTesting = nil }
256+
257+
guard case .performed = try performInAppBack(app: app) else {
258+
return XCTFail("a pop read after the budget must read as performed")
259+
}
260+
XCTAssertEqual(confirmationReads, 3, "both reads in the budget failed, and one read followed it")
261+
XCTAssertEqual(lastNavigationBarBackTraceForTesting?.confirmed, "yes")
262+
}
263+
240264
/// A bar button titled "Back" is not the system back button, so back never presses it: it swipes,
241265
/// and the swipe that changes nothing reports so.
242266
@MainActor

‎apple/runner/RUNNER_PROTOCOL.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,8 @@ and bar count, such as one between two screens whose back buttons have the same
155155
back button whose title updates itself, misread as a navigation. No change, or reads that fail, answer `BACK_SWIPE_UNVERIFIED`; a delivered tap is
156156
never followed by the swipe. Before and after the tap, reads start for up to 1 s each, checked
157157
before each bar's snapshot; a query in flight finishes or ends at XCTest's own query timeout.
158+
After the tap, one more read follows the 1 s and runs to its end, so the screen after the pop is
159+
always read: a snapshot of a bar that vanished mid-pop can spend the 1 s in XCTest's retries.
158160

159161
Every bar read holds its XCTest issues. A held issue means the read proved nothing. A hung query
160162
("Timed out while evaluating UI query") is recorded again, and no further query follows it: before

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ iOS Simulator launch behavior:
124124
Back, orientation, and hardware buttons:
125125

126126
- `back` uses app-owned back navigation: the back keyevent on Android, the visible back control on macOS, and the Siri Remote Menu button on tvOS. `--in-app` names this default explicitly.
127-
- On iOS, `back` first looks for the navigation bar's system back button (a button with the identifier `BackButton` inside a navigation bar), read fresh in the same call. It presses that button only when exactly one live, hittable one is found, the app's accessibility tree is readable (it is not read when XCTest could not query it earlier in the session), and two reads of the navigation bars at least 0.05 s apart agree, so a screen that is still being pushed is not taken for the screen to go back from. After the tap, `back` reads the navigation bars again, about every 0.05 s, and succeeds once one of these changes: the number of navigation bars, the number of back buttons (for example, the button is gone at the root screen), or a back button's title. If none changes, it reports `back_swipe_unverified` and does not swipe. A bar's title alone never counts, because a title can update itself, such as a countdown timer. A back button that only moves never counts either, because a tap that goes nowhere can still move it, such as when a large title collapses. So a pop that leaves the back button as it was, for example between two screens whose back buttons show the same title (such as an app that titles every back button "Back"), also reports `back_swipe_unverified`. A back button whose own title updates itself can still be misread as a navigation. Before and after the tap, reads start for up to 1 s each; a read that is still running at that point finishes, or ends at XCTest's own query timeout. When an XCTest query hangs, `back` sends nothing more: before the tap it reports `back_swipe_not_dispatched`, and after the tap `back_swipe_unverified`. Only the system back button counts: a custom bar button titled "Back", or a "Back" button in a toolbar, is never pressed. With no such button, with two or more (such as a sheet over a stack), when the tree cannot be read, when the bars do not hold still within 1 s, or when the tap is refused before it is sent, `back` is the left-edge back swipe, with or without `--in-app` or `--system`. When XCTest cannot resolve the app's window, the swipe samples the device's only screen: on a single-screen iPhone, or on an iPad only while the app provably fills its screen; otherwise it refuses with `APP_SCREEN_UNRESOLVED`. The swipe succeeds only when the app's screen, drawn as a 48×96 grayscale grid with its top 7 rows left out (in portrait, the status bar and Dynamic Island band, whether the status bar is shown or hidden; in landscape the iPhone status bar is hidden and the Dynamic Island sits at a side edge, outside those rows), differs in more than 1% of the remaining cells by more than 24 of 256 levels. A small spinner usually stays under that; a larger animation can still count as a change. The change counted is the one between the sample taken immediately before the swipe and the one taken about 0.1 s after it, so a screen that changes on its own inside that gap, after it looked still, can be misread as a back navigation; verify with `snapshot` when the screen has auto-advancing content. Before it swipes, it waits for the screen to stop changing, with a 0.05 s pause between captures (each capture adds its own time), within a settle budget of 2 s that extends on a host whose captures are slow enough that a still screen needs more time. The screen must stay still for the contact allowance (0.2 s on a simulator, 0.4 s on a physical device), the 0.1 s wait after the swipe, and the slowest capture: about 0.4 s in all on a simulator. A screen that repaints on its own more often than that never settles, and `back` reports `back_swipe_unverified`; the runner's tests check this with a screen that repaints every 0.3 s. A still screen usually settles in about 0.45 s on a simulator, and in about 1 s on a physical iPhone, based on the 0.4 s capture measured on one iPhone 17 Pro with iOS 27.0.1 ([PR #4460 evidence](https://github.com/callstack/agent-device/pull/4460#issuecomment-6828546541)).
127+
- On iOS, `back` first looks for the navigation bar's system back button (a button with the identifier `BackButton` inside a navigation bar), read fresh in the same call. It presses that button only when exactly one live, hittable one is found, the app's accessibility tree is readable (it is not read when XCTest could not query it earlier in the session), and two reads of the navigation bars at least 0.05 s apart agree, so a screen that is still being pushed is not taken for the screen to go back from. After the tap, `back` reads the navigation bars again, about every 0.05 s, and succeeds once one of these changes: the number of navigation bars, the number of back buttons (for example, the button is gone at the root screen), or a back button's title. If none changes, it reports `back_swipe_unverified` and does not swipe. A bar's title alone never counts, because a title can update itself, such as a countdown timer. A back button that only moves never counts either, because a tap that goes nowhere can still move it, such as when a large title collapses. So a pop that leaves the back button as it was, for example between two screens whose back buttons show the same title (such as an app that titles every back button "Back"), also reports `back_swipe_unverified`. A back button whose own title updates itself can still be misread as a navigation. Before and after the tap, reads start for up to 1 s each; a read that is still running at that point finishes, or ends at XCTest's own query timeout. After the tap, one more read follows that 1 s, so the screen after the pop is always read, even on a slow host. When an XCTest query hangs, `back` sends nothing more: before the tap it reports `back_swipe_not_dispatched`, and after the tap `back_swipe_unverified`. Only the system back button counts: a custom bar button titled "Back", or a "Back" button in a toolbar, is never pressed. With no such button, with two or more (such as a sheet over a stack), when the tree cannot be read, when the bars do not hold still within 1 s, or when the tap is refused before it is sent, `back` is the left-edge back swipe, with or without `--in-app` or `--system`. When XCTest cannot resolve the app's window, the swipe samples the device's only screen: on a single-screen iPhone, or on an iPad only while the app provably fills its screen; otherwise it refuses with `APP_SCREEN_UNRESOLVED`. The swipe succeeds only when the app's screen, drawn as a 48×96 grayscale grid with its top 7 rows left out (in portrait, the status bar and Dynamic Island band, whether the status bar is shown or hidden; in landscape the iPhone status bar is hidden and the Dynamic Island sits at a side edge, outside those rows), differs in more than 1% of the remaining cells by more than 24 of 256 levels. A small spinner usually stays under that; a larger animation can still count as a change. The change counted is the one between the sample taken immediately before the swipe and the one taken about 0.1 s after it, so a screen that changes on its own inside that gap, after it looked still, can be misread as a back navigation; verify with `snapshot` when the screen has auto-advancing content. Before it swipes, it waits for the screen to stop changing, with a 0.05 s pause between captures (each capture adds its own time), within a settle budget of 2 s that extends on a host whose captures are slow enough that a still screen needs more time. The screen must stay still for the contact allowance (0.2 s on a simulator, 0.4 s on a physical device), the 0.1 s wait after the swipe, and the slowest capture: about 0.4 s in all on a simulator. A screen that repaints on its own more often than that never settles, and `back` reports `back_swipe_unverified`; the runner's tests check this with a screen that repaints every 0.3 s. A still screen usually settles in about 0.45 s on a simulator, and in about 1 s on a physical iPhone, based on the 0.4 s capture measured on one iPhone 17 Pro with iOS 27.0.1 ([PR #4460 evidence](https://github.com/callstack/agent-device/pull/4460#issuecomment-6828546541)).
128128
- `reason: back_swipe_no_change`: the swipe was sent and the screen did not change. Press the visible back or close control.
129129
- `reason: back_swipe_unverified`: the back button was pressed but the navigation bar's back buttons did not change within 1 s or an XCTest query hung after the tap, or the swipe may or may not have navigated, because the screen kept changing before the swipe, the swipe took longer than the screen had been still, or the screen could not be sampled after it. Check with `snapshot`. A real pop on a stack of three or more screens also answers this after the full 1 s when the outgoing and incoming back buttons share a label.
130130
- `reason: back_swipe_not_dispatched`: the swipe was refused before any touch went down, or an XCTest query hung before the back button tap, so nothing reached the device.

0 commit comments

Comments
 (0)