Skip to content

Commit f87fa1c

Browse files
thymikeeclaude
andcommitted
fix(ios): tap a visible Back control on header-less screens and refuse without a blind tap
In-app back looked for a back control only in the navigation bar and in the header band. A screen with its header hidden (a React Native native-stack screen with headerShown: false) keeps its back control in the content, and React Native exposes a Pressable without a role as an untyped element. Back missed it, tapped the empty top-leading point, saw no change, and refused. The runner now accepts a hittable button or untyped element whose whole label is "Back", anywhere on screen, after the header candidates. The top-leading tap now runs only when XCTest rejected a query or the channel is penalized, as the docs already state. A readable tree that names no back control refuses without tapping whatever sits top-leading. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 799e9c0 commit f87fa1c

5 files changed

Lines changed: 95 additions & 4 deletions

File tree

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerAppFixtures.m‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -579,6 +579,22 @@ - (void)viewDidLoad {
579579
item.leftBarButtonItem = [[UIBarButtonItem alloc] initWithTitle:@"Go back" style:UIBarButtonItemStylePlain target:self action:@selector(navigationBackPressed)];
580580
bar.items = @[item];
581581
[self.view addSubview:bar];
582+
} else if ([NSProcessInfo.processInfo.arguments containsObject:@"--content-back"]) {
583+
// A header-less screen whose back control sits mid-screen as an untyped element, the way
584+
// React Native exposes a Pressable that has no accessibility role.
585+
UIView *back = [[UIView alloc] init];
586+
back.translatesAutoresizingMaskIntoConstraints = NO;
587+
back.isAccessibilityElement = YES;
588+
back.accessibilityLabel = @"Back";
589+
back.backgroundColor = UIColor.systemBlueColor;
590+
[back addGestureRecognizer:[[UITapGestureRecognizer alloc] initWithTarget:self action:@selector(navigationBackPressed)]];
591+
[self.view addSubview:back];
592+
[NSLayoutConstraint activateConstraints:@[
593+
[back.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
594+
[back.centerYAnchor constraintEqualToAnchor:self.view.centerYAnchor constant:120],
595+
[back.widthAnchor constraintEqualToConstant:200],
596+
[back.heightAnchor constraintEqualToConstant:44],
597+
]];
582598
} else {
583599
// The control must sit under the point the runner's top-leading fallback aims at, whose
584600
// bounds are known: x lands at most 44 from the frame's leading edge, y at
@@ -588,7 +604,9 @@ - (void)viewDidLoad {
588604
// whole clamped y range for every inset an iPhone or iPad reports.
589605
UIButton *back = [UIButton buttonWithType:UIButtonTypeSystem];
590606
back.translatesAutoresizingMaskIntoConstraints = NO;
591-
[back setTitle:@"Go back" forState:UIControlStateNormal];
607+
// `--top-leading-menu` names the same control as something other than back.
608+
NSString *title = [NSProcessInfo.processInfo.arguments containsObject:@"--top-leading-menu"] ? @"Menu" : @"Go back";
609+
[back setTitle:title forState:UIControlStateNormal];
592610
[back addTarget:self action:@selector(navigationBackPressed) forControlEvents:UIControlEventTouchUpInside];
593611
[self.view addSubview:back];
594612
[NSLayoutConstraint activateConstraints:@[

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import XCTest
44
extension RunnerTests {
55
static let navigationBackKeywords = ["back", "close", "cancel"]
66
static let systemBackButtonIdentifier = "BackButton"
7+
static let contentBackControlLabel = "Back"
78
static let floatingBarBackControlMaxYFraction: CGFloat = 0.4
89
static let navigationFallbackVerificationDelay: TimeInterval = 0.25
910

@@ -58,8 +59,12 @@ extension RunnerTests {
5859
|| isSnapshotXCTestChannelPenalized(bundleId: mainOwned.bundleId)
5960
{
6061
NSLog("AGENT_DEVICE_RUNNER_IN_APP_BACK_SKIPPED_XCTEST_ENUMERATION bundle=%@", mainOwned.bundleId ?? "")
61-
} else if let frame = try topNavigationBackFrame(app: app) {
62+
} else if let frame = try topNavigationBackFrame(app: app) ?? contentBackControlFrame(app: app) {
6263
return try tapResolvedNavigationBackControl(app: app, frame: frame)
64+
} else if navigationRejectedQueryCount() == rejectedQueriesBefore {
65+
// The top-leading tap stands in only for a tree XCTest cannot read. A readable tree that names
66+
// no back control has none, and a blind tap there would press whatever sits top-leading.
67+
return .unavailable
6368
}
6469
return try tapTopLeadingNavigationFallback(
6570
app: app,
@@ -139,6 +144,33 @@ extension RunnerTests {
139144
#endif
140145
}
141146

147+
/// A screen without a navigation header, such as a React Native native-stack screen with its
148+
/// header hidden, places its back control in the content. React Native renders a control without
149+
/// a role as an untyped element. The control qualifies at any height only when its whole label is
150+
/// "Back", so content that merely mentions back ("Back up photos", "Feedback") is never tapped.
151+
private func contentBackControlFrame(app: XCUIApplication) throws -> CGRect? {
152+
#if os(iOS)
153+
let window = try onScreenWindowFrame(app: app)
154+
return try elementQueries(app: app, matching: .any).flatMap {
155+
$0.matching(Self.contentBackControlPredicate()).allElementsBoundByIndex
156+
}.compactMap { element -> CGRect? in
157+
guard element.exists, element.isHittable else { return nil }
158+
let frame = element.frame
159+
return window.contains(CGPoint(x: frame.midX, y: frame.midY)) ? frame : nil
160+
}.min { ($0.minY, $0.minX) < ($1.minY, $1.minX) }
161+
#else
162+
return nil
163+
#endif
164+
}
165+
166+
static func contentBackControlPredicate() -> NSPredicate {
167+
NSPredicate(
168+
format: "elementType IN %@ AND label ==[c] %@",
169+
[XCUIElement.ElementType.button.rawValue, XCUIElement.ElementType.other.rawValue],
170+
contentBackControlLabel
171+
)
172+
}
173+
142174
static func navigationBackPredicate() -> NSPredicate {
143175
let clauses = navigationBackKeywords.flatMap { _ in
144176
["label CONTAINS[c] %@", "identifier CONTAINS[c] %@"]

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+NavigationTests.swift‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,33 @@ extension RunnerTests {
118118
XCTAssertEqual(status.label, "Back performed")
119119
}
120120

121+
/// A header-less screen keeps its back control in the content, as an untyped element when the app
122+
/// gave it no button role.
123+
@MainActor
124+
func testInAppBackTapsAnUntypedContentBackControlOnAHeaderlessScreen() throws {
125+
let status = try launchBackFixture(["--agent-device-back-regression", "--content-back"])
126+
defer { endBackFixture() }
127+
128+
guard case .performed = try tapInAppBackControl(app: app) else {
129+
return XCTFail("a visible content control labeled Back must be tapped")
130+
}
131+
let changed = expectation(for: NSPredicate(format: "label == %@", "Back performed"), evaluatedWith: status)
132+
wait(for: [changed], timeout: appExistenceTimeout)
133+
}
134+
135+
/// The top-leading tap stands in only for a tree XCTest cannot read. On a readable tree that names
136+
/// no back control, a blind tap would press the menu that sits where a back button would.
137+
@MainActor
138+
func testInAppBackOnAReadableTreeWithoutABackControlRefusesWithoutATap() throws {
139+
let status = try launchBackFixture(["--agent-device-back-regression", "--top-leading-menu"])
140+
defer { endBackFixture() }
141+
142+
guard case .unavailable = try tapInAppBackControl(app: app) else {
143+
return XCTFail("a readable tree without a back control must refuse")
144+
}
145+
XCTAssertNotEqual(status.label, "Back performed")
146+
}
147+
121148
/// Stands in for a device showing the app in part of its screen as far as the probe can tell: the
122149
/// probe's `primaryApp` read answers nothing, which the evidence seam reports as `unknown`. The
123150
/// windowed geometry itself needs an iPad set to Windowed Apps, which no lane runs; `unknown`
@@ -301,6 +328,19 @@ extension RunnerTests {
301328
XCTAssertFalse(predicate.evaluate(with: ["label": "Search for more feeds", "identifier": ""]))
302329
}
303330

331+
func testContentBackControlPredicateAcceptsOnlyAWholeBackLabelOnAControl() {
332+
let predicate = Self.contentBackControlPredicate()
333+
let other = XCUIElement.ElementType.other.rawValue
334+
let button = XCUIElement.ElementType.button.rawValue
335+
let staticText = XCUIElement.ElementType.staticText.rawValue
336+
337+
XCTAssertTrue(predicate.evaluate(with: ["elementType": other, "label": "Back"]))
338+
XCTAssertTrue(predicate.evaluate(with: ["elementType": button, "label": "back"]))
339+
XCTAssertFalse(predicate.evaluate(with: ["elementType": staticText, "label": "Back"]))
340+
XCTAssertFalse(predicate.evaluate(with: ["elementType": other, "label": "Back up photos"]))
341+
XCTAssertFalse(predicate.evaluate(with: ["elementType": button, "label": "Feedback"]))
342+
}
343+
304344
func testTopNavigationControlFrameAcceptsOnlyHeaderBand() {
305345
let window = CGRect(x: 0, y: 0, width: 430, height: 932)
306346

‎packages/command-surface/src/system/index.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,7 @@ export const SYSTEM_COMMAND_OUTPUT_SCHEMAS = {
286286
const appStateCommandDescription =
287287
'Show foreground app/activity (Android and HarmonyOS; iOS and macOS answer per session)';
288288
const backCommandDescription =
289-
'Navigate back in the app or through system navigation. Use in-app for the app navigation stack and system when the platform back behavior is required. On iOS, a single-screen iPhone or an iPad whose app provably fills its screen can use a top-leading tap when the app tree is unreadable, with before/after visual verification. A capture failure reports an unknown outcome.';
289+
'Navigate back in the app or through system navigation. Use in-app for the app navigation stack and system when the platform back behavior is required. On iOS, in-app back taps the header back control or a visible control labeled exactly Back, and refuses on a readable screen that has neither. A single-screen iPhone or an iPad whose app provably fills its screen can use a top-leading tap when the app tree is unreadable, with before/after visual verification. A capture failure reports an unknown outcome.';
290290
const homeCommandDescription =
291291
'Send the selected device to its home screen. This leaves the app session open but moves the foreground away from the app.';
292292
const orientationCommandDescription =

‎website/docs/docs/commands.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,8 @@ iOS Simulator launch behavior:
124124
Back, orientation, and hardware buttons:
125125

126126
- `back` uses app-owned back navigation: visible in-app back UI on Apple targets, and the back keyevent on Android. `--in-app` names this default explicitly.
127-
- On iOS, a single-screen iPhone or an iPad whose app provably fills its screen, `back` can use a top-leading tap when the app tree is unreadable. It compares the screen before and after the tap. If the screen cannot be captured, the command reports an unknown outcome; inspect the screen before you retry.
127+
- On iOS, `back` taps the header back control. On a screen without one, it taps a visible control labeled exactly `Back`, wherever it sits. When the app tree is readable and has neither, `back` refuses with `in-app back control is not available` and taps nothing; use `back --system` for the edge-swipe gesture.
128+
- On a single-screen iPhone or an iPad whose app provably fills its screen, `back` can use a top-leading tap when the app tree is unreadable. It compares the screen before and after the tap. If the screen cannot be captured, the command reports an unknown outcome; inspect the screen before you retry.
128129
- `back --system` sends system back input: the Android back keyevent, the iOS back gesture, or the tvOS Siri Remote Menu button. macOS has no system back, so it reports unavailable.
129130
- `orientation portrait|portrait-upside-down|landscape-left|landscape-right` works on iOS and Android mobile targets only. On ADB-backed Android it turns auto-rotate off, and the lock persists after the session ends until something turns it back on. A foldable iPhone simulator (iPhone Duo) refuses `portrait-upside-down`, and refuses every rotation in a scoped simulator set (`--ios-simulator-device-set`).
130131
- `orientation auto` hands rotation back to the sensor on ADB-backed Android, local or Limrun (`accelerometer_rotation=1`, `user_rotation` untouched), and reports `orientation: "auto"`. It does not wait for the display to rotate, because there is no target rotation: the sensor decides, and an emulator usually settles on portrait. Every other backend refuses `auto` with `UNSUPPORTED_OPERATION`.

0 commit comments

Comments
 (0)