Skip to content

Commit b813505

Browse files
committed
fix(ios): address review round on the sole-screen route and nightly legs
- Help and docs: scope the top-leading back fallback to iOS (a single-screen iPhone, or an iPad whose app provably fills its screen); Android uses a back keyevent. - Fixture: anchor the regression back control to the view's leading edge and the safe-area top with a 64-pt height so it stays under the bounded fallback sweep (x <= 44, y in [56, 96]) on landscape iPhones with side insets. - Tests: restore iPad window-route coverage by closing the sole-screen route through the probe's own unknown path (primaryApp answers nothing) and dispatching back through the resolved window. - Tests: assert .windowed exactly, and non-finite frames as .unknown. - Shared rule: a non-finite frame component is an unknown read, not a window; the bridge keeps the capture and the runner's evidence seam answers Unknown. - Nightly: each leg asserts the booted simulator's family, the iPad leg records a result bundle and asserts its seven selected tests executed, and the iPad device name matches the models the macos-26 image ships for iOS 26.2 (M5).
1 parent 5b4986a commit b813505

8 files changed

Lines changed: 161 additions & 31 deletions

File tree

‎.github/workflows/xctest-nightly.yml‎

Lines changed: 53 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,16 @@ jobs:
3333
# of the rule — the route opens only while the in-process full-screen evidence proves
3434
# the app fills its screen — can execute nowhere else, and the evidence line that probe
3535
# prints is what turns a revert to idiom-only routing red instead of silently green.
36+
# `name-prefix` is what each leg asserts the booted simulator's name starts with: the
37+
# boot action falls back to any simulator when the preferred name is missing, so a
38+
# renamed or absent device model must fail the leg that depends on its idiom.
3639
- name: iPhone full suite
3740
device-name: iPhone 17 Pro
41+
name-prefix: iPhone
3842
lane: nightly
3943
- name: iPad sole-screen evidence
40-
device-name: iPad Pro 11-inch (M4)
44+
device-name: iPad Pro 11-inch (M5)
45+
name-prefix: iPad
4146
lane: nightly-ipad
4247
env:
4348
# Must match apple.yml. The runtime version rides in the derived-data cache key, and
@@ -85,6 +90,30 @@ jobs:
8590
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
8691
preferred-device-name: ${{ matrix.leg.device-name }}
8792

93+
# The boot action falls back to any simulator when the preferred name is missing, so each
94+
# leg proves it booted the family it claims: the full-suite leg's coverage is iPhone-only at
95+
# the idiom, and the iPad leg's evidence line can only come from an iPad. A renamed or
96+
# removed device model on the image fails here instead of quietly testing the wrong idiom.
97+
- name: Assert the booted simulator's family
98+
run: |
99+
set -euo pipefail
100+
node -e '
101+
const { execFileSync } = require("node:child_process");
102+
const [udid, prefix] = process.argv.slice(1);
103+
const payload = JSON.parse(
104+
execFileSync("xcrun", ["simctl", "list", "devices", "-j"], { maxBuffer: 64 * 1024 * 1024 }).toString(),
105+
);
106+
const device = Object.values(payload.devices ?? {})
107+
.flat()
108+
.find((entry) => entry.udid === udid);
109+
if (!device?.name?.startsWith(prefix)) {
110+
console.error(`Booted ${device?.name ?? udid}; this leg needs a simulator named "${prefix}…"`);
111+
process.exit(1);
112+
}
113+
console.log(`Booted ${device.name}`);
114+
' '${{ steps.ios-simulator.outputs.simulator-udid }}' '${{ matrix.leg.name-prefix }}'
115+
shell: bash
116+
88117
# Same command as apple.yml's targeted step minus every `-only-testing:` flag, so the
89118
# xctestrun's own test plan decides what runs — with one exception.
90119
#
@@ -119,10 +148,11 @@ jobs:
119148
}
120149
'
121150
122-
# The iPad leg's liveness check is its own grep below: it runs a named selection, not the
123-
# whole bundle, so the count assertion the full-suite leg makes does not apply to it. The
124-
# boot action falls back to any simulator when the preferred name is missing, so this step
125-
# proves it booted the iPad that leg claims — the evidence line can only come from one.
151+
# The iPad leg's liveness checks are its own assertions below: it runs a named selection, not
152+
# the whole bundle, so the count assertion the full-suite leg makes does not apply to it, and
153+
# `xcodebuild` exits 0 when a selection matches nothing — a renamed test would shrink the run,
154+
# not fail it. The step asserts the executed count equals the selection, then greps the probe
155+
# line the iPad half of the rule prints.
126156
- name: Run the sole-screen tests on iPad
127157
if: matrix.leg.lane == 'nightly-ipad'
128158
run: |
@@ -131,18 +161,6 @@ jobs:
131161
# it must create it in its own leg before `tee` writes into it.
132162
mkdir -p .tmp/xctest-nightly
133163
UDID='${{ steps.ios-simulator.outputs.simulator-udid }}'
134-
xcrun simctl list devices -j | node -e '
135-
const fs = require("node:fs");
136-
const udid = process.argv[1];
137-
const payload = JSON.parse(fs.readFileSync(0, "utf8"));
138-
const device = Object.values(payload.devices ?? {})
139-
.flat()
140-
.find((entry) => entry.udid === udid);
141-
if (!device?.name?.startsWith("iPad")) {
142-
console.error(`Booted ${device?.name ?? udid}; this leg needs an iPad simulator`);
143-
process.exit(1);
144-
}
145-
' "$UDID"
146164
XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)"
147165
test -n "$XCTESTRUN_PATH"
148166
xcodebuild test-without-building \
@@ -153,8 +171,25 @@ jobs:
153171
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen \
154172
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackVerifiesSoleScreenWhenXCTestCannotReadFramesOrEnumerate \
155173
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryTapsThroughTheRouteThatOpens \
174+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryOnAWindowedIpadTapsThroughTheWindow \
156175
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRankedInAppBackUsesSynthesisWithoutXCTestTap \
157-
2>&1 | tee .tmp/xctest-nightly/runner.log
176+
-resultBundlePath "$RESULT_BUNDLE_PATH" 2>&1 | tee .tmp/xctest-nightly/runner.log
177+
# `xcodebuild` reports an unmatched selection by running fewer tests, not by failing, so
178+
# the executed count from the result bundle is what proves this leg's whole selection ran.
179+
node --input-type=module -e '
180+
import { execFileSync } from "node:child_process";
181+
const summary = JSON.parse(execFileSync(
182+
"xcrun", ["xcresulttool", "get", "test-results", "summary", "--path", process.env.RESULT_BUNDLE_PATH, "--compact"],
183+
{ maxBuffer: 64 * 1024 * 1024 },
184+
).toString());
185+
const expected = 7;
186+
if (summary.totalTestCount !== expected) {
187+
throw new Error(
188+
`The iPad sole-screen selection ran ${summary.totalTestCount} tests, expected ${expected}; ` +
189+
"a selection stopped matching its test",
190+
);
191+
}
192+
'
158193
# The probe prints this line only when a non-phone idiom asks it. Its absence means the
159194
# iPad never probed (a revert to idiom-only routing), not that a test failed.
160195
grep -q 'AGENT_DEVICE_RUNNER_FULL_SCREEN_EVIDENCE result=fills-screen' \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerApp.m‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -672,19 +672,22 @@ - (void)viewDidLoad {
672672
bar.items = @[item];
673673
[self.view addSubview:bar];
674674
} else {
675-
// The control sits in the top bar band, whose height above the content differs by family
676-
// (59 pt on an iPhone 17 Pro, 32 pt on an iPad Pro), so it anchors to the safe-area top
677-
// instead of a hardcoded iPhone y.
675+
// The control must sit under the point the runner's top-leading fallback aims at, whose
676+
// bounds are known: x lands at most 44 from the frame's leading edge, y at
677+
// clamp(safeTopInset + 22, 56, 96). Anchoring to the view's leading (not the safe-area's)
678+
// keeps the button under x=44 on a landscape iPhone, where the side inset would otherwise
679+
// start a safe-area anchor past it; the safe-area top plus 12 with a 64-pt height spans the
680+
// whole clamped y range for every inset an iPhone or iPad reports.
678681
UIButton *back = [UIButton buttonWithType:UIButtonTypeSystem];
679682
back.translatesAutoresizingMaskIntoConstraints = NO;
680683
[back setTitle:@"Go back" forState:UIControlStateNormal];
681684
[back addTarget:self action:@selector(navigationBackPressed) forControlEvents:UIControlEventTouchUpInside];
682685
[self.view addSubview:back];
683686
[NSLayoutConstraint activateConstraints:@[
684-
[back.leadingAnchor constraintEqualToAnchor:self.view.safeAreaLayoutGuide.leadingAnchor constant:12],
685-
[back.topAnchor constraintEqualToAnchor:self.view.safeAreaLayoutGuide.topAnchor],
687+
[back.leadingAnchor constraintEqualToAnchor:self.view.leadingAnchor constant:12],
688+
[back.topAnchor constraintEqualToAnchor:self.view.safeAreaLayoutGuide.topAnchor constant:12],
686689
[back.widthAnchor constraintEqualToConstant:88],
687-
[back.heightAnchor constraintEqualToConstant:44],
690+
[back.heightAnchor constraintEqualToConstant:64],
688691
]];
689692
}
690693
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.m‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,16 @@ static BOOL AgentDeviceForegroundOwnershipMatches(pid_t primaryProcessID, pid_t
3434
// of the screen (Windowed Apps or Stage Manager).
3535
static const CGFloat AgentDeviceFullScreenFrameTolerance = 0.5;
3636

37+
// A frame with a non-finite component is not a measurement at all. It proves neither the fill nor
38+
// the window, so it must reach each caller's unknown policy, never a comparison that reads as a
39+
// difference (the bridge would refuse `app-window-not-full-screen` on a runtime that said nothing).
40+
static BOOL AgentDeviceForegroundFramesAreFinite(CGRect window, CGRect screen) {
41+
return isfinite(window.origin.x) && isfinite(window.origin.y) &&
42+
isfinite(window.size.width) && isfinite(window.size.height) &&
43+
isfinite(screen.origin.x) && isfinite(screen.origin.y) &&
44+
isfinite(screen.size.width) && isfinite(screen.size.height);
45+
}
46+
3747
static BOOL AgentDeviceForegroundWindowFillsScreen(CGRect window, CGRect screen) {
3848
return fabs(window.origin.x - screen.origin.x) <= AgentDeviceFullScreenFrameTolerance &&
3949
fabs(window.origin.y - screen.origin.y) <= AgentDeviceFullScreenFrameTolerance &&
@@ -276,6 +286,10 @@ RunnerFullScreenEvidence RunnerFullScreenEvidenceFromForeground(
276286
CGRectIsEmpty(appFrame) || CGRectIsEmpty(screenFrame)) {
277287
return RunnerFullScreenEvidenceUnknown;
278288
}
289+
// A frame with a non-finite component is a runtime that said nothing, not a window that differs.
290+
if (!AgentDeviceForegroundFramesAreFinite(appFrame, screenFrame)) {
291+
return RunnerFullScreenEvidenceUnknown;
292+
}
279293
return AgentDeviceForegroundWindowFillsScreen(appFrame, screenFrame)
280294
? RunnerFullScreenEvidenceFillsScreen
281295
: RunnerFullScreenEvidenceWindowed;

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+NavigationTests.swift‎

Lines changed: 54 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,9 @@ extension RunnerTests {
5252

5353
/// After the tree rejects a query, `back` reaches the top-leading control through whichever route
5454
/// is open: the sole screen where the app proves it fills it (both families here), the resolved
55-
/// window everywhere else. The refusal half — a device where no route opens — is the negative in
56-
/// `testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen`, which no simulator here can
57-
/// reach because its foreground app always fills its screen.
55+
/// window everywhere else. The refusal half on a real device is
56+
/// `testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen` and the windowed-iPad test
57+
/// below, which stand where no simulator here can: its foreground app always fills its screen.
5858
@MainActor
5959
func testInAppBackAfterARejectedQueryTapsThroughTheRouteThatOpens() throws {
6060
let status = try launchBackFixture(["--agent-device-back-regression"])
@@ -70,6 +70,57 @@ extension RunnerTests {
7070
XCTAssertEqual(status.label, "Back performed")
7171
}
7272

73+
/// The window route the sole-screen rule leaves for an iPad whose app it cannot prove full: with
74+
/// the probe's primary-app read stubbed to answer nothing (the evidence seam's `unknown`), the
75+
/// route closes, and a rejected tree still dispatches `back` through the resolved window. An
76+
/// iPhone keeps the sole-screen route by idiom — it answers before the probe is ever asked, so no
77+
/// stub can close it there and the window route is the iPad half's own.
78+
@MainActor
79+
func testInAppBackAfterARejectedQueryOnAWindowedIpadTapsThroughTheWindow() throws {
80+
let status = try launchBackFixture(["--agent-device-back-regression"])
81+
defer { endBackFixture() }
82+
// Runs the real probe once, which is also what registers the AXRuntime class this seam stubs.
83+
XCTAssertTrue(soleScreenRouteIsOpen(for: app))
84+
guard UIDevice.current.userInterfaceIdiom != .phone else {
85+
return
86+
}
87+
let restoreProbe = stubForegroundPrimaryAppAsUnprovable()
88+
defer { restoreProbe() }
89+
XCTAssertFalse(
90+
soleScreenRouteIsOpen(for: app),
91+
"an iPad whose app the probe cannot prove keeps requiring a resolved window"
92+
)
93+
94+
let outcome = try withRejectedAppTree(framesReadable: true, rejections: 1) {
95+
tapInAppBackControl(app: app)
96+
}
97+
guard case .performed = outcome else {
98+
return XCTFail("a refused sole-screen route must still dispatch back through the resolved window")
99+
}
100+
XCTAssertEqual(status.label, "Back performed")
101+
}
102+
103+
/// Stands in for a device showing the app in part of its screen as far as the probe can tell: the
104+
/// probe's `primaryApp` read answers nothing, which the evidence seam reports as `unknown`. The
105+
/// windowed geometry itself needs an iPad set to Windowed Apps, which no lane runs; `unknown`
106+
/// closes the same route for the same reason — the fill is not proven.
107+
private func stubForegroundPrimaryAppAsUnprovable() -> () -> Void {
108+
let element = NSClassFromString("AXElement")
109+
let method = element.flatMap { class_getClassMethod($0, NSSelectorFromString("primaryApp")) }
110+
guard let method else {
111+
preconditionFailure("AXRuntime did not register AXElement before the probe seam was stubbed")
112+
}
113+
let original = method_getImplementation(method)
114+
let replacement = imp_implementationWithBlock(
115+
{ (_: AnyObject) -> AnyObject? in nil } as @convention(block) (AnyObject) -> AnyObject?
116+
)
117+
method_setImplementation(method, replacement)
118+
return {
119+
method_setImplementation(method, original)
120+
imp_removeBlock(replacement)
121+
}
122+
}
123+
73124
private var showsAppsOnlyFullScreen: Bool {
74125
soleScreenRouteIsOpen(for: app)
75126
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+XCTestEventBridgeTests.swift‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -226,9 +226,9 @@ extension RunnerTests {
226226
RunnerFullScreenEvidenceFromForeground(100, 100, screen.offsetBy(dx: 0.4, dy: -0.4), screen),
227227
.fillsScreen
228228
)
229-
XCTAssertNotEqual(
229+
XCTAssertEqual(
230230
RunnerFullScreenEvidenceFromForeground(100, 100, screen.offsetBy(dx: 1, dy: 0), screen),
231-
.fillsScreen
231+
.windowed
232232
)
233233

234234
// A different app owns the foreground, so the screen is not this app's to fill.
@@ -254,6 +254,20 @@ extension RunnerTests {
254254
RunnerFullScreenEvidenceFromForeground(100, 100, screen, .zero),
255255
.unknown
256256
)
257+
// A non-finite frame component is a runtime that said nothing, not a window that differs:
258+
// the answer is Unknown, never the Windowed a comparison of the numbers would read as.
259+
XCTAssertEqual(
260+
RunnerFullScreenEvidenceFromForeground(
261+
100, 100, CGRect(x: CGFloat.nan, y: 0, width: 834, height: 1210), screen
262+
),
263+
.unknown
264+
)
265+
XCTAssertEqual(
266+
RunnerFullScreenEvidenceFromForeground(
267+
100, 100, screen, CGRect(x: 0, y: 0, width: CGFloat.infinity, height: 1210)
268+
),
269+
.unknown
270+
)
257271
}
258272

259273
/// The routing half of the rule off the phone idiom: anything but the phone idiom routes only on

‎apple/snapshot-bridge/SnapshotBridgeRuntime.m‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,16 @@ static BOOL AgentDeviceForegroundOwnershipMatches(pid_t primaryProcessID, pid_t
9191
// of the screen (Windowed Apps or Stage Manager).
9292
static const CGFloat AgentDeviceFullScreenFrameTolerance = 0.5;
9393

94+
// A frame with a non-finite component is not a measurement at all. It proves neither the fill nor
95+
// the window, so it must reach each caller's unknown policy, never a comparison that reads as a
96+
// difference (the bridge would refuse `app-window-not-full-screen` on a runtime that said nothing).
97+
static BOOL AgentDeviceForegroundFramesAreFinite(CGRect window, CGRect screen) {
98+
return isfinite(window.origin.x) && isfinite(window.origin.y) &&
99+
isfinite(window.size.width) && isfinite(window.size.height) &&
100+
isfinite(screen.origin.x) && isfinite(screen.origin.y) &&
101+
isfinite(screen.size.width) && isfinite(screen.size.height);
102+
}
103+
94104
static BOOL AgentDeviceForegroundWindowFillsScreen(CGRect window, CGRect screen) {
95105
return fabs(window.origin.x - screen.origin.x) <= AgentDeviceFullScreenFrameTolerance &&
96106
fabs(window.origin.y - screen.origin.y) <= AgentDeviceFullScreenFrameTolerance &&
@@ -234,6 +244,9 @@ - (BOOL)primaryAppWindowDiffersFromScreen
234244
CGRect window = [application frame];
235245
CGRect screen = [system frame];
236246
if (CGRectIsEmpty(window) || CGRectIsEmpty(screen)) return NO;
247+
// The documented unknown-frame fallback: a frame with a non-finite component is a runtime that
248+
// reported nothing, not a window that differs from the screen, so the capture is kept.
249+
if (!AgentDeviceForegroundFramesAreFinite(window, screen)) return NO;
237250
return !AgentDeviceForegroundWindowFillsScreen(window, screen);
238251
}
239252

‎src/commands/system/index.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,7 @@ export const SYSTEM_COMMAND_OUTPUT_SCHEMAS = {
287287
const appStateCommandDescription =
288288
'Show foreground app/activity (Android and HarmonyOS; iOS and macOS answer per session)';
289289
const backCommandDescription =
290-
'Navigate back in the app or through system navigation. Use in-app for the app navigation stack and system when the platform back behavior is required. On a single-screen device where the app provably fills that screen, an unreadable app tree can use a top-leading tap with before/after visual verification. A capture failure reports an unknown outcome.';
290+
'Navigate back in the app or through system navigation. Use in-app for the app navigation stack and system when the platform back behavior is required. On iOS, a single-screen iPhone or an iPad whose app provably fills its screen can use a top-leading tap when the app tree is unreadable, with before/after visual verification. A capture failure reports an unknown outcome.';
291291
const homeCommandDescription =
292292
'Send the selected device to its home screen. This leaves the app session open but moves the foreground away from the app.';
293293
const orientationCommandDescription =

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ iOS Simulator launch behavior:
118118
Back, orientation, and hardware buttons:
119119

120120
- `back` uses app-owned back navigation: visible in-app back UI on Apple targets, and the back keyevent on Android. `--in-app` names this default explicitly.
121-
- On a single-screen device where the app provably fills that screen, `back` can use a top-leading tap when the app tree is unreadable. It compares the screen before and after the tap. If the screen cannot be captured, the command reports an unknown outcome; inspect the screen before you retry.
121+
- On iOS, a single-screen iPhone or an iPad whose app provably fills its screen, `back` can use a top-leading tap when the app tree is unreadable. It compares the screen before and after the tap. If the screen cannot be captured, the command reports an unknown outcome; inspect the screen before you retry.
122122
- `back --system` sends system back input: the Android back keyevent, the iOS back gesture, or the tvOS Siri Remote Menu button. macOS has no system back, so it reports unavailable.
123123
- `orientation portrait|portrait-upside-down|landscape-left|landscape-right` works on iOS and Android mobile targets only. On ADB-backed Android it turns auto-rotate off, and the lock persists after the session ends until something turns it back on.
124124
- `orientation auto` hands rotation back to the sensor on ADB-backed Android, local or Limrun (`accelerometer_rotation=1`, `user_rotation` untouched), and reports `orientation: "auto"`. It does not wait for the display to rotate, because there is no target rotation: the sensor decides, and an emulator usually settles on portrait. Every other backend refuses `auto` with `UNSUPPORTED_OPERATION`.

0 commit comments

Comments
 (0)