Skip to content

Commit d2426ff

Browse files
thymikeeclaude
andauthored
fix(ios): make back the verified left-edge swipe (#4460)
iOS back (with or without --in-app or --system) is the left-edge back swipe, verified by a change of the app's screen; the runner no longer guesses at buttons. A simulator sends one edge-masked HID touch; a physical device sends a hold-free 200 ms synthesized fling. Both are resolved before the sample the change is measured from. The screen must be still for the contact allowance plus one delay plus the slowest capture, and a change counts only when the gap did not outrun that still span. Each back logs AGENT_DEVICE_RUNNER_BACK_SWIPE_VERIFICATION with its decision and timings. visionOS is refused at the back capability fact. backSystem is removed. Rebased onto #4474 as one commit; PreparedTouch(notSent:) maps #4474's .notDispatched to the same pre-touch refusal as .unsupported. Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 0495ee1 commit d2426ff

43 files changed

Lines changed: 1296 additions & 676 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/xctest-nightly.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,8 @@ jobs:
152152
# The iPad leg runs a named selection: the smallest set that can execute the off-phone half of
153153
# the sole-screen rule — the admission boundary itself, the refusal when the display box a
154154
# synthesized turn swaps cannot be read and the app is not proven full-screen (#4091), one
155-
# hostile-tree command that must reach synthesis, and `back`'s route witness.
155+
# hostile-tree command that must reach synthesis, and `back`'s route witness: it swipes and
156+
# verifies through the route that opens after a rejected query.
156157
# Geometry and error cases live in the host-runnable evidence matrix instead. `xcodebuild`
157158
# exits 0 when a selection matches nothing, so the reporter below asserts the executed count
158159
# against this lane's reach (`XCTEST_LANE=nightly-ipad`, derived by check:xctest-selection).
@@ -172,8 +173,7 @@ jobs:
172173
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSoleScreenDisplayIDIsNamedOnlyForAnAppThatFillsItsScreen \
173174
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testTapRefusesBeforeDispatchWhenTheSoleDisplayCannotBeMeasured \
174175
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testLongPressOnAnUnserializableTreeIsSynthesizedOnTheSoleScreen \
175-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryTapsThroughTheRouteThatOpens \
176-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQueryWithUnprovableForegroundTapsThroughTheWindow \
176+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testInAppBackAfterARejectedQuerySwipesAndVerifiesOnTheRouteThatOpens \
177177
-resultBundlePath "$RESULT_BUNDLE_PATH" 2>&1 | tee .tmp/xctest-nightly/runner.log
178178
179179
# Best-effort and never the job's verdict on its own; the step below is what asserts.

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerAppFixtures.m‎

Lines changed: 46 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ @interface AgentDeviceRunnerFixtureViewController () <UNUserNotificationCenterDe
4343
// Held for the app's lifetime: a window nobody retained would be gone before the test queried it.
4444
@property(nonatomic, strong) UIWindow *secondWindow;
4545
@property(nonatomic, strong) UILabel *navigationBackStatus;
46+
@property(nonatomic, strong) NSTimer *animationTimer;
4647
@property(nonatomic, strong) UILabel *maskedToggleStatus;
4748
@property(nonatomic, strong) UILabel *maskedDeliveryStatus;
4849
@property(nonatomic, strong) UILabel *maskedTextLengthStatus;
@@ -205,6 +206,12 @@ - (BOOL)textFieldShouldReturn:(UITextField *)textField {
205206

206207
@implementation AgentDeviceRunnerFixtureViewController
207208

209+
// `--hide-status-bar` stands in for a landscape iPhone, full-screen video, or an RN screen that hides
210+
// its status bar.
211+
- (BOOL)prefersStatusBarHidden {
212+
return [NSProcessInfo.processInfo.arguments containsObject:@"--hide-status-bar"];
213+
}
214+
208215
// An animation that never ends is what "busy" looks like to XCTest while it decides whether the app
209216
// may receive an event: the app keeps reporting work in flight, which is the state that cost an alert
210217
// command its whole deadline in #2546. It stops the moment an alert button is answered, since that
@@ -544,8 +551,12 @@ - (void)updateMaskedToggleStatus {
544551
[NSString stringWithFormat:@"presses=%lu", (unsigned long)self.maskedTogglePresses];
545552
}
546553

547-
- (void)navigationBackPressed {
548-
self.navigationBackStatus.text = @"Back performed";
554+
- (void)navigationBackSwiped:(UIScreenEdgePanGestureRecognizer *)recognizer {
555+
if (recognizer.state == UIGestureRecognizerStateEnded) {
556+
// A back navigation redraws most of the screen, so the fixture repaints all of it too.
557+
self.view.backgroundColor = UIColor.systemTealColor;
558+
self.navigationBackStatus.text = @"Back performed";
559+
}
549560
}
550561

551562
- (void)agentDeviceWillAttachToWindow:(UIWindow *)window {
@@ -569,34 +580,42 @@ - (void)viewDidLoad {
569580

570581
UILabel *label = self.label;
571582

572-
// Keep the fixture behind a launch argument so normal runner snapshots remain unchanged.
583+
// Keep the fixture behind a launch argument so normal runner snapshots remain unchanged. A
584+
// left-edge pan stands in for UIKit's interactive pop: it begins only for an edge-masked touch.
573585
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-back-regression"]) {
574586
self.navigationBackStatus = label;
575587
label.accessibilityIdentifier = @"agent-device-back-status";
576-
if ([NSProcessInfo.processInfo.arguments containsObject:@"--navigation-bar"]) {
577-
UINavigationBar *bar = [[UINavigationBar alloc] initWithFrame:CGRectMake(0, 62, self.view.bounds.size.width, 44)];
578-
UINavigationItem *item = [[UINavigationItem alloc] initWithTitle:@"Thread"];
579-
item.leftBarButtonItem = [[UIBarButtonItem alloc] initWithTitle:@"Go back" style:UIBarButtonItemStylePlain target:self action:@selector(navigationBackPressed)];
580-
bar.items = @[item];
581-
[self.view addSubview:bar];
582-
} else {
583-
// The control must sit under the point the runner's top-leading fallback aims at, whose
584-
// bounds are known: x lands at most 44 from the frame's leading edge, y at
585-
// clamp(safeTopInset + 22, 56, 96). Anchoring to the view's leading (not the safe-area's)
586-
// keeps the button under x=44 on a landscape iPhone, where the side inset would otherwise
587-
// start a safe-area anchor past it; the safe-area top plus 12 with a 64-pt height spans the
588-
// whole clamped y range for every inset an iPhone or iPad reports.
589-
UIButton *back = [UIButton buttonWithType:UIButtonTypeSystem];
590-
back.translatesAutoresizingMaskIntoConstraints = NO;
591-
[back setTitle:@"Go back" forState:UIControlStateNormal];
592-
[back addTarget:self action:@selector(navigationBackPressed) forControlEvents:UIControlEventTouchUpInside];
593-
[self.view addSubview:back];
594-
[NSLayoutConstraint activateConstraints:@[
595-
[back.leadingAnchor constraintEqualToAnchor:self.view.leadingAnchor constant:12],
596-
[back.topAnchor constraintEqualToAnchor:self.view.safeAreaLayoutGuide.topAnchor constant:12],
597-
[back.widthAnchor constraintEqualToConstant:88],
598-
[back.heightAnchor constraintEqualToConstant:64],
599-
]];
588+
UIScreenEdgePanGestureRecognizer *edgePan =
589+
[[UIScreenEdgePanGestureRecognizer alloc] initWithTarget:self action:@selector(navigationBackSwiped:)];
590+
edgePan.edges = UIRectEdgeLeft;
591+
[self.view addGestureRecognizer:edgePan];
592+
// `--animating` keeps the whole screen changing on its own, as a launch that is still rendering:
593+
// 16 bands take independent random shades, so no two samples of the screen match.
594+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--animating"]) {
595+
NSMutableArray<UIView *> *bands = [NSMutableArray array];
596+
CGRect bounds = self.view.bounds;
597+
for (NSUInteger index = 0; index < 16; index++) {
598+
UIView *band = [[UIView alloc] initWithFrame:CGRectMake(
599+
0, bounds.size.height * index / 16, bounds.size.width, bounds.size.height / 16)];
600+
band.autoresizingMask = UIViewAutoresizingFlexibleWidth;
601+
[self.view insertSubview:band atIndex:0];
602+
[bands addObject:band];
603+
}
604+
[self.animationTimer invalidate];
605+
self.animationTimer = [NSTimer scheduledTimerWithTimeInterval:0.05 repeats:YES block:^(NSTimer *timer) {
606+
for (UIView *band in bands) {
607+
band.backgroundColor = [UIColor colorWithWhite:arc4random_uniform(256) / 255.0 alpha:1];
608+
}
609+
}];
610+
}
611+
// `--slow-toggle` repaints the whole screen every 0.5 s, slower than one verification delay, as
612+
// a carousel does: one quiet pair of samples must not pass it for settled.
613+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--slow-toggle"]) {
614+
__weak UIView *view = self.view;
615+
[self.animationTimer invalidate];
616+
self.animationTimer = [NSTimer scheduledTimerWithTimeInterval:0.5 repeats:YES block:^(NSTimer *timer) {
617+
view.backgroundColor = [view.backgroundColor isEqual:UIColor.blackColor] ? UIColor.whiteColor : UIColor.blackColor;
618+
}];
600619
}
601620
}
602621

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSimulatorEdgeContact.h‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,13 +155,15 @@ static inline void RunnerWaitUntil(uint64_t startTicks, double offsetMs) {
155155

156156
// Owns one edge contact from touch-down to its terminal event. The cancel event is built before
157157
// touch-down; after touch-down, every exit dispatches either the touch-up or that cancel event.
158-
// Returns nil on success, or the reason the contact did not complete.
158+
// Returns nil on success, or the reason the contact did not complete. `touchDown` turns true once
159+
// the touch-down is dispatched, so a caller can tell a contact that never began from one that ended.
159160
static inline NSString * _Nullable RunnerDispatchEdgeContact(
160161
const RunnerHIDFunctions *functions,
161162
CFTypeRef client,
162163
uint64_t sender,
163164
NSArray<NSDictionary<NSString *, NSNumber *> *> *samples,
164-
CGSize screenSize
165+
CGSize screenSize,
166+
BOOL *touchDown
165167
) {
166168
double durationMs = samples.lastObject[@"offsetMs"].doubleValue;
167169
CGPoint start = RunnerPointAtOffset(samples, 0, screenSize);
@@ -179,6 +181,7 @@ static inline NSString * _Nullable RunnerDispatchEdgeContact(
179181
}
180182
uint64_t startTicks = mach_absolute_time();
181183
RunnerDispatchAndRelease(functions, client, down);
184+
*touchDown = YES;
182185
NSString *failure = nil;
183186
for (double offsetMs = RunnerFrameIntervalMs; offsetMs < durationMs + RunnerFrameIntervalMs;
184187
offsetMs += RunnerFrameIntervalMs) {

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSimulatorEdgeTouch.h‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,25 @@ NS_ASSUME_NONNULL_BEGIN
1111
// XCTest synthesized touches never carry the digitizer edge mask that UIKit reads.
1212
@interface RunnerSimulatorEdgeTouch : NSObject
1313

14-
// The touch goes to the touchscreen of the display that hosts the app: `resolvedWindow`'s display,
15-
// or the sole screen when it is nil, as for XCTest synthesis. `samples` is the ordered
16-
// { x, y, offsetMs } path of one contact in portrait screen points, and `screenSize` is that
17-
// screen's portrait size in points. The call returns after the contact lifts.
18-
// Returns nil on success, or the reason the touch could not be dispatched.
19-
+ (NSString * _Nullable)dispatchWithApplication:(id)application
14+
// Resolves everything the touch needs before its contact: the display that hosts the app
15+
// (`resolvedWindow`'s display, or the sole screen when it is nil, as for XCTest synthesis), the
16+
// IOKit HID entry points, and that display's touchscreen sender. Returns nil with `failure` set
17+
// when any of them is missing; nothing has been sent then.
18+
+ (nullable instancetype)prepareWithApplication:(id)application
2019
resolvedWindow:(id _Nullable)resolvedWindow
21-
samples:(NSArray<NSDictionary<NSString *, NSNumber *> *> *)samples
22-
screenSize:(CGSize)screenSize;
20+
failure:(NSString * _Nullable __autoreleasing * _Nonnull)failure
21+
NS_SWIFT_NAME(prepare(application:resolvedWindow:failure:));
22+
23+
- (instancetype)init NS_UNAVAILABLE;
24+
25+
// Runs one contact. `samples` is the ordered { x, y, offsetMs } path of the contact in portrait
26+
// screen points, and `screenSize` is that screen's portrait size in points. The call returns after
27+
// the contact lifts. Returns nil on success, or the reason the touch could not be dispatched.
28+
// `touchDown` is YES once the contact went down, so a failure with it NO reached nothing.
29+
- (NSString * _Nullable)dispatchSamples:(NSArray<NSDictionary<NSString *, NSNumber *> *> *)samples
30+
screenSize:(CGSize)screenSize
31+
touchDown:(BOOL *)touchDown
32+
NS_SWIFT_NAME(dispatch(samples:screenSize:touchDown:));
2333

2434
@end
2535

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSimulatorEdgeTouch.m‎

Lines changed: 50 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -69,36 +69,69 @@
6969
return failure;
7070
}
7171

72-
@implementation RunnerSimulatorEdgeTouch
72+
@implementation RunnerSimulatorEdgeTouch {
73+
RunnerHIDFunctions _functions;
74+
CFTypeRef _client;
75+
uint64_t _sender;
76+
}
7377

74-
+ (NSString * _Nullable)dispatchWithApplication:(id)application
78+
+ (nullable instancetype)prepareWithApplication:(id)application
7579
resolvedWindow:(id _Nullable)resolvedWindow
76-
samples:(NSArray<NSDictionary<NSString *, NSNumber *> *> *)samples
77-
screenSize:(CGSize)screenSize {
78-
if (samples.count < 2 || screenSize.width <= 0 || screenSize.height <= 0) {
79-
return @"an edge touch needs at least two samples and a measured screen";
80-
}
80+
failure:(NSString * _Nullable __autoreleasing * _Nonnull)failure {
81+
*failure = nil;
8182
NSUInteger displayID = 0;
8283
id soleScreen = nil;
8384
NSString *displayError = resolvedWindow == nil
8485
? RunnerResolveSoleScreen(application, &soleScreen, &displayID)
8586
: RunnerResolveWindowDisplayID(resolvedWindow, &displayID);
86-
if (displayError != nil) return displayError;
87+
if (displayError != nil) {
88+
*failure = displayError;
89+
return nil;
90+
}
8791
RunnerHIDFunctions functions = {0};
88-
NSString *failure = RunnerResolveHIDFunctions(&functions);
89-
if (failure != nil) return failure;
92+
NSString *hidError = RunnerResolveHIDFunctions(&functions);
93+
if (hidError != nil) {
94+
*failure = hidError;
95+
return nil;
96+
}
9097
CFTypeRef client = functions.createClient(kCFAllocatorDefault);
91-
if (client == NULL) return @"simulator HID dispatch is unavailable: no event system client";
98+
if (client == NULL) {
99+
*failure = @"simulator HID dispatch is unavailable: no event system client";
100+
return nil;
101+
}
92102
uint64_t sender = 0;
93-
failure = RunnerResolveTouchscreenSender(&functions, client, displayID, &sender);
94-
if (failure != nil) {
103+
NSString *senderError = RunnerResolveTouchscreenSender(&functions, client, displayID, &sender);
104+
if (senderError != nil) {
95105
CFRelease(client);
96-
return failure;
106+
*failure = senderError;
107+
return nil;
97108
}
98-
99-
failure = RunnerDispatchEdgeContact(&functions, client, sender, samples, screenSize);
109+
RunnerSimulatorEdgeTouch *touch = [[self alloc] initWithFunctions:functions client:client sender:sender];
100110
CFRelease(client);
101-
return failure;
111+
return touch;
112+
}
113+
114+
- (instancetype)initWithFunctions:(RunnerHIDFunctions)functions client:(CFTypeRef)client sender:(uint64_t)sender {
115+
if ((self = [super init])) {
116+
_functions = functions;
117+
_client = CFRetain(client);
118+
_sender = sender;
119+
}
120+
return self;
121+
}
122+
123+
- (void)dealloc {
124+
CFRelease(_client);
125+
}
126+
127+
- (NSString * _Nullable)dispatchSamples:(NSArray<NSDictionary<NSString *, NSNumber *> *> *)samples
128+
screenSize:(CGSize)screenSize
129+
touchDown:(BOOL *)touchDown {
130+
*touchDown = NO;
131+
if (samples.count < 2 || screenSize.width <= 0 || screenSize.height <= 0) {
132+
return @"an edge touch needs at least two samples and a measured screen";
133+
}
134+
return RunnerDispatchEdgeContact(&_functions, _client, _sender, samples, screenSize, touchDown);
102135
}
103136

104137
@end

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSurfacePolicy.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ extension CommandType {
1515
.appState, .pasteboardWrite, .activate, .terminate, .targetReset, .shutdown:
1616
return .none
1717
case .tap, .longPress, .drag, .type, .swipe, .scroll, .findText, .querySelector, .readText,
18-
.snapshot, .backInApp, .backSystem, .gesture, .gestureViewport, .appSwitcher,
18+
.snapshot, .backInApp, .gesture, .gestureViewport, .appSwitcher,
1919
.keyboardDismiss, .sequence:
2020
return .appWindows
2121
case .screenshot, .recordStart:

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 2 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -548,24 +548,12 @@ extension RunnerTests {
548548
)
549549
#endif
550550
case .backInApp:
551-
switch try tapInAppBackControl(app: activeApp) {
551+
switch try performInAppBack(app: activeApp) {
552552
case .performed:
553553
return Response(ok: true, data: DataPayload(message: "backInApp"))
554-
case .unavailable:
555-
return Response(
556-
ok: false,
557-
error: ErrorPayload(message: "in-app back control is not available")
558-
)
559-
case .unverified(let error):
560-
// The fallback gesture ran but the display refused to be sampled. Reporting the typed refusal
561-
// keeps an unknown outcome from being laundered into a definitive "no back control" (#2728).
554+
case .refused(let error), .unconfirmed(let error):
562555
return Response(ok: false, error: error)
563556
}
564-
case .backSystem:
565-
if try performSystemBackAction(app: activeApp) {
566-
return Response(ok: true, data: DataPayload(message: "backSystem"))
567-
}
568-
return Response(ok: false, error: ErrorPayload(message: "system back is not available"))
569557
case .home:
570558
pressHomeButton()
571559
return Response(ok: true, data: DataPayload(message: "home"))

0 commit comments

Comments
 (0)