Skip to content

Commit a913cb2

Browse files
authored
fix(ios): keep alert activation out of XCTest's interruption handling (#2843)
* fix(ios): keep alert activation out of XCTest's interruption handling Before every event XCTest looks for SpringBoard banners and alerts over the target and runs its interruption handler on them. For a notification banner the default handler waits up to 15 s for it to leave, so an `alert accept` issued while a banner was visible tapped after the command's deadline and answered ALERT_DEADLINE_EXCEEDED for a button it had in fact pressed (#2546's late tap, from the banner side). Four iOS CI failures of the alert replacement tests carry this trace, including main job 106127169259: banner wait from t=7.98 s, tap at t=15.12 s on a ~15 s deadline. `withBoundedInteractionIdleTimeoutIfSupported(.preEventSkipped)` drops only the idle wait. Alert activation now also sets XCUIApplication's `doesNotHandleUIInterruptions` on the alert's owner app for the one tap. It keeps the XCTest coordinate tap and its post-event settle, which alert verification relies on; the synthesized-event path would have dropped that settle too. The runner host app gains a `--agent-device-alert-banner` fixture that keeps a notification banner over its alert until the alert is answered. The new test fails on the old dispatch with the CI signature (interruption monitors consulted, ALERT_DEADLINE_EXCEEDED after the button was activated) and passes with the opt-out. * test(ios): resolve the alert before the deadline the replacement fixtures assert on Nine iOS CI failures of the alert replacement fixtures answered ALERT_DEADLINE_EXCEEDED with "No alert button was activated": the command deadline also covered resolution, about two dozen accessibility reads before any tap, and a contended hosted simulator spent it all there (job 106268275388 sampled a 0.823 s read, sized a 39.5 s deadline, then spent 39 s resolving). The latency-sampled deadline from #2708 cannot predict a host that slows down mid-command. The fixtures now resolve the alert first under a 60 s allowance and hand it to the command through `alertResolutionOverrideForTesting`, so `timeoutMs` bounds only activation and verification, which is what they prove: the replacement is never activated. That window gets a fixed 30 s. A confirmed answer returns as soon as verification sees the replacement; only the unprovable-replacement fixture pays it in full. The latency sampler and its decomposition test are removed. The busy-app fixture asserted wall-clock success under a fixed 6 s deadline. It now records whether the button was answered while its animation was still in flight, and the test asserts that instead. Reverting the seam to a plain `activateElement` fails it with "Answered after the app went idle". * test(ios): keep the busy alert fixture busy for the test's whole alert budget The fixture's fixed 20 s backstop could stop the in-flight animation while a contended host was still resolving the alert under the 60 s allowance, so a correct activation read "Answered after the app went idle". The test now passes the backstop as resolution allowance plus activation deadline.
1 parent db4feb9 commit a913cb2

5 files changed

Lines changed: 202 additions & 105 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,10 +189,10 @@ jobs:
189189
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertAcceptDoesNotActivateAReplacementWithASharedButton \
190190
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDismissDoesNotActivateAReplacementWithTheSameTitle \
191191
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt \
192-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertReplacementCommandDeadlineScalesWithMeasuredLatency \
193192
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched \
194193
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertHittableProbeCompletingAfterDeadlineLeavesTheOriginalUntouched \
195194
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertActivationIgnoresAnAppThatNeverSettlesBeforeTheDeadline \
195+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertActivationDoesNotWaitOutANotificationBanner \
196196
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSystemModalProbeSliceSharesAndClampsToPlanDeadline \
197197
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDispatchRecoverySkipsBookkeepingWhileXCTestChannelOccupied \
198198
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrain \

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,13 @@
1010
CI it spent its whole 10 s budget there (`wait_capture_stalled` with `captures: 1`). The probe now
1111
joins the running discovery, bounded by the discovery's own deadline, and then observes the launch
1212
as before. A resolution failure other than a pending discovery still ends the probe at once.
13+
- Fixed (ios): `alert accept` and `alert dismiss` no longer land their tap after the command's
14+
deadline when a notification banner is on screen. XCTest checks for SpringBoard banners and alerts
15+
before every event and its default handler waited up to 15 s for a banner to leave, so the command
16+
answered `ALERT_DEADLINE_EXCEEDED` ("The button was activated once") for a tap that arrived after
17+
the caller was told it had failed (#2546's late tap, from the banner side). Alert activation now
18+
opts out of XCTest's interruption handling, which also stops that handler from pressing a button of
19+
its own choosing on an unrelated system alert while the command answers the alert it resolved.
1320
- Fixed (ios): a local Simulator snapshot taken through the host AX bridge once again publishes the
1421
geometric `hittable` fact, so `is hittable` and a `hittable:` selector resolve the same controls on
1522
the bridge and the XCTest runner. The snapshot capability table has declared `hittable =

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerApp.m‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -56,31 +56,47 @@ int main(int argc, const char *argv[]) {
5656

5757
#else
5858
#import <UIKit/UIKit.h>
59+
#if TARGET_OS_IOS
60+
#import <UserNotifications/UserNotifications.h>
61+
#endif
5962

6063
@interface AgentDeviceRunnerViewController : UIViewController
6164
@property(nonatomic, strong) UILabel *alertActionStatus;
65+
@property(nonatomic, strong) UILabel *alertActivationBusyAnswer;
6266
@property(nonatomic, assign) NSUInteger firstAlertActions;
6367
@property(nonatomic, assign) NSUInteger replacementAlertActions;
6468
@property(nonatomic, assign) BOOL alertFixtureStarted;
6569
@property(nonatomic, strong) NSTimer *alertActivationBusyBackstop;
70+
@property(nonatomic, strong) NSTimer *alertBannerRepost;
6671
@end
6772

73+
#if TARGET_OS_IOS
74+
@interface AgentDeviceRunnerViewController () <UNUserNotificationCenterDelegate>
75+
@end
76+
#endif
77+
6878
@implementation AgentDeviceRunnerViewController
6979

7080
#if TARGET_OS_IOS
7181
// An animation that never ends is what "busy" looks like to XCTest while it decides whether the app
7282
// may receive an event: the app keeps reporting work in flight, which is the state that cost an alert
7383
// command its whole deadline in #2546. It stops the moment an alert button is answered, since that
7484
// answer is the event the runner is trying to land, and the backstop stops it even when no answer
75-
// arrives so a regressed run finishes rather than waiting out XCTest's own timeout. A layer
85+
// arrives so a regressed run finishes rather than waiting out XCTest's own timeout. The test passes
86+
// the backstop after `--agent-device-alert-activation-busy`, sized to outlast its whole resolution
87+
// and activation budget, so a slow host cannot end the busy state before the answer lands. A layer
7688
// animation on its own is not enough; only a UIView animation counts as in-flight work here.
77-
static NSTimeInterval const kAgentDeviceAlertActivationBusyWindow = 20.0;
89+
static NSTimeInterval AgentDeviceAlertActivationBusyWindow(void) {
90+
NSArray<NSString *> *arguments = NSProcessInfo.processInfo.arguments;
91+
NSUInteger flag = [arguments indexOfObject:@"--agent-device-alert-activation-busy"];
92+
return flag + 1 < arguments.count ? arguments[flag + 1].doubleValue : 0;
93+
}
7894

7995
- (void)startAlertActivationBusy {
8096
if (self.alertActivationBusyBackstop != nil) {
8197
return;
8298
}
83-
self.alertActivationBusyBackstop = [NSTimer scheduledTimerWithTimeInterval:kAgentDeviceAlertActivationBusyWindow
99+
self.alertActivationBusyBackstop = [NSTimer scheduledTimerWithTimeInterval:AgentDeviceAlertActivationBusyWindow()
84100
target:self
85101
selector:@selector(stopAlertActivationBusy)
86102
userInfo:nil
@@ -101,6 +117,57 @@ - (void)stopAlertActivationBusy {
101117
self.alertActivationBusyBackstop = nil;
102118
}
103119

120+
// A banner from this app, shown over its own alert and re-posted before the previous one expires so
121+
// one is on screen for as long as the first alert is unanswered. XCTest treats such a banner as an
122+
// interruption of every event aimed at the app (#2546's late tap, from the banner side).
123+
- (void)startAlertBannerThen:(dispatch_block_t)presentAlert {
124+
UNUserNotificationCenter *center = UNUserNotificationCenter.currentNotificationCenter;
125+
center.delegate = self;
126+
[center requestAuthorizationWithOptions:UNAuthorizationOptionAlert
127+
completionHandler:^(BOOL granted, NSError *error) {
128+
(void)error;
129+
if (!granted) {
130+
return;
131+
}
132+
dispatch_async(dispatch_get_main_queue(), ^{
133+
presentAlert();
134+
[self postAlertBanner];
135+
self.alertBannerRepost = [NSTimer scheduledTimerWithTimeInterval:2.0
136+
target:self
137+
selector:@selector(postAlertBanner)
138+
userInfo:nil
139+
repeats:YES];
140+
});
141+
}];
142+
}
143+
144+
- (void)postAlertBanner {
145+
UNMutableNotificationContent *content = [[UNMutableNotificationContent alloc] init];
146+
content.title = @"Agent Device banner";
147+
content.body = @"Shown over the alert fixture";
148+
UNNotificationRequest *request = [UNNotificationRequest requestWithIdentifier:NSUUID.UUID.UUIDString
149+
content:content
150+
trigger:nil];
151+
[UNUserNotificationCenter.currentNotificationCenter addNotificationRequest:request withCompletionHandler:nil];
152+
}
153+
154+
- (void)stopAlertBanner {
155+
if (self.alertBannerRepost == nil) {
156+
return;
157+
}
158+
[self.alertBannerRepost invalidate];
159+
self.alertBannerRepost = nil;
160+
[UNUserNotificationCenter.currentNotificationCenter removeAllDeliveredNotifications];
161+
}
162+
163+
- (void)userNotificationCenter:(UNUserNotificationCenter *)center
164+
willPresentNotification:(UNNotification *)notification
165+
withCompletionHandler:(void (^)(UNNotificationPresentationOptions options))completionHandler {
166+
(void)center;
167+
(void)notification;
168+
completionHandler(UNNotificationPresentationOptionBanner);
169+
}
170+
104171

105172
- (void)updateAlertActionStatus {
106173
self.alertActionStatus.text = [NSString stringWithFormat:@"First actions: %lu; replacement actions: %lu",
@@ -123,7 +190,12 @@ - (void)presentAlertFixtureReplacement:(BOOL)replacement {
123190
? UIAlertActionStyleCancel : UIAlertActionStyleDefault;
124191
[alert addAction:[UIAlertAction actionWithTitle:buttonTitle style:style handler:^(UIAlertAction *action) {
125192
(void)action;
193+
if (!replacement) {
194+
self.alertActivationBusyAnswer.text = self.alertActivationBusyBackstop != nil
195+
? @"Answered while busy" : @"Answered after the app went idle";
196+
}
126197
[self stopAlertActivationBusy];
198+
[self stopAlertBanner];
127199
if (replacement) {
128200
self.replacementAlertActions += 1;
129201
} else {
@@ -145,9 +217,16 @@ - (void)viewDidAppear:(BOOL)animated {
145217
if (!self.alertFixtureStarted &&
146218
[NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-replacement-regression"]) {
147219
self.alertFixtureStarted = YES;
148-
[self presentAlertFixtureReplacement:NO];
149-
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-activation-busy"]) {
150-
[self startAlertActivationBusy];
220+
dispatch_block_t presentAlert = ^{
221+
[self presentAlertFixtureReplacement:NO];
222+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-activation-busy"]) {
223+
[self startAlertActivationBusy];
224+
}
225+
};
226+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-banner"]) {
227+
[self startAlertBannerThen:presentAlert];
228+
} else {
229+
presentAlert();
151230
}
152231
}
153232
}
@@ -185,6 +264,19 @@ - (void)viewDidLoad {
185264
[self updateAlertActionStatus];
186265
}
187266

267+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-activation-busy"]) {
268+
UILabel *busyAnswer = [[UILabel alloc] init];
269+
busyAnswer.text = @"Unanswered";
270+
busyAnswer.accessibilityIdentifier = @"agent-device-alert-busy-answer";
271+
busyAnswer.translatesAutoresizingMaskIntoConstraints = NO;
272+
[self.view addSubview:busyAnswer];
273+
[NSLayoutConstraint activateConstraints:@[
274+
[busyAnswer.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
275+
[busyAnswer.topAnchor constraintEqualToAnchor:label.bottomAnchor constant:24],
276+
]];
277+
self.alertActivationBusyAnswer = busyAnswer;
278+
}
279+
188280
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-regression"]) {
189281
UITextField *textField = [[UITextField alloc] init];
190282
textField.accessibilityIdentifier = @"agent-device-hardware-keyboard-input";

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Alert.swift‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,15 +76,17 @@ extension RunnerTests {
7676
buttonFrame.origin.x, buttonFrame.origin.y, buttonFrame.size.width, buttonFrame.size.height,
7777
buttonFrame.midX, buttonFrame.midY
7878
)
79-
// The hittable read above is this activation's readiness gate, so XCTest's pre-synthesis wait
80-
// adds nothing and can cost more than the command has: the tap would land after the deadline
79+
// The hittable read above is this activation's readiness gate, so XCTest's pre-synthesis waits
80+
// add nothing and can cost more than the command has: the tap would land after the deadline
8181
// expired and the alert would be answered by a button the caller was told nothing about
8282
// (#2546). The post-tap settle stays, because the verification below reads the alert this tap
8383
// replaces; an alert that dismisses and presents an identical replacement passes through a
8484
// window with no alert, and a first read landing there reports a dismissal nothing proved.
8585
var outcome = RunnerInteractionOutcome.performed
86-
withBoundedInteractionIdleTimeoutIfSupported(alert.ownerApp, waits: .preEventSkipped) {
87-
outcome = activateElement(app: alert.ownerApp, element: button, action: "alert \(action)")
86+
withUIInterruptionHandlingDisabledIfSupported(alert.ownerApp) {
87+
withBoundedInteractionIdleTimeoutIfSupported(alert.ownerApp, waits: .preEventSkipped) {
88+
outcome = activateElement(app: alert.ownerApp, element: button, action: "alert \(action)")
89+
}
8890
}
8991
if let response = unsupportedResponse(for: outcome) {
9092
return response
@@ -114,6 +116,22 @@ extension RunnerTests {
114116
)
115117
}
116118

119+
/// Before each event XCTest looks for SpringBoard elements over the target and hands them to its
120+
/// interruption handler, which waits up to 15 s for a notification banner to leave and taps a
121+
/// button of its own choosing on any other alert. An alert command answers exactly the alert it
122+
/// resolved, with the button it chose, before its deadline, so it opts out of both.
123+
private func withUIInterruptionHandlingDisabledIfSupported(_ target: XCUIApplication, operation: () -> Void) {
124+
let key = "doesNotHandleUIInterruptions"
125+
guard target.responds(to: NSSelectorFromString("setDoesNotHandleUIInterruptions:")) else {
126+
operation()
127+
return
128+
}
129+
let previous = target.value(forKey: key) as? NSNumber
130+
target.setValue(true, forKey: key)
131+
defer { target.setValue(previous?.boolValue ?? false, forKey: key) }
132+
operation()
133+
}
134+
117135
private func runnerAlert(_ modal: ResolvedBlockingSystemModal) -> RunnerAlert? {
118136
let buttons = modal.actions.filter { isEnabledElement($0) }
119137
guard !buttons.isEmpty else {

0 commit comments

Comments
 (0)