Skip to content

Commit 55d02e9

Browse files
authored
fix(runner): bound XCTest's idle wait around alert activation (#2610)
* fix(runner): bound XCTest's idle wait around alert activation An alert button is read as hittable before it is tapped, so XCTest's own wait for the app to idle before synthesizing that tap adds nothing the runner has not already checked. Its default outlives the command, so the tap lands after the caller was told the alert timed out (#2546). The helper that bounds that wait was named for the scroll path that owned it; gesture, type and swipe already went through it, and alert activation now does too, so it is named for the wait it bounds. Each caller states which of XCTest's two waits it gives up: the gesture, type and swipe paths skip both, because their next step is a poll of their own, while alert activation drops only the pre-event wait. Its verification reads the alert this tap replaced, and an alert that dismisses in order to present an identical replacement passes through a moment with no alert at all — a first read landing in that moment reported a dismissal nothing had proved. * test(ios-runner): cover alert activation against an app that never idles Reverting the alert path to a plain `activateElement` kept every test green, so the #2546 failure had no guard: the deadline tests returned before activation, and the ones that reached it used a 10 s timeout on a fixture that had already settled. The fixture now animates without end until an alert button is answered, which is the state that made XCTest hold the event past the caller's deadline in the field. A test that reaches activation there and is given 6 s reports what actually happens: the response comes back in well under a second of activation, and if it ever reports `ALERT_DEADLINE_EXCEEDED` again the fixture says whether a button was activated behind that answer. Reverting the seam costs the run 18 s and produces exactly the original defect — `ALERT_DEADLINE_EXCEEDED` alongside `First actions: 1`. Only a `UIView` animation counts here: a repeating main-thread timer and a `CABasicAnimation` both left the app looking idle and the reverted path still passed. * test(ios-runner): branch the busy-app assertion on what the command answered The label check ran twice over the same response: it always expected `First actions: 1`, and the block after it expected `First actions: 0` only when the answer was a deadline failure, so the second could never pass and a correct refusal failed anyway. Now the assertion follows the answer — an accepted alert must show the button press, a refused one must not — and the comment said "main thread busy" where the fixture runs a repeating animation with the main thread free. The test also joins the targeted list in `.github/workflows/ios.yml`, which is where the alert regressions it guards are selected.
1 parent 7f49312 commit 55d02e9

7 files changed

Lines changed: 127 additions & 9 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,7 @@ jobs:
187187
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt \
188188
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched \
189189
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertHittableProbeCompletingAfterDeadlineLeavesTheOriginalUntouched \
190+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertActivationIgnoresAnAppThatNeverSettlesBeforeTheDeadline \
190191
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSystemModalProbeSliceSharesAndClampsToPlanDeadline \
191192
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDispatchRecoverySkipsBookkeepingWhileXCTestChannelOccupied \
192193
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrain \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerApp.m‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,11 +62,46 @@ @interface AgentDeviceRunnerViewController : UIViewController
6262
@property(nonatomic, assign) NSUInteger firstAlertActions;
6363
@property(nonatomic, assign) NSUInteger replacementAlertActions;
6464
@property(nonatomic, assign) BOOL alertFixtureStarted;
65+
@property(nonatomic, strong) NSTimer *alertActivationBusyBackstop;
6566
@end
6667

6768
@implementation AgentDeviceRunnerViewController
6869

6970
#if TARGET_OS_IOS
71+
// An animation that never ends is what "busy" looks like to XCTest while it decides whether the app
72+
// may receive an event: the app keeps reporting work in flight, which is the state that cost an alert
73+
// command its whole deadline in #2546. It stops the moment an alert button is answered, since that
74+
// answer is the event the runner is trying to land, and the backstop stops it even when no answer
75+
// arrives so a regressed run finishes rather than waiting out XCTest's own timeout. A layer
76+
// animation on its own is not enough; only a UIView animation counts as in-flight work here.
77+
static NSTimeInterval const kAgentDeviceAlertActivationBusyWindow = 20.0;
78+
79+
- (void)startAlertActivationBusy {
80+
if (self.alertActivationBusyBackstop != nil) {
81+
return;
82+
}
83+
self.alertActivationBusyBackstop = [NSTimer scheduledTimerWithTimeInterval:kAgentDeviceAlertActivationBusyWindow
84+
target:self
85+
selector:@selector(stopAlertActivationBusy)
86+
userInfo:nil
87+
repeats:NO];
88+
[UIView animateWithDuration:0.4
89+
delay:0
90+
options:(UIViewAnimationOptionRepeat | UIViewAnimationOptionAutoreverse)
91+
animations:^{
92+
self.alertActionStatus.transform = CGAffineTransformMakeTranslation(0, 8);
93+
}
94+
completion:nil];
95+
}
96+
97+
- (void)stopAlertActivationBusy {
98+
[self.alertActionStatus.layer removeAllAnimations];
99+
self.alertActionStatus.transform = CGAffineTransformIdentity;
100+
[self.alertActivationBusyBackstop invalidate];
101+
self.alertActivationBusyBackstop = nil;
102+
}
103+
104+
70105
- (void)updateAlertActionStatus {
71106
self.alertActionStatus.text = [NSString stringWithFormat:@"First actions: %lu; replacement actions: %lu",
72107
(unsigned long)self.firstAlertActions,
@@ -88,6 +123,7 @@ - (void)presentAlertFixtureReplacement:(BOOL)replacement {
88123
? UIAlertActionStyleCancel : UIAlertActionStyleDefault;
89124
[alert addAction:[UIAlertAction actionWithTitle:buttonTitle style:style handler:^(UIAlertAction *action) {
90125
(void)action;
126+
[self stopAlertActivationBusy];
91127
if (replacement) {
92128
self.replacementAlertActions += 1;
93129
} else {
@@ -110,6 +146,9 @@ - (void)viewDidAppear:(BOOL)animated {
110146
[NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-replacement-regression"]) {
111147
self.alertFixtureStarted = YES;
112148
[self presentAlertFixtureReplacement:NO];
149+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-alert-activation-busy"]) {
150+
[self startAlertActivationBusy];
151+
}
113152
}
114153
}
115154
#endif

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Alert.swift‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,16 @@ extension RunnerTests {
6868
guard waitUntilAlertButtonHittable(button, deadline: deadline) else {
6969
return alertVerificationResponse(.timedOut, action: action, activated: false)
7070
}
71-
let outcome = activateElement(app: alert.ownerApp, element: button, action: "alert \(action)")
71+
// The hittable read above is this activation's readiness gate, so XCTest's pre-synthesis wait
72+
// adds nothing and can cost more than the command has: the tap would land after the deadline
73+
// expired and the alert would be answered by a button the caller was told nothing about
74+
// (#2546). The post-tap settle stays, because the verification below reads the alert this tap
75+
// replaces; an alert that dismisses and presents an identical replacement passes through a
76+
// window with no alert, and a first read landing there reports a dismissal nothing proved.
77+
var outcome = RunnerInteractionOutcome.performed
78+
withBoundedInteractionIdleTimeoutIfSupported(alert.ownerApp, waits: .preEventSkipped) {
79+
outcome = activateElement(app: alert.ownerApp, element: button, action: "alert \(action)")
80+
}
7281
if let response = unsupportedResponse(for: outcome) {
7382
return response
7483
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,9 @@ extension RunnerTests {
9797
var outcome = RunnerInteractionOutcome.performed
9898
let timing = measureGesture {
9999
if idleTimeout {
100-
withTemporaryScrollIdleTimeoutIfSupported(app) { outcome = action() }
100+
withBoundedInteractionIdleTimeoutIfSupported(app, waits: .bothSkipped) {
101+
outcome = action()
102+
}
101103
} else {
102104
outcome = action()
103105
}
@@ -1975,7 +1977,7 @@ extension RunnerTests {
19751977
return Response(ok: true, data: DataPayload(message: "remote pressed"))
19761978
case .type:
19771979
var response: Response?
1978-
withTemporaryScrollIdleTimeoutIfSupported(activeApp) {
1980+
withBoundedInteractionIdleTimeoutIfSupported(activeApp, waits: .bothSkipped) {
19791981
response = executeTypeCommand(activeApp: activeApp, command: command)
19801982
}
19811983
return response ?? Response(ok: false, error: ErrorPayload(message: "type produced no response"))
@@ -1987,7 +1989,7 @@ extension RunnerTests {
19871989
// keeps raw measureGesture and only routes the success payload through gestureResponse.
19881990
var executedFrame: DragVisualizationFrame?
19891991
let timing = measureGesture {
1990-
withTemporaryScrollIdleTimeoutIfSupported(activeApp) {
1992+
withBoundedInteractionIdleTimeoutIfSupported(activeApp, waits: .bothSkipped) {
19911993
executedFrame = swipe(app: activeApp, direction: direction)
19921994
}
19931995
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Lifecycle.swift‎

Lines changed: 36 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,24 @@ func runnerCGImage(from image: RunnerImage) -> CGImage? {
2121
#endif
2222
}
2323

24+
/// Which of XCTest's two waits around one synthesized event a caller gives up.
25+
///
26+
/// The two waits answer to different callers. The pre-event wait is what #2546 bounds: the runner
27+
/// has already decided the interaction may be synthesized, and XCTest's default wait for the app to
28+
/// idle outlives a bounded command, so the event lands after the caller was told it failed. The
29+
/// post-event wait is the settle margin before the runner reads the app back, and a caller whose
30+
/// verdict is that next read cannot give it up.
31+
enum RunnerInteractionIdleWaits {
32+
/// Neither wait, for a caller whose next step is its own poll rather than a verdict read off this
33+
/// event: a scroll re-checks its own content, a text field was already located, a swipe has
34+
/// nothing to verify.
35+
case bothSkipped
36+
/// Pre-event wait dropped, post-event quiescence kept under the same bound, for a caller whose
37+
/// verdict is the state this event produced. Alert verification reads the alert the tap replaced,
38+
/// and a read taken mid-transition finds no alert and reports a dismissal nothing proved.
39+
case preEventSkipped
40+
}
41+
2442
extension RunnerTests {
2543
// MARK: - Recording
2644

@@ -209,8 +227,14 @@ extension RunnerTests {
209227
return target
210228
}
211229

212-
func withTemporaryScrollIdleTimeoutIfSupported(
230+
/// Bounds what XCTest waits around one synthesized event instead of letting it spend a command's
231+
/// whole deadline, keeping whichever settle the caller named in `waits`. Callers gate the
232+
/// interaction themselves first (a scroll needs no extra wait, a text field is located, an alert
233+
/// button is read as hittable), which is what the dropped pre-event wait replaces rather than a
234+
/// check the runner skips (#2546).
235+
func withBoundedInteractionIdleTimeoutIfSupported(
213236
_ target: XCUIApplication,
237+
waits: RunnerInteractionIdleWaits,
214238
operation: () -> Void
215239
) {
216240
let setter = NSSelectorFromString("setWaitForIdleTimeout:")
@@ -219,19 +243,20 @@ extension RunnerTests {
219243
? (target.value(forKey: "waitForIdleTimeout") as? NSNumber)
220244
: nil
221245
if supportsWaitForIdleTimeout {
222-
target.setValue(scrollInteractionIdleTimeoutDefault, forKey: "waitForIdleTimeout")
246+
target.setValue(interactionIdleTimeoutDefault, forKey: "waitForIdleTimeout")
223247
}
224248
defer {
225249
if let previous {
226250
target.setValue(previous.doubleValue, forKey: "waitForIdleTimeout")
227251
}
228252
}
229-
performWithQuiescenceSkippedIfSupported(target, operation: operation)
253+
performWithQuiescenceSkippedIfSupported(target, waits: waits, operation: operation)
230254
}
231255

232256
// Some apps never report post-gesture quiescence, even after XCTest has synthesized the event.
233257
private func performWithQuiescenceSkippedIfSupported(
234258
_ target: XCUIApplication,
259+
waits: RunnerInteractionIdleWaits,
235260
operation: () -> Void
236261
) {
237262
let selector = NSSelectorFromString("_performWithInteractionOptions:block:")
@@ -252,12 +277,19 @@ extension RunnerTests {
252277
)
253278
let skipPreEventQuiescence = UInt(1)
254279
let skipPostEventQuiescence = UInt(2)
280+
let options: UInt
281+
switch waits {
282+
case .bothSkipped:
283+
options = skipPreEventQuiescence | skipPostEventQuiescence
284+
case .preEventSkipped:
285+
options = skipPreEventQuiescence
286+
}
255287
withoutActuallyEscaping(operation) { escapableOperation in
256288
let block: @convention(block) () -> Void = escapableOperation
257289
performWithOptions(
258290
target,
259291
selector,
260-
skipPreEventQuiescence | skipPostEventQuiescence,
292+
options,
261293
block
262294
)
263295
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ final class RunnerTests: XCTestCase {
6666
let retryCooldown: TimeInterval = 0.2
6767
let postSnapshotInteractionDelay: TimeInterval = 0.2
6868
let firstInteractionAfterActivateDelay: TimeInterval = 0.25
69-
let scrollInteractionIdleTimeoutDefault: TimeInterval = 1.0
69+
let interactionIdleTimeoutDefault: TimeInterval = 1.0
7070
let tvRemoteDoublePressDelayDefault: TimeInterval = 0.0
7171
// Keep a periodic XCTest liveness marker in runner.log without flooding long-lived sessions.
7272
let xctestIdleKeepaliveInterval: TimeInterval = 60.0

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,41 @@ extension RunnerTests {
5757
XCTAssertEqual(app.staticTexts["agent-device-alert-actions"].label, "First actions: 0; replacement actions: 0")
5858
}
5959

60+
func testAlertActivationIgnoresAnAppThatNeverSettlesBeforeTheDeadline() throws {
61+
app.launchArguments = [
62+
"--agent-device-alert-replacement-regression",
63+
"--agent-device-alert-activation-busy"
64+
]
65+
app.launch()
66+
defer {
67+
invalidateCachedTarget(reason: "unit_test_cleanup")
68+
app.terminate()
69+
}
70+
XCTAssertTrue(app.alerts.firstMatch.waitForExistence(timeout: appExistenceTimeout))
71+
let alert = try XCTUnwrap(resolveAlert(app: app, deadline: Date().addingTimeInterval(10)))
72+
73+
let deadline = Date().addingTimeInterval(6)
74+
let startedAt = Date()
75+
let response = handleAlert(alert, action: "accept", deadline: deadline)
76+
let elapsed = Date().timeIntervalSince(startedAt)
77+
78+
// The fixture keeps an animation in flight, which is what XCTest waits out before it synthesises
79+
// an event, so this comes back early only because activation refused to wait for an app that has
80+
// no intention of settling (#2546).
81+
XCTAssertLessThan(elapsed, 9, "activation waited \(elapsed)s for a busy app to idle")
82+
XCTAssertTrue(response.ok, String(describing: response.error))
83+
let recordedActions = app.staticTexts["agent-device-alert-actions"].label
84+
if response.ok {
85+
XCTAssertEqual(recordedActions, "First actions: 1; replacement actions: 0")
86+
} else {
87+
XCTAssertEqual(
88+
recordedActions,
89+
"First actions: 0; replacement actions: 0",
90+
"a caller told about an expired deadline must not have a button activated behind it"
91+
)
92+
}
93+
}
94+
6095
private func assertReplacementAlertUntouched(action: String, arguments: [String], confirmed: Bool) throws {
6196
app.launchArguments = ["--agent-device-alert-replacement-regression"] + arguments
6297
app.launch()

0 commit comments

Comments
 (0)