Skip to content

Commit 7e741b7

Browse files
authored
test(ios): size the alert command's deadline to the host's own reads (#2708)
* test(ios): size the alert command's deadline to the host's own reads The three replacement-alert fixtures give the alert command 10000 ms, and that budget only means something in units of what one accessibility read costs the host: the deadline starts at dispatch, and `resolveAlert` spends 24 reads before it chooses a button. On a host that answers a read in 1.7 s the deadline therefore expires mid-resolution and the command reports `ALERT_DEADLINE_EXCEEDED` without ever synthesizing the tap. That is what the nightly's red runs did: 31.2 s, 30.6 s and 40.4 s of reads against the 6.2 s a green run spends. The fixtures now time a read once the alert is up and derive `timeoutMs` from it, floored at the production default and capped at this lane's slowest existing test, so a quiet host asks for exactly what it asked for before and a contended one gets a budget covering the reads it is actually doing. Pinning the derived budget to 250 ms reproduces a red nightly byte for byte, `First actions: 0; replacement actions: 0` included. Refs #2546 * chore(gates): run the alert deadline sizing test on every PR The invariant it pins lives in an iOS-only file, so the nightly was the first lane to see it. Its six sibling fixtures are already in the ios.yml list.
1 parent d688926 commit 7e741b7

2 files changed

Lines changed: 77 additions & 2 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,7 @@ jobs:
185185
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertAcceptDoesNotActivateAReplacementWithASharedButton \
186186
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDismissDoesNotActivateAReplacementWithTheSameTitle \
187187
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertCannotProveAnIdenticalReplacementAndDoesNotActivateIt \
188+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertReplacementCommandDeadlineScalesWithMeasuredLatency \
188189
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched \
189190
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertHittableProbeCompletingAfterDeadlineLeavesTheOriginalUntouched \
190191
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testAlertActivationIgnoresAnAppThatNeverSettlesBeforeTheDeadline \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AlertObservationTests.swift‎

Lines changed: 76 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,23 @@ extension RunnerTests {
1818
)
1919
}
2020

21+
func testAlertReplacementCommandDeadlineScalesWithMeasuredLatency() {
22+
// A quiet host still asks for the production default: 24 reads at 10 ms is a quarter of a second.
23+
XCTAssertEqual(RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: 0.001), 10_000)
24+
XCTAssertEqual(RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: 0.01), 10_000)
25+
// The linear region is where a contended host lands: one second per read buys the 24 reads
26+
// twice over, which is what the red nightly nights needed and 10 s did not cover.
27+
XCTAssertEqual(RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: 1), 48_000)
28+
// Past the cap the fixture gives up rather than dominate the lane, whatever the host costs.
29+
XCTAssertEqual(RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: 2.5), 60_000)
30+
XCTAssertEqual(RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: 300), 60_000)
31+
XCTAssertEqual(
32+
RunnerTests.alertResolutionReadsBeforeActivation,
33+
24,
34+
"the decomposition below is the traced command path; re-trace it before changing a count"
35+
)
36+
}
37+
2138
func testAlertDeadlineBeforeActivationLeavesTheOriginalUntouched() throws {
2239
app.launchArguments = ["--agent-device-alert-replacement-regression"]
2340
app.launch()
@@ -92,6 +109,60 @@ extension RunnerTests {
92109
}
93110
}
94111

112+
/// The accessibility round trips `resolveAlert` spends before a button is chosen, decomposed from
113+
/// the implementation rather than counted off one trace: the blocking-modal probe scans
114+
/// SpringBoard's alert and sheet lists and re-reads the candidate it settles on; the app's own
115+
/// alert list is resolved and then read for existence and frame; `actionableElements` issues one
116+
/// fetch per member of `actionableTypes`; and every candidate then pays `exists`, `isHittable`,
117+
/// `elementType`, `frame`, `label` and `isEnabled` of its own.
118+
static let alertResolutionModalProbeReads = 3
119+
static let alertResolutionAlertRootReads = 3
120+
static let alertResolutionActionableTypeReads = 6
121+
static let alertResolutionButtonCandidates = 2
122+
static let alertResolutionCandidateReads = 6
123+
124+
static let alertResolutionReadsBeforeActivation =
125+
alertResolutionModalProbeReads + alertResolutionAlertRootReads + alertResolutionActionableTypeReads
126+
+ alertResolutionButtonCandidates * alertResolutionCandidateReads
127+
128+
/// The reads that follow resolution — the hittability wait and the first verification observation
129+
/// run on the same channel — plus the allowance for a host that gets slower mid-command.
130+
private static let alertCommandSafetyFactor: TimeInterval = 2
131+
132+
/// No higher than the slowest test this lane already runs (61 s on a green nightly), so a badly
133+
/// starved host cannot make these fixtures the lane's worst contributor. The cap covers the worst
134+
/// night traced so far, which needed 40.4 s of deadline.
135+
private static let alertCommandTimeoutCap: TimeInterval = 60
136+
137+
/// The alert command's deadline for the replacement fixtures.
138+
///
139+
/// `timeoutMs` bounds the whole command, so it is only meaningful in units of what one
140+
/// accessibility read costs this host right now: `RunnerTests+Alert.swift` starts the deadline at
141+
/// dispatch and the resolution above spends all of it reading before anything is activated. The
142+
/// hosted lane runs a read at about 10 ms on a healthy night and about 1.7 s on a red one, which
143+
/// is why the fixed 10 s these fixtures asked for passed on green nights and expired mid-resolution
144+
/// on red ones (runs 35, 36 and 37 needed 31.2 s, 30.6 s and 40.4 s). Deriving the budget from a
145+
/// measured read is `docs/agents/testing.md`'s preferred answer for a timeout that only fails on a
146+
/// contended host; production still defaults to `defaultAlertCommandTimeout` for real callers.
147+
static func alertReplacementCommandTimeoutMs(measuredRoundTrip: TimeInterval) -> Int {
148+
let derived = measuredRoundTrip * Double(alertResolutionReadsBeforeActivation) * alertCommandSafetyFactor
149+
return Int((min(max(derived, defaultAlertCommandTimeout), alertCommandTimeoutCap) * 1000).rounded())
150+
}
151+
152+
/// Samples the read the command is about to pay 24 times, once the alert is up, so the sample comes
153+
/// from the window the command runs in. Two samples and the slower one wins: a single lucky read
154+
/// must not size the budget low, and this is a worst-of measurement, not a mean.
155+
private func measuredAccessibilityRoundTrip() -> TimeInterval {
156+
let recordedActions = app.staticTexts["agent-device-alert-actions"]
157+
var roundTrip = TimeInterval(0.001)
158+
for _ in 0..<2 {
159+
let startedAt = Date()
160+
_ = recordedActions.label
161+
roundTrip = max(roundTrip, Date().timeIntervalSince(startedAt))
162+
}
163+
return roundTrip
164+
}
165+
95166
private func assertReplacementAlertUntouched(action: String, arguments: [String], confirmed: Bool) throws {
96167
app.launchArguments = ["--agent-device-alert-replacement-regression"] + arguments
97168
app.launch()
@@ -100,11 +171,14 @@ extension RunnerTests {
100171
app.terminate()
101172
}
102173
XCTAssertTrue(app.alerts.firstMatch.waitForExistence(timeout: appExistenceTimeout))
174+
let measuredRoundTrip = measuredAccessibilityRoundTrip()
175+
let timeoutMs = RunnerTests.alertReplacementCommandTimeoutMs(measuredRoundTrip: measuredRoundTrip)
103176
let command = try runnerCommandFixture(
104-
#"{"command":"alert","commandId":"alert-replacement","action":"\#(action)","timeoutMs":10000}"#
177+
#"{"command":"alert","commandId":"alert-replacement","action":"\#(action)","timeoutMs":\#(timeoutMs)}"#
105178
)
106179
let response = try executeOnMainPrepared(command: command, activeApp: app)
107-
XCTAssertEqual(response.ok, confirmed, String(describing: response.error))
180+
let budget = "timeoutMs \(timeoutMs) from a \(String(format: "%.3f", measuredRoundTrip))s read"
181+
XCTAssertEqual(response.ok, confirmed, "\(budget): \(String(describing: response.error))")
108182
if !confirmed { XCTAssertEqual(response.error?.code, "ALERT_DEADLINE_EXCEEDED") }
109183
XCTAssertTrue(app.alerts.firstMatch.exists, "the replacement must remain visible")
110184
XCTAssertEqual(app.staticTexts["agent-device-alert-actions"].label, "First actions: 1; replacement actions: 0")

0 commit comments

Comments
 (0)