Skip to content

Commit c186363

Browse files
authored
fix(ios): keyboard-dismiss content settle race (#1542) — partial, defect 2 needs a decision (#1559)
* fix(ios): wait for post-dismiss content settle before the next gesture (#1542) Dismissing the keyboard can trigger the app's own ScrollView content-offset correction (e.g. releasing the inset it grew to keep a focused field above the keyboard). That correction is a separate, unsynchronized animation that `keyboard.waitForNonExistence` knows nothing about — the keyboard AX element can disappear well before the app visually settles. The very next command is frequently a synthesized, AX-free drag (scroll/gesture, kept AX-free so it still works under #1105-family AX degradation), which has no XCTest quiescence wait of its own, so it can land mid-animation and net to zero — the "scroll does nothing" symptom on the Form screen's checkout-form.ad leg. Add a bounded, AX-free screenshot-stability wait to dismissKeyboard() so the runner only returns once the screen has actually stopped changing (or a generous cap elapses). The stopping decision is a pure function (runnerScreenshotStabilitySettled) covered by unit tests under AGENT_DEVICE_RUNNER_UNIT_TESTS; the surrounding capture/sleep loop is the thin, untestable I/O shell around it. Live-verified on iPhone 17 Pro / iOS 26.2: the scroll now visually lands at the correct position (confirmed via screen-recording frame correlation) instead of leaving content at its pre-scroll offset. Not a full fix for #1542: the checkout-form.ad corpus leg still fails at the same step, now because the daemon's shared post-gesture snapshot stabilization (src/daemon/post-gesture-stabilization.ts) can read a stale-but-internally-consistent AX tree after the AX-free scroll and mistake "unchanged across polls" for "settled", so the following click's off-screen guard sees pre-scroll node positions. That is a cross-platform, cross-command stabilization semantics change and needs a design decision, not a unilateral fix here — see the PR description. * ci(ios): execute the screenshot-stability runner tests (#1559 review)
1 parent 480e388 commit c186363

2 files changed

Lines changed: 139 additions & 1 deletion

File tree

‎.github/workflows/ios.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,13 @@ jobs:
100100
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testDispatchRecoverySkipsBookkeepingWhileXCTestChannelOccupied \
101101
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrain \
102102
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBoundedSystemModalProbeTimeoutRecoversThenReleasesOnDrainForSnapshotRaw \
103-
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds
103+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSnapshotTraversalIdentityPreservesSameOriginNodesWithDifferentBounds \
104+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledNeedsEnoughSamples \
105+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledTrueWhenWindowMatches \
106+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledFalseOnMidWindowMismatch \
107+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledFalseOnFailedCapture \
108+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledOnlyLooksAtTheTrailingWindow \
109+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testRunnerScreenshotStabilitySettledRejectsDegenerateRequirement
104110
105111
- name: Preflight iOS runner through public CLI
106112
run: |

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,23 @@ import XCTest
22

33
private enum KeyboardDismissObservationTiming {
44
static let timeout: TimeInterval = 2
5+
// #1542: dismissing the keyboard can trigger the host app's own content-offset
6+
// adjustment (e.g. a ScrollView correcting the inset it grew to keep a focused
7+
// field above the keyboard). That adjustment is a separate, unsynchronized
8+
// animation the keyboard's own `waitForNonExistence` above knows nothing about:
9+
// the keyboard AX element can vanish well before the app settles. The very next
10+
// command is frequently a synthesized, AX-free drag (scroll/gesture — kept
11+
// AX-free so it still works under #1105-family AX degradation), which has no
12+
// XCTest quiescence wait of its own, so it can land mid-animation and net to
13+
// zero. These bounds cap the settle wait this trades in.
14+
// A spring-driven content-offset correction can pass through a near-zero-
15+
// velocity inflection (the top of an overshoot) that two adjacent samples
16+
// alone cannot distinguish from true rest. Requiring 3 consecutive matching
17+
// samples demands ~2 full sample intervals of actual stillness before
18+
// stopping early, so a momentary pause mid-animation cannot look settled.
19+
static let settleTimeout: TimeInterval = 2.0
20+
static let settleSampleInterval: TimeInterval = 0.15
21+
static let settleRequiredConsecutiveMatches: Int = 3
522
}
623

724
extension RunnerTests {
@@ -24,6 +41,11 @@ extension RunnerTests {
2441
#else
2542
if tapKeyboardDismissControl(app: app) {
2643
_ = keyboard.waitForNonExistence(timeout: KeyboardDismissObservationTiming.timeout)
44+
waitForScreenshotStability(
45+
timeout: KeyboardDismissObservationTiming.settleTimeout,
46+
sampleInterval: KeyboardDismissObservationTiming.settleSampleInterval,
47+
requiredConsecutiveMatches: KeyboardDismissObservationTiming.settleRequiredConsecutiveMatches
48+
)
2749
let visible = isKeyboardVisible(app: app)
2850
return (wasVisible: true, dismissed: !visible, visible: visible)
2951
}
@@ -32,6 +54,35 @@ extension RunnerTests {
3254
#endif
3355
}
3456

57+
// AX-free on purpose (screenshot bytes, not the accessibility tree) so it holds
58+
// under the same AX degradation the synthesized gesture lane is built to survive.
59+
// Bounded and self-terminating: returns as soon as `requiredConsecutiveMatches`
60+
// consecutive samples match, so an already-settled screen (the common case)
61+
// pays close to nothing. The stopping decision itself
62+
// (`runnerScreenshotStabilitySettled`) is a pure function of the samples taken
63+
// so far, so it is unit-testable without a real screenshot pipeline; this loop
64+
// is the thin, untestable I/O shell around it.
65+
private func waitForScreenshotStability(
66+
timeout: TimeInterval,
67+
sampleInterval: TimeInterval,
68+
requiredConsecutiveMatches: Int
69+
) {
70+
let deadline = Date().addingTimeInterval(timeout)
71+
var samples: [Data?] = [screenshotFingerprintForStabilityCheck()]
72+
while Date() < deadline {
73+
sleepFor(sampleInterval)
74+
samples.append(screenshotFingerprintForStabilityCheck())
75+
if runnerScreenshotStabilitySettled(samples, requiredConsecutiveMatches: requiredConsecutiveMatches) {
76+
return
77+
}
78+
}
79+
}
80+
81+
private func screenshotFingerprintForStabilityCheck() -> Data? {
82+
guard let image = captureRunnerFrame() else { return nil }
83+
return runnerPngData(for: image)
84+
}
85+
3586
func pressKeyboardReturn(app: XCUIApplication) -> (wasVisible: Bool, pressed: Bool, visible: Bool) {
3687
#if os(tvOS)
3788
return (wasVisible: false, pressed: pressTvRemote(.select), visible: false)
@@ -182,3 +233,84 @@ extension RunnerTests {
182233
return frame.intersects(keyboardFrame) || abs(frame.maxY - keyboardFrame.minY) <= 80
183234
}
184235
}
236+
237+
/// True once the `requiredConsecutiveMatches` most recent samples are all present
238+
/// and byte-identical — i.e. the screen held still across that whole run of
239+
/// polls, not just the last two. A spring-driven content-offset correction can
240+
/// pass through a near-zero-velocity inflection that two adjacent samples alone
241+
/// cannot distinguish from true rest; requiring a longer run of matches demands
242+
/// real elapsed stillness before stopping early. `nil` entries (a screenshot
243+
/// capture that failed) never count toward a match, so a flaky capture cannot
244+
/// look like stability; the caller's bounded loop still terminates on its
245+
/// deadline regardless.
246+
func runnerScreenshotStabilitySettled(
247+
_ samples: [Data?],
248+
requiredConsecutiveMatches: Int
249+
) -> Bool {
250+
guard requiredConsecutiveMatches >= 2, samples.count >= requiredConsecutiveMatches else {
251+
return false
252+
}
253+
let window = samples.suffix(requiredConsecutiveMatches)
254+
guard let first = window.first, let firstData = first else { return false }
255+
return window.allSatisfy { $0 == firstData }
256+
}
257+
258+
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
259+
extension RunnerTests {
260+
func testRunnerScreenshotStabilitySettledNeedsEnoughSamples() {
261+
XCTAssertFalse(runnerScreenshotStabilitySettled([], requiredConsecutiveMatches: 3))
262+
XCTAssertFalse(runnerScreenshotStabilitySettled([Data([1])], requiredConsecutiveMatches: 3))
263+
let frame = Data([1, 2, 3])
264+
XCTAssertFalse(
265+
runnerScreenshotStabilitySettled([frame, frame], requiredConsecutiveMatches: 3)
266+
)
267+
}
268+
269+
func testRunnerScreenshotStabilitySettledTrueWhenWindowMatches() {
270+
let frame = Data([1, 2, 3])
271+
XCTAssertTrue(
272+
runnerScreenshotStabilitySettled([Data([9]), frame, frame, frame], requiredConsecutiveMatches: 3)
273+
)
274+
}
275+
276+
func testRunnerScreenshotStabilitySettledFalseOnMidWindowMismatch() {
277+
// A momentary pause (two matching samples) followed by resumed movement
278+
// must not read as settled: the 3-sample window still spans the mismatch.
279+
let frame = Data([1, 2, 3])
280+
let moved = Data([4, 5, 6])
281+
XCTAssertFalse(
282+
runnerScreenshotStabilitySettled([frame, frame, moved], requiredConsecutiveMatches: 3)
283+
)
284+
}
285+
286+
func testRunnerScreenshotStabilitySettledFalseOnFailedCapture() {
287+
// A nil sample (failed screenshot) never counts as a match, even against
288+
// other nils — an unverifiable run must not look "stable".
289+
XCTAssertFalse(runnerScreenshotStabilitySettled([nil, nil, nil], requiredConsecutiveMatches: 3))
290+
let frame = Data([1, 2, 3])
291+
XCTAssertFalse(
292+
runnerScreenshotStabilitySettled([frame, frame, nil], requiredConsecutiveMatches: 3)
293+
)
294+
}
295+
296+
func testRunnerScreenshotStabilitySettledOnlyLooksAtTheTrailingWindow() {
297+
// An older mismatch before the trailing window must not block settlement
298+
// once the required run of most-recent samples agrees.
299+
let frame = Data([9])
300+
XCTAssertTrue(
301+
runnerScreenshotStabilitySettled(
302+
[Data([1]), Data([2]), frame, frame, frame],
303+
requiredConsecutiveMatches: 3
304+
)
305+
)
306+
}
307+
308+
func testRunnerScreenshotStabilitySettledRejectsDegenerateRequirement() {
309+
// Fewer than 2 required matches would make any single sample "settled" —
310+
// guard against a misconfigured caller rather than silently no-op the wait.
311+
XCTAssertFalse(
312+
runnerScreenshotStabilitySettled([Data([1])], requiredConsecutiveMatches: 1)
313+
)
314+
}
315+
}
316+
#endif

0 commit comments

Comments
 (0)