Skip to content

Commit 16e5b9c

Browse files
committed
fix(ios): grant the text-entry commit wait time against progress
The synthesized commit wait used a flat 3s deadline, which cannot tell a throttled simulator input pipeline (characters keep landing, slowly) from a wedged one (nothing lands) — it condemned both at the same instant and reported TEXT_INPUT_COMMIT_NOT_OBSERVED over a `type`/`fill` that was still working, on branches touching no iOS code. SynthesizedCommitBudget grants time against progress instead: while the observed value's expected-prefix grows — the same length-only evidence logCommitCadence already emits — the wait continues, up to a 10s ceiling. A pipeline making no progress expires at exactly the 3s the flat deadline used, so a wedge is condemned no later than before. It is a reference type, and the observe/expire coupling carries a structural guard, because as a struct that coupling would rest on Swift boxing one captured var and could revert to the flat deadline silently. Text-entry readiness' hardware-keyboard fallback also stops returning a possibly-unfocused element after 0.35s of "no software keyboard seen"; it now returns only on confirmed focus of the target and re-arms otherwise. And the keyboard-hidden precondition of testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden skips rather than fails, so an environment flip cannot read as a product regression. The issue's remaining ask — pinning the simulator keyboard preference — is deliberately not done: measured on a dedicated simulator, per-device ConnectHardwareKeyboard makes no difference to a headless `simctl boot`, which always shows the software keyboard. See the PR body for the A/B. Refs #1874 — not a closing keyword on purpose. This is a mitigation; the unidentified simulator input-throttle mechanism that issue tracks is untouched here, so it stays open.
1 parent 72cae2b commit 16e5b9c

8 files changed

Lines changed: 323 additions & 14 deletions

‎.github/workflows/ios.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,7 @@ jobs:
140140
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSinglePointerFlingFallsBackToXCTestCoordinateDragWhenPrivateSynthesisFails \
141141
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testTypeWithoutResolvedInputReturnsTypedFailureBeforeDispatchingText \
142142
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden \
143+
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testHardwareKeyboardResponderConfirmsItsOwnKeyboardFocus \
143144
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testPenalizedCoordinateTapOnNonTextControlDoesNotAuthorizeBareType \
144145
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand \
145146
-only-testing:AgentDeviceRunnerUITests/RunnerTests/testSynthesizedTextCommitProgressWalksExpectedPrefixOnly \

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 56 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -379,9 +379,13 @@ extension RunnerTests {
379379
)
380380
let tapResponse = try executeOnMainPrepared(command: tapCommand, activeApp: app)
381381
XCTAssertTrue(tapResponse.ok, String(describing: tapResponse.error))
382-
XCTAssertFalse(
382+
// A precondition, not a product claim. The fixture's empty `inputView` is what keeps the
383+
// keyboard down, but nothing in this bundle owns the simulator's own keyboard settings, so an
384+
// ambient flip that raised one here would be an environment fact — and reporting it as a
385+
// failed assertion is what made this read as a product regression on unrelated PRs (#1874).
386+
try XCTSkipIf(
383387
isKeyboardVisible(app: app),
384-
"the test must exercise a focused responder with the software keyboard hidden"
388+
"software keyboard is up: this simulator cannot exercise the hidden-keyboard responder path"
385389
)
386390

387391
let failureCountBefore = currentXCTestFailureCount()
@@ -407,6 +411,56 @@ extension RunnerTests {
407411
XCTAssertEqual(String(describing: textField.value ?? ""), "hardware-keyboard")
408412
}
409413

414+
// `waitForTextEntryReadiness`'s hardware-keyboard fallback returns early only on confirmed
415+
// focus (#1874), and `keyboardFocusConfirmed` reads that from the app-wide focus predicate this
416+
// bundle otherwise refuses to trust. Two XCTest facts it rests on, neither a repository
417+
// invariant: the predicate reports a responder that shows NO software keyboard at all, and it
418+
// names the element well enough to tell the tapped field from another one. The fixture field is
419+
// the exact shape the fallback exists for — a real responder with an empty `inputView` — so this
420+
// is where both are observable. If either regressed, readiness would silently stop taking the
421+
// fallback and spend the full readinessTimeout on every hardware-keyboard field, which no other
422+
// assertion would notice.
423+
func testHardwareKeyboardResponderConfirmsItsOwnKeyboardFocus() throws {
424+
app.launchArguments = ["--agent-device-text-entry-regression"]
425+
app.launch()
426+
defer {
427+
invalidateCachedTarget(reason: "unit_test_cleanup")
428+
app.terminate()
429+
}
430+
XCTAssertTrue(app.waitForExistence(timeout: appExistenceTimeout))
431+
432+
let textField = app.textFields["agent-device-hardware-keyboard-input"]
433+
XCTAssertTrue(textField.waitForExistence(timeout: appExistenceTimeout))
434+
let otherElement = app.staticTexts["Agent Device Runner"]
435+
XCTAssertTrue(otherElement.waitForExistence(timeout: appExistenceTimeout))
436+
XCTAssertFalse(
437+
keyboardFocusConfirmed(app: app, element: textField),
438+
"an untapped field must not confirm focus, or the fallback would fire immediately"
439+
)
440+
441+
let tapCommand = try runnerCommandFixture(
442+
#"{"command":"tap","commandId":"tap-focus-confirmation","selectorKey":"id","selectorValue":"agent-device-hardware-keyboard-input"}"#
443+
)
444+
let tapResponse = try executeOnMainPrepared(command: tapCommand, activeApp: app)
445+
XCTAssertTrue(tapResponse.ok, String(describing: tapResponse.error))
446+
try XCTSkipIf(
447+
isKeyboardVisible(app: app),
448+
"software keyboard is up: this simulator cannot exercise the hidden-keyboard responder path"
449+
)
450+
451+
let deadline = Date().addingTimeInterval(TextEntryTiming.readinessTimeout)
452+
var confirmed = keyboardFocusConfirmed(app: app, element: textField)
453+
while !confirmed && Date() < deadline {
454+
sleepFor(TextEntryTiming.pollInterval)
455+
confirmed = keyboardFocusConfirmed(app: app, element: textField)
456+
}
457+
XCTAssertTrue(confirmed, "a tapped responder must confirm its own keyboard focus")
458+
XCTAssertFalse(
459+
keyboardFocusConfirmed(app: app, element: otherElement),
460+
"focus held by another element must read as a refusal, never as this element's focus"
461+
)
462+
}
463+
410464
func testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand() throws {
411465
app.launchArguments = [
412466
"--agent-device-text-entry-regression",
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
import XCTest
2+
3+
// How long the synthesized text-entry commit wait is willing to keep looking. Split out of
4+
// RunnerTests+SynthesizedTextEntry.swift so the policy is a pure, clock-injected value type the
5+
// macOS host lane can exercise without a simulator.
6+
extension RunnerTests {
7+
/// The commit wait's deadline.
8+
///
9+
/// A synthesized burst can be *throttled* rather than dropped: on a loaded simulator the
10+
/// characters keep landing, just slowly, and everything touching the input system slows with
11+
/// them (#1874). A flat wall-clock budget cannot tell that apart from a wedged pipeline, so it
12+
/// condemned both at the same instant — and the throttled case is a working command reported as
13+
/// `TEXT_INPUT_COMMIT_NOT_OBSERVED`, which is what turned an environment episode into a red
14+
/// lane on branches touching no iOS code.
15+
///
16+
/// So time is granted against *progress* — the observed value's expected-prefix growing, the
17+
/// same length-only evidence `logCommitCadence` already emits — with an absolute ceiling, so a
18+
/// pipeline that delivers one character per stall window cannot hold a command open forever.
19+
///
20+
/// A pipeline making no progress at all still expires at exactly the `stallBudget` the flat
21+
/// deadline used, so nothing that fails today starts passing merely by waiting longer: the wait
22+
/// extends only while characters are still arriving.
23+
///
24+
/// A reference type on purpose. Its two readers are separate escaping closures — the commit
25+
/// wait's `observe` records into it, its `isExpired` reads it — and the whole fix is the
26+
/// coupling between them. As a struct that coupling rests on Swift boxing one captured `var`,
27+
/// which a later refactor could quietly break back into the flat deadline with every test still
28+
/// green. Sharing one instance makes that unrepresentable instead of merely true today.
29+
final class SynthesizedCommitBudget {
30+
let startedAt: Date
31+
let stallBudget: TimeInterval
32+
let ceiling: TimeInterval
33+
private var bestPrefixLength: Int
34+
private var lastProgressAt: Date
35+
36+
init(startedAt: Date, stallBudget: TimeInterval, ceiling: TimeInterval) {
37+
self.startedAt = startedAt
38+
self.stallBudget = stallBudget
39+
self.ceiling = ceiling
40+
// Nothing has landed yet, and an observation of "still nothing" must not read as progress.
41+
self.bestPrefixLength = 0
42+
self.lastProgressAt = startedAt
43+
}
44+
45+
/// Records one observation's expected-prefix length. Only forward movement counts: a shorter
46+
/// read (the app clearing the field mid-flight, an unreadable poll reporting -1) is not
47+
/// evidence the burst is still landing, so it neither buys time nor takes any back.
48+
func record(expectedPrefixLength: Int, at now: Date) {
49+
guard expectedPrefixLength > bestPrefixLength else { return }
50+
bestPrefixLength = expectedPrefixLength
51+
lastProgressAt = now
52+
}
53+
54+
func isExpired(at now: Date) -> Bool {
55+
now.timeIntervalSince(startedAt) >= ceiling
56+
|| now.timeIntervalSince(lastProgressAt) >= stallBudget
57+
}
58+
}
59+
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedTextEntry.swift‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -360,23 +360,32 @@ extension RunnerTests {
360360
waitForNextObservation: () -> Void
361361
) {
362362
let placeholder = resolveTextEntryElement(app: app, target: target)?.placeholderValue
363-
let deadline = Date().addingTimeInterval(TextEntryTiming.synthesizedCommitTimeout)
364363
let waitStartedAt = Date()
364+
// Throttling and a wedge are the same to a flat deadline, so the expected-prefix walk decides
365+
// instead of the wall clock alone — the very distinction the cadence line below was added to
366+
// make visible. See `SynthesizedCommitBudget`.
367+
let budget = SynthesizedCommitBudget(
368+
startedAt: waitStartedAt,
369+
stallBudget: TextEntryTiming.synthesizedCommitStallTimeout,
370+
ceiling: TextEntryTiming.synthesizedCommitCeiling
371+
)
365372
return (
366373
placeholder: placeholder,
367-
isExpired: { Date() >= deadline },
374+
isExpired: { budget.isExpired(at: Date()) },
368375
observe: {
369376
let observedText = self.editableTextValue(
370377
for: self.resolveTextEntryElement(app: app, target: target),
371378
treatingPlaceholderAsEmpty: true
372379
)
380+
let expectedPrefixLen = observedText.map { Self.commonPrefixLength($0, expectedText) } ?? -1
381+
budget.record(expectedPrefixLength: expectedPrefixLen, at: Date())
373382
// Cadence evidence stays value-free: the polled value is user content typed through
374383
// `type`/`fill` and must never reach runner.log. Lengths and the expected-prefix walk
375384
// are enough to distinguish throttling (prefix grows slowly) from a wedge (it freezes).
376385
Self.logCommitCadence(
377386
elapsedMs: Int(waitStartedAt.timeIntervalSinceNow * -1000),
378387
observedLen: observedText?.count ?? -1,
379-
expectedPrefixLen: observedText.map { Self.commonPrefixLength($0, expectedText) } ?? -1
388+
expectedPrefixLen: expectedPrefixLen
380389
)
381390
return observedText
382391
},

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextEntry.swift‎

Lines changed: 57 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,14 @@ extension RunnerTests {
4545
static let pollInterval: TimeInterval = 0.02
4646
static let warmupValueTimeout: TimeInterval = 0.4
4747
static let verificationStabilityWindow: TimeInterval = 0.2
48-
static let synthesizedCommitTimeout: TimeInterval = 3.0
48+
/// How long the commit wait tolerates seeing NO further progress toward the expected value.
49+
/// Numerically the flat deadline this replaced, so a pipeline that delivers nothing is
50+
/// condemned at exactly the same instant it always was (see `SynthesizedCommitBudget`).
51+
static let synthesizedCommitStallTimeout: TimeInterval = 3.0
52+
/// The commit wait's absolute bound, however long characters keep arriving. Sits well inside
53+
/// the daemon's per-command budget (`RUNNER_COMMAND_TIMEOUT_MS`, 45s), which also has to cover
54+
/// focus, clear and verification around this wait.
55+
static let synthesizedCommitCeiling: TimeInterval = 10.0
4956
static let synthesizedCommitPollInterval: TimeInterval = 0.2
5057
}
5158

@@ -384,7 +391,7 @@ extension RunnerTests {
384391
var latest = resolveTextEntryElement(app: app, target: target)
385392
let keyboardVisibleAtEntry = isKeyboardVisible(app: app)
386393
let deadline = Date().addingTimeInterval(timeout)
387-
let hardwareKeyboardFallback = Date().addingTimeInterval(
394+
var hardwareKeyboardFallback = Date().addingTimeInterval(
388395
min(TextEntryTiming.hardwareKeyboardFallbackTimeout, timeout)
389396
)
390397
var sawSoftwareKeyboard = false
@@ -404,8 +411,19 @@ extension RunnerTests {
404411
return latest
405412
}
406413
sawSoftwareKeyboard = sawSoftwareKeyboard || keyboardElementExists(app: app)
407-
if !sawSoftwareKeyboard && Date() >= hardwareKeyboardFallback && latest != nil {
408-
return latest
414+
// A responder that takes no software keyboard (hardware keyboard connected, or a custom
415+
// `inputView`) would otherwise burn the whole readinessTimeout waiting for one that is never
416+
// coming. Leaving that window a bare wall-clock guess made readiness a function of ambient
417+
// simulator state (#1874): on a loaded host the keyboard is merely late, and returning here
418+
// handed the caller an element that had not taken focus yet. Ask the target itself instead,
419+
// and re-arm rather than re-asking every poll — the query is cheap, not free.
420+
if !sawSoftwareKeyboard, Date() >= hardwareKeyboardFallback, let candidate = latest {
421+
if keyboardFocusConfirmed(app: app, element: candidate) {
422+
return candidate
423+
}
424+
hardwareKeyboardFallback = Date().addingTimeInterval(
425+
TextEntryTiming.hardwareKeyboardFallbackTimeout
426+
)
409427
}
410428
sleepFor(TextEntryTiming.pollInterval)
411429
}
@@ -464,6 +482,41 @@ extension RunnerTests {
464482
return !wasVisibleAtEntry && isKeyboardVisible(app: app)
465483
}
466484

485+
/// Positive evidence that this element — the one readiness is about to hand its caller — holds
486+
/// keyboard focus. Readable even with no software keyboard on screen, which is what lets the
487+
/// hardware-keyboard fallback stop guessing from a wall clock.
488+
///
489+
/// This is the same app-wide predicate `focusedTextInput` refuses to trust on iOS, used the
490+
/// other way round. There, the query PICKS the target, so a stale or unrelated match becomes
491+
/// the field that gets typed into. Here the target is already chosen and the query only
492+
/// corroborates it: at most one element holds keyboard focus, so an answer that is not this
493+
/// element is a refusal, not a substitution. Every way of being wrong therefore ends as `false`
494+
/// and costs the remaining readiness timeout — exactly what the wait would spend with no
495+
/// fallback at all.
496+
func keyboardFocusConfirmed(app: XCUIApplication, element: XCUIElement) -> Bool {
497+
#if os(iOS)
498+
return safely("TEXT_ENTRY_FOCUS_CONFIRMED", false) {
499+
// An element that no longer resolves reads as identifier "" and frame `.zero`, which would
500+
// match any focused element that also reports an empty frame. Require a real frame first,
501+
// so a dead handle cannot corroborate anything.
502+
let frame = element.frame
503+
guard !frame.isEmpty else {
504+
return false
505+
}
506+
let focused = app
507+
.descendants(matching: .any)
508+
.matching(NSPredicate(format: "hasKeyboardFocus == 1"))
509+
.firstMatch
510+
guard focused.exists else {
511+
return false
512+
}
513+
return focused.identifier == element.identifier && focused.frame == frame
514+
}
515+
#else
516+
return false
517+
#endif
518+
}
519+
467520
private func keyboardElementExists(app: XCUIApplication) -> Bool {
468521
#if os(iOS)
469522
return safely("KEYBOARD_EXISTS", false) { app.keyboards.firstMatch.exists }
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
import XCTest
2+
3+
extension RunnerTests {
4+
#if AGENT_DEVICE_RUNNER_UNIT_TESTS
5+
private static func budget(
6+
startedAt: Date,
7+
stallBudget: TimeInterval = 3,
8+
ceiling: TimeInterval = 10
9+
) -> SynthesizedCommitBudget {
10+
SynthesizedCommitBudget(startedAt: startedAt, stallBudget: stallBudget, ceiling: ceiling)
11+
}
12+
13+
// The #1874 regression, stated as the budget sees it: a burst that keeps landing must keep its
14+
// wait alive past the old flat deadline. Before this policy the wait expired at 3s regardless,
15+
// so a throttled-but-working `type`/`fill` reported TEXT_INPUT_COMMIT_NOT_OBSERVED — the red
16+
// that kept appearing on branches touching no iOS code. Reverting `record` to a no-op turns
17+
// this red.
18+
func testProgressKeepsTheCommitWaitAliveBeyondTheFlatDeadline() {
19+
let start = Date(timeIntervalSinceReferenceDate: 0)
20+
let budget = Self.budget(startedAt: start)
21+
// One character per 2 seconds: never idle for 3s, so never expired on the stall rule.
22+
for (index, second) in [2.0, 4.0, 6.0].enumerated() {
23+
budget.record(expectedPrefixLength: index + 1, at: start.addingTimeInterval(second))
24+
XCTAssertFalse(
25+
budget.isExpired(at: start.addingTimeInterval(second)),
26+
"progress at \(second)s must not be expired"
27+
)
28+
}
29+
XCTAssertFalse(budget.isExpired(at: start.addingTimeInterval(8)))
30+
}
31+
32+
// The other half, and the reason the stall budget keeps the flat deadline's number: a pipeline
33+
// that delivers nothing is condemned at exactly the instant it always was. Nothing that fails
34+
// today starts passing by waiting longer.
35+
func testAWedgedPipelineStillExpiresAtTheStallBudget() {
36+
let start = Date(timeIntervalSinceReferenceDate: 0)
37+
let budget = Self.budget(startedAt: start)
38+
XCTAssertFalse(budget.isExpired(at: start.addingTimeInterval(2.99)))
39+
XCTAssertTrue(budget.isExpired(at: start.addingTimeInterval(3)))
40+
}
41+
42+
// Progress buys time, but not without bound: one character per stall window would otherwise
43+
// hold a command open until the daemon's own 45s timeout killed the runner's request.
44+
func testTheCeilingBoundsAnIndefinitelyThrottledPipeline() {
45+
let start = Date(timeIntervalSinceReferenceDate: 0)
46+
let budget = Self.budget(startedAt: start)
47+
for index in 1...5 {
48+
let now = start.addingTimeInterval(Double(index) * 2)
49+
budget.record(expectedPrefixLength: index, at: now)
50+
XCTAssertEqual(budget.isExpired(at: now), index == 5, "at \(index * 2)s")
51+
}
52+
}
53+
54+
// Only forward movement is evidence the burst is still landing. A field the app clears
55+
// mid-flight, or an unreadable poll (`observe` reports -1), must not reset the stall clock —
56+
// that would let a wedge disguised as churn hold the wait open to the ceiling every time.
57+
func testBackwardsAndUnreadableObservationsBuyNoTime() {
58+
let start = Date(timeIntervalSinceReferenceDate: 0)
59+
let budget = Self.budget(startedAt: start)
60+
budget.record(expectedPrefixLength: 5, at: start.addingTimeInterval(1))
61+
for (prefixLength, second) in [(0, 2.0), (-1, 3.0), (5, 3.5)] {
62+
budget.record(expectedPrefixLength: prefixLength, at: start.addingTimeInterval(second))
63+
}
64+
XCTAssertTrue(
65+
budget.isExpired(at: start.addingTimeInterval(4)),
66+
"the stall clock must still date from the 1s observation that actually advanced"
67+
)
68+
}
69+
#endif
70+
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextEntryPolicyTests.swift‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -416,11 +416,13 @@ extension RunnerTests {
416416
// poll and the value never becomes "abc" — under the replacement-mode outcome function that is
417417
// correctly a failure (see `testSynthesizedReplacementCommitCatchesDroppedMiddleCharacters` for
418418
// why it must NOT be waved through as success), so this call runs the real 3-second deadline
419-
// (`TextEntryTiming.synthesizedCommitTimeout`) before returning. That is deliberate here, not a
420-
// flake: this test only runs in the nightly XCUITest lane (see `runner-xctest-local-run-gotchas`
421-
// memory / ios.yml's `-only-testing:` allowlist), where a few extra seconds is a non-issue, and
422-
// the alternative — asserting `nil` on a wiring path that can never actually observe the
423-
// expected text — would silently reintroduce the exact bug this fix closes.
419+
// (`TextEntryTiming.synthesizedCommitStallTimeout`; a nil read never advances the expected
420+
// prefix, so `SynthesizedCommitBudget` grants it no extra time) before returning. That is
421+
// deliberate here, not a flake: this test only runs in the nightly XCUITest lane (see
422+
// `runner-xctest-local-run-gotchas` memory / ios.yml's `-only-testing:` allowlist), where a
423+
// few extra seconds is a non-issue, and the alternative — asserting `nil` on a wiring path
424+
// that can never actually observe the expected text — would silently reintroduce the exact
425+
// bug this fix closes.
424426
XCTAssertEqual(result.failure, .commitNotObserved)
425427
}
426428

0 commit comments

Comments
 (0)