Skip to content

Commit 142cb67

Browse files
thymikeethymikee
committed
fix(ios): report an unobserved text commit instead of a partial success
awaitSynthesizedFirstResponderCommit returned Void, so its three exits were indistinguishable to the caller: the expected text committed, the app transformed the input, or the 3s deadline expired with a strict prefix still outstanding. The caller returned dispatched-with-no-failure in all three, and `type` answered ok with textEntryRoute synthesized-first-responder over a field holding part of the requested text. The wait now returns a SynthesizedTextCommitOutcome and an expired deadline becomes TEXT_INPUT_COMMIT_NOT_OBSERVED, whose hint points at fill rather than a type retry — type appends, so retrying it concatenates onto whatever committed. The tail is still not re-synthesized: #1676 rejected that because a stalled prefix cannot be told apart from a suffix still queued, so repair double-posts. Reporting is what the runner does instead. typeIntoCurrentTarget loses its `dispatched` flag, which was exactly `failure == nil` and could not express the new state — characters posted, commit unconfirmed, command must refuse. Failure is now the single discriminator. The decision moves behind an injected clock/observer so the deadline branch runs in the macOS host lane on every PR instead of needing a simulator. Refs #1874, #1844
1 parent 46eff36 commit 142cb67

5 files changed

Lines changed: 129 additions & 33 deletions

File tree

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedTextEntry.swift‎

Lines changed: 53 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -184,36 +184,72 @@ extension RunnerTests {
184184
return expectedText.hasPrefix(observedText) ? .pending : .diverged
185185
}
186186

187+
/// How the commit wait ended. Distinct from `SynthesizedTextCommitProgress`, which classifies a
188+
/// single observation: this is the whole wait's verdict, and it exists so the deadline can be
189+
/// told apart from success. The wait used to return `Void`, which made an expired deadline
190+
/// indistinguishable from a committed one — `type` then reported ok with a partial value in the
191+
/// field (#1874, #1844).
192+
enum SynthesizedTextCommitOutcome: Equatable {
193+
/// The app answered: the expected text committed, or the app transformed the input and the
194+
/// runner must not second-guess it.
195+
case settled
196+
/// There was nothing to wait for — no readable baseline, or the text carries a submit key.
197+
case unobservable
198+
/// The deadline expired with a strict prefix of the expected text still outstanding.
199+
case notObserved
200+
}
201+
202+
/// The commit wait's decision, with observation, pacing and the clock injected so the deadline
203+
/// branch is exercisable without a simulator (the macOS host lane runs this; the member wrapper
204+
/// below binds the real XCUI reads).
205+
static func awaitSynthesizedCommitOutcome(
206+
expectedText: String,
207+
isExpired: () -> Bool,
208+
observe: () -> String?,
209+
waitForNextObservation: () -> Void
210+
) -> SynthesizedTextCommitOutcome {
211+
while !isExpired() {
212+
switch synthesizedTextCommitProgress(observedText: observe(), expectedText: expectedText) {
213+
case .committed, .diverged:
214+
return .settled
215+
case .pending:
216+
waitForNextObservation()
217+
}
218+
}
219+
return .notObserved
220+
}
221+
187222
/// Blocks until the synthesized bare-type text is observable in the target field, so `type`
188223
/// cannot report ok while trailing characters are still uncommitted on a slow simulator.
189224
///
190225
/// Observation only. A stalled prefix cannot be told apart from a suffix still queued in the
191226
/// event stream, so re-synthesizing the difference risks committing it twice after the command
192-
/// already reported success. Text carrying a submit key is skipped outright: the app may clear
193-
/// or rewrite the field on submit, so `textBefore + typedText` is not the value to wait for.
227+
/// already reported success (#1676 rejected exactly that repair). Reporting `.notObserved` is
228+
/// what the caller does instead: the partial value is the agent's to resolve, and a named
229+
/// failure beats a success that misdescribes the field. Text carrying a submit key is skipped
230+
/// outright: the app may clear or rewrite the field on submit, so `textBefore + typedText` is
231+
/// not the value to wait for.
194232
func awaitSynthesizedFirstResponderCommit(
195233
app: XCUIApplication,
196234
target: TextEntryTarget,
197235
textBefore: String?,
198236
typedText: String
199-
) {
237+
) -> SynthesizedTextCommitOutcome {
200238
guard let textBefore, !typedText.contains("\n"), !typedText.contains("\r") else {
201-
return
239+
return .unobservable
202240
}
203-
let expectedText = textBefore + typedText
204241
let deadline = Date().addingTimeInterval(TextEntryTiming.synthesizedCommitTimeout)
205-
while Date() < deadline {
206-
let observedText = editableTextValue(
207-
for: resolveTextEntryElement(app: app, target: target),
208-
treatingPlaceholderAsEmpty: true
209-
)
210-
switch Self.synthesizedTextCommitProgress(observedText: observedText, expectedText: expectedText) {
211-
case .committed, .diverged:
212-
return
213-
case .pending:
214-
sleepFor(TextEntryTiming.pollInterval)
215-
}
216-
}
242+
return Self.awaitSynthesizedCommitOutcome(
243+
expectedText: textBefore + typedText,
244+
isExpired: { Date() >= deadline },
245+
observe: {
246+
editableTextValue(
247+
for: resolveTextEntryElement(app: app, target: target),
248+
treatingPlaceholderAsEmpty: true
249+
)
250+
},
251+
waitForNextObservation: { sleepFor(TextEntryTiming.pollInterval) }
252+
)
217253
}
218254

219255
static func shouldUseResolvedCoordinateTextEntryRoute(

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextEntry.swift‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,16 @@ extension RunnerTests {
77
enum TextEntryFailure: String {
88
case notFocused = "TEXT_INPUT_NOT_FOCUSED"
99
case synthesisUnavailable = "TEXT_INPUT_SYNTHESIS_UNAVAILABLE"
10+
case commitNotObserved = "TEXT_INPUT_COMMIT_NOT_OBSERVED"
1011

1112
var message: String {
1213
switch self {
1314
case .notFocused:
1415
return "No focused text input was available for typing."
1516
case .synthesisUnavailable:
1617
return "Reliable text synthesis is unavailable while the software keyboard is hidden."
18+
case .commitNotObserved:
19+
return "The app committed only part of the typed text before the commit deadline."
1720
}
1821
}
1922

@@ -23,6 +26,8 @@ extension RunnerTests {
2326
return "Focus a visible text input, then retry type or fill. If the input is not exposed by accessibility, use a coordinate focus command before typing."
2427
case .synthesisUnavailable:
2528
return "Show the software keyboard, then retry type or fill."
29+
case .commitNotObserved:
30+
return "The field holds a partial value. Read it back before retrying, and prefer fill, which replaces the whole value, over type, which appends to whatever committed."
2631
}
2732
}
2833
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextEntryPolicyTests.swift‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,57 @@ extension RunnerTests {
9090
)
9191
}
9292

93+
// The regression behind #1874/#1844: the wait used to return Void, so an expired deadline was
94+
// indistinguishable from a commit and `type` reported ok over a partially committed field. The
95+
// CI signature was a field holding "h" out of "hardware-keyboard" with the command successful.
96+
func testSynthesizedCommitDeadlineIsNotReportedAsACommit() {
97+
var observations = 0
98+
let outcome = Self.awaitSynthesizedCommitOutcome(
99+
expectedText: "hardware-keyboard",
100+
isExpired: { observations >= 3 },
101+
observe: { "h" },
102+
waitForNextObservation: { observations += 1 }
103+
)
104+
XCTAssertEqual(outcome, .notObserved)
105+
XCTAssertEqual(observations, 3, "a pending prefix must keep polling until the deadline")
106+
}
107+
108+
func testSynthesizedCommitStopsAtTheFirstSettledObservation() {
109+
for observed in ["hardware-keyboard", "hardwarX", nil] {
110+
var polls = 0
111+
let outcome = Self.awaitSynthesizedCommitOutcome(
112+
expectedText: "hardware-keyboard",
113+
isExpired: { false },
114+
observe: { observed },
115+
waitForNextObservation: { polls += 1 }
116+
)
117+
// `.diverged` settles the wait too: the app transformed the input and the runner must not
118+
// second-guess it. Only an outstanding strict prefix keeps waiting.
119+
XCTAssertEqual(outcome, .settled, "observed: \(observed ?? "nil")")
120+
XCTAssertEqual(polls, 0, "observed: \(observed ?? "nil")")
121+
}
122+
}
123+
124+
func testSynthesizedCommitWalksAPrefixToCompletion() {
125+
let steps = ["", "hardware-", "hardware-keyboard"]
126+
var index = 0
127+
let outcome = Self.awaitSynthesizedCommitOutcome(
128+
expectedText: "hardware-keyboard",
129+
isExpired: { false },
130+
observe: { steps[min(index, steps.count - 1)] },
131+
waitForNextObservation: { index += 1 }
132+
)
133+
XCTAssertEqual(outcome, .settled)
134+
XCTAssertEqual(index, 2)
135+
}
136+
137+
func testCommitNotObservedCarriesItsOwnCodeAndRecovery() {
138+
XCTAssertEqual(TextEntryFailure.commitNotObserved.rawValue, "TEXT_INPUT_COMMIT_NOT_OBSERVED")
139+
// The recovery has to name fill: `type` appends, so retrying it over a partial value would
140+
// concatenate onto whatever committed rather than repair it.
141+
XCTAssertTrue(TextEntryFailure.commitNotObserved.hint.contains("fill"))
142+
}
143+
93144
#if os(iOS)
94145
func testSynthesizedTextEntryFallsBackOnlyWhenPrivateSynthesisIsUnavailable() {
95146
XCTAssertEqual(

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextTyping.swift‎

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -104,11 +104,11 @@ extension RunnerTests {
104104
}
105105
}
106106

107-
func typeIntoCurrentTarget(_ value: String) -> (element: XCUIElement?, dispatched: Bool, failure: TextEntryFailure?) {
107+
func typeIntoCurrentTarget(_ value: String) -> (element: XCUIElement?, failure: TextEntryFailure?) {
108108
#if os(iOS)
109109
if shouldUseSynthesizedFirstResponderType {
110110
guard let currentTarget = resolveTextEntryElement(app: app, target: activeTarget) else {
111-
return (nil, false, .notFocused)
111+
return (nil, .notFocused)
112112
}
113113
textEntryRoute = "synthesized-first-responder"
114114
NSLog("AGENT_DEVICE_RUNNER_TEXT_ENTRY_ROUTE route=synthesized-first-responder")
@@ -117,7 +117,7 @@ extension RunnerTests {
117117
case .continueTyping:
118118
// No refresh point: like the tap-witness target itself, the commit wait must observe
119119
// only the element the tap selected, never rediscover a different field.
120-
awaitSynthesizedFirstResponderCommit(
120+
let commit = awaitSynthesizedFirstResponderCommit(
121121
app: app,
122122
target: TextEntryTarget(
123123
element: currentTarget,
@@ -128,9 +128,12 @@ extension RunnerTests {
128128
textBefore: textBefore,
129129
typedText: value
130130
)
131-
return (currentTarget, true, nil)
131+
// The characters were posted, so the element stays on the tuple for the caller's
132+
// bookkeeping; the failure is what makes the command refuse. Reporting ok here is the
133+
// defect this route had — the deadline was indistinguishable from a commit.
134+
return (currentTarget, commit == .notObserved ? .commitNotObserved : nil)
132135
case .fallback:
133-
return (nil, false, .synthesisUnavailable)
136+
return (nil, .synthesisUnavailable)
134137
case .raise(let message):
135138
NSException(
136139
name: NSExceptionName.internalInconsistencyException,
@@ -142,7 +145,7 @@ extension RunnerTests {
142145
if let currentTarget = resolveTextEntryElement(app: app, target: activeTarget) {
143146
textEntryRoute = "xctest-element"
144147
currentTarget.typeText(value)
145-
return (currentTarget, true, nil)
148+
return (currentTarget, nil)
146149
} else if activeTarget.prefersFocusedElement && isKeyboardVisible(app: app) {
147150
#if os(iOS)
148151
textEntryRoute = "synthesized-first-responder"
@@ -167,9 +170,9 @@ extension RunnerTests {
167170
#else
168171
app.typeText(value)
169172
#endif
170-
return (resolveTextEntryElement(app: app, target: activeTarget), true, nil)
173+
return (resolveTextEntryElement(app: app, target: activeTarget), nil)
171174
}
172-
return (nil, false, .notFocused)
175+
return (nil, .notFocused)
173176
}
174177

175178
func dispatchFailureResult(_ failure: TextEntryFailure) -> TextEntryResult {
@@ -203,8 +206,8 @@ extension RunnerTests {
203206
let delayedTypeStartedAt = Date()
204207
for (index, character) in characters.enumerated() {
205208
let dispatch = typeIntoCurrentTarget(String(character))
206-
guard dispatch.dispatched else {
207-
return dispatchFailureResult(dispatch.failure ?? .notFocused)
209+
if let failure = dispatch.failure {
210+
return dispatchFailureResult(failure)
208211
}
209212
typedTarget = dispatch.element ?? typedTarget
210213
if index + 1 < characters.count {
@@ -252,8 +255,8 @@ extension RunnerTests {
252255
let firstCharacter = String(characters[0])
253256
let firstStartedAt = Date()
254257
let firstDispatch = typeIntoCurrentTarget(firstCharacter)
255-
guard firstDispatch.dispatched else {
256-
return dispatchFailureResult(firstDispatch.failure ?? .notFocused)
258+
if let failure = firstDispatch.failure {
259+
return dispatchFailureResult(failure)
257260
}
258261
var firstTypedTarget = firstDispatch.element
259262
logTextEntryPhase(
@@ -281,8 +284,8 @@ extension RunnerTests {
281284
let remainingText = String(characters.dropFirst())
282285
let remainingStartedAt = Date()
283286
let remainingDispatch = typeIntoCurrentTarget(remainingText)
284-
guard remainingDispatch.dispatched else {
285-
return dispatchFailureResult(remainingDispatch.failure ?? .notFocused)
287+
if let failure = remainingDispatch.failure {
288+
return dispatchFailureResult(failure)
286289
}
287290
firstTypedTarget = remainingDispatch.element ?? firstTypedTarget
288291
logTextEntryPhase(
@@ -296,8 +299,8 @@ extension RunnerTests {
296299
} else {
297300
let typeStartedAt = Date()
298301
let dispatch = typeIntoCurrentTarget(text)
299-
guard dispatch.dispatched else {
300-
return dispatchFailureResult(dispatch.failure ?? .notFocused)
302+
if let failure = dispatch.failure {
303+
return dispatchFailureResult(failure)
301304
}
302305
typedTarget = dispatch.element
303306
logTextEntryPhase(

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,6 +427,7 @@ agent-device gesture transform 200 420 80 -40 2 35 700 # combined pan, zoom, and
427427
`type` accepts text only. Do not pass `@ref` to `type`; use `fill @ref "text"` to target a field directly, or `press @ref` then `type "text"` to append in the focused field.
428428
If `type` reports `TEXT_INPUT_NOT_FOCUSED`, focus a visible text input and retry; when accessibility does not expose the input, use a coordinate focus command before typing.
429429
On iOS, if `type` reports `TEXT_INPUT_SYNTHESIS_UNAVAILABLE` while the software keyboard is hidden, show the software keyboard, then retry `type` or `fill`. The runner reports this error instead of risking partial input through an unreliable text-entry path.
430+
On iOS, if `type` reports `TEXT_INPUT_COMMIT_NOT_OBSERVED`, the app committed only part of the typed text before the runner's commit deadline. The field holds a partial value: read it back, then prefer `fill`, which replaces the whole value, over `type`, which appends to whatever committed. The runner reports this instead of succeeding over a field whose contents it could not confirm.
430431
Use plain `fill` or `type` first for ordinary login and form fields. Use `--delay-ms` on `type` or `fill` only when a debounced search field or search-as-you-type input actually misses characters, or when the app must receive incremental updates.
431432
Delayed typing intentionally prefers paced character entry over clipboard-style fallbacks so the target field receives each incremental update.
432433
On Android, `fill` also verifies text and treats IME-owned capture as a terminal failure instead of retrying against the wrong field.

0 commit comments

Comments
 (0)