Skip to content

Commit 7baa118

Browse files
committed
fix(ios-runner): charge the warmup split the type plan really makes
The review found `projectedSeconds` charging one synthesize call for an append-mode `type` while the route posts a warmup character and then the rest, so the estimate undercounted one call and the wait between them. The over-budget branch that asks is the one with no element, so its warmup wait is one poll: `waitForWarmupValue` has no expected value to wait for. That branch now charges 2 calls and the poll, which is one character less than a replacement of the same length. Also: the budget hint and the docs named one limit and said "the number" for a delayed retry, which reads as the larger one; both now name the limit for the delay in play and say the delay falls between characters. The pace's header comment claimed the acknowledge window is derived from it, which it is not. The synthesized-replacement helper builds its command with JSONSerialization, and the over-budget lane test polls the field under its own deadline instead of reading `value` once on a route that verifies nothing.
1 parent a58713b commit 7baa118

8 files changed

Lines changed: 92 additions & 24 deletions

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedTextEntry.h‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,10 @@ typedef NS_ENUM(NSInteger, RunnerSynthesizedTextEntryStatus) {
1717

1818
@interface RunnerSynthesizedTextEntry : NSObject
1919

20-
// Characters per second the synthesized text-input records are typed at. The delivery budget and
21-
// the app-acknowledge window a burst has to fit inside are both derived from it, so it is declared
22-
// once, here, where the typing happens.
20+
// Characters per second the synthesized text-input records are typed at. Declared here, where the
21+
// typing happens, so the delivery budget that bounds a burst is charged the same pace the app sees.
22+
// The edit-acknowledge window that pace is sized for is a separate assumption about the app
23+
// (TextEntryTiming.synthesizedAcknowledgeWindowSeconds), not a value derived from this one.
2324
+ (NSUInteger)typingSpeedCharactersPerSecond;
2425

2526
// Synthesizes keyboard input for the current first responder without resolving an

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedTextEntry.swift‎

Lines changed: 20 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -109,20 +109,31 @@ extension RunnerTests {
109109
1.0 / Double(RunnerSynthesizedTextEntry.typingSpeedCharactersPerSecond())
110110
}
111111

112-
/// Seconds the plan spends posting, charged per `synthesizedReplacementSteps` step: each
113-
/// synthesize call types its characters at the pace and pays its overhead, and a spaced plan
114-
/// sleeps `delaySeconds` between two calls.
115-
static func projectedSeconds(textLength: Int, delaySeconds: TimeInterval) -> TimeInterval {
116-
let calls = synthesizedReplacementIsSpaced(characterCount: textLength, delaySeconds: delaySeconds)
117-
? textLength
118-
: 1
112+
/// Seconds the plan spends posting: each synthesize call types its characters at the pace and
113+
/// pays its overhead, a spaced plan sleeps `delaySeconds` between two calls, and a plan that
114+
/// peels one character as a warmup (`typeWarmup`) pays one more call and the wait before the
115+
/// rest is posted. That wait is one poll here because the caller that asks has no element to
116+
/// read the warmup character back from, so `waitForWarmupValue` has no value to wait for.
117+
static func projectedSeconds(
118+
textLength: Int,
119+
delaySeconds: TimeInterval,
120+
typeWarmup: Bool = false
121+
) -> TimeInterval {
122+
let spaced = synthesizedReplacementIsSpaced(characterCount: textLength, delaySeconds: delaySeconds)
123+
let warmupSplit = typeWarmup && textLength > 1 && !spaced
124+
let calls = spaced ? textLength : (warmupSplit ? 2 : 1)
119125
return Double(textLength) * characterInterval
120126
+ Double(calls) * TextEntryTiming.synthesizeCallOverhead
121127
+ Double(calls - 1) * delaySeconds
128+
+ (warmupSplit ? TextEntryTiming.pollInterval : 0)
122129
}
123130

124-
static func exceeds(textLength: Int, delaySeconds: TimeInterval) -> Bool {
125-
projectedSeconds(textLength: textLength, delaySeconds: delaySeconds)
131+
static func exceeds(
132+
textLength: Int,
133+
delaySeconds: TimeInterval,
134+
typeWarmup: Bool = false
135+
) -> Bool {
136+
projectedSeconds(textLength: textLength, delaySeconds: delaySeconds, typeWarmup: typeWarmup)
126137
> TextEntryTiming.synthesizedDeliveryCeiling
127138
}
128139

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextEntry.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ extension RunnerTests {
3636
let recoveryBudget = SynthesizedDeliveryBudget.maxTextLength(
3737
delaySeconds: Double(recoveryDelay) / 1000
3838
)
39-
return "Fill at most \(SynthesizedDeliveryBudget.maxTextLength(delaySeconds: 0)) characters at a time without --delay-ms and append the rest with separate type commands, keeping each command inside that budget. --delay-ms lowers the budget, because every character then pays its own synthesize call and the delay: \(recoveryBudget) characters at --delay-ms \(recoveryDelay). A longer timeout does not help: this route is chosen when the accessibility channel is already degraded, and the pace is what makes the text long."
39+
return "Fill at most \(SynthesizedDeliveryBudget.maxTextLength(delaySeconds: 0)) characters at a time without --delay-ms and append the rest with separate type commands, keeping each command inside that budget. --delay-ms lowers the budget, because each character then gets its own synthesize call and each gap between characters pays the delay: \(recoveryBudget) characters at --delay-ms \(recoveryDelay). A longer timeout does not help: this route is chosen when the accessibility channel is already degraded, and the pace is what makes the text long."
4040
}
4141
}
4242
}

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+TextTyping.swift‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,11 @@ extension RunnerTests {
170170
// neither chunk would look long on its own. This branch's target has no element to type into,
171171
// so nothing can be read back afterwards: the value arrives unverified, as it does for this
172172
// route's older synthesizer-unavailable fallback.
173-
if SynthesizedDeliveryBudget.exceeds(textLength: text.count, delaySeconds: delaySeconds) {
173+
if SynthesizedDeliveryBudget.exceeds(
174+
textLength: text.count,
175+
delaySeconds: delaySeconds,
176+
typeWarmup: repairMode != .none
177+
) {
174178
textEntryRoute = "xctest-application-fallback"
175179
NSLog(
176180
"AGENT_DEVICE_RUNNER_TEXT_ENTRY_ROUTE route=xctest-application-fallback "

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedTextEntryTests.swift‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,18 @@ extension RunnerTests {
5858
commandId: String
5959
) throws -> Response {
6060
let frame = textField.frame
61-
let command = try runnerCommandFixture(
62-
#"{"command":"type","commandId":"\#(commandId)","text":"\#(text)","textEntryMode":"replace","x":\#(frame.midX),"y":\#(frame.midY)}"#
61+
// Assembled with JSONSerialization so a text carrying a quote or a backslash stays one command
62+
// rather than invalid JSON.
63+
let command = try JSONDecoder().decode(
64+
Command.self,
65+
from: JSONSerialization.data(withJSONObject: [
66+
"command": "type",
67+
"commandId": commandId,
68+
"text": text,
69+
"textEntryMode": "replace",
70+
"x": frame.midX,
71+
"y": frame.midY,
72+
])
6373
)
6474
let failuresBeforeType = currentXCTestFailureCount()
6575
let response = executeTypeCommand(activeApp: app, command: command)

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextEntryPolicyTests.swift‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,38 @@ extension RunnerTests {
283283
XCTAssertLessThan(SynthesizedDeliveryBudget.maxTextLength(delaySeconds: 0.2), fits)
284284
}
285285

286+
// A `type` plan peels one character as a warmup and posts the rest afterwards, so the same text
287+
// costs one synthesize call and one wait more than the single burst the replacement route posts.
288+
// Without this the estimate charged a burst, which is what made the over-budget branch of the
289+
// keyboard-visible route unreachable: 215 characters looked like 1 + 214, each inside the budget.
290+
func testTypeWarmupSplitCostsOneMoreCallThanASingleBurst() {
291+
let length = 20
292+
let withWarmup = SynthesizedDeliveryBudget.projectedSeconds(
293+
textLength: length,
294+
delaySeconds: 0,
295+
typeWarmup: true
296+
)
297+
XCTAssertGreaterThan(
298+
withWarmup,
299+
SynthesizedDeliveryBudget.projectedSeconds(textLength: length, delaySeconds: 0)
300+
)
301+
XCTAssertEqual(
302+
withWarmup - SynthesizedDeliveryBudget.projectedSeconds(textLength: length, delaySeconds: 0),
303+
TextEntryTiming.synthesizeCallOverhead + TextEntryTiming.pollInterval,
304+
accuracy: 1e-9
305+
)
306+
// The split mirrors the plan: a spaced `type` already posts per character, and a single
307+
// character has no rest to post.
308+
XCTAssertEqual(
309+
SynthesizedDeliveryBudget.projectedSeconds(textLength: length, delaySeconds: 0.2, typeWarmup: true),
310+
SynthesizedDeliveryBudget.projectedSeconds(textLength: length, delaySeconds: 0.2)
311+
)
312+
XCTAssertEqual(
313+
SynthesizedDeliveryBudget.projectedSeconds(textLength: 1, delaySeconds: 0, typeWarmup: true),
314+
SynthesizedDeliveryBudget.projectedSeconds(textLength: 1, delaySeconds: 0)
315+
)
316+
}
317+
286318
func testSynthesizedBudgetExceededCarriesItsOwnCodeAndRecovery() {
287319
XCTAssertEqual(
288320
TextEntryFailure.synthesisBudgetExceeded.rawValue,

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+TextTypingTests.swift‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -149,11 +149,11 @@ extension RunnerTests {
149149
}
150150

151151
// Text past the delivery budget cannot be paced into a field the runner cannot resolve, so it goes
152-
// through application-wide typing. The budget is charged the whole command, which is what the
153-
// length below pins: an append peels its first character for warmup, so a per-dispatch charge
154-
// would find both of its pieces inside the budget and pace all 215 characters. The target carries
155-
// no element by construction, so nothing on that route can read the value back: the command reports
156-
// it unverified and this test reads the field itself to show every character arrived.
152+
// through application-wide typing. The budget is charged the whole command, warmup split included:
153+
// an append peels its first character for warmup, so a per-dispatch charge would find both of its
154+
// pieces inside the budget and pace all these characters. The target carries no element by
155+
// construction, so nothing on that route can read the value back: the command reports it
156+
// unverified and this test reads the field itself to show every character arrived.
157157
func testOverBudgetTypeWithoutResolvableElementTypesApplicationWide() throws {
158158
app.launchArguments = [
159159
"--agent-device-text-entry-regression",
@@ -180,6 +180,8 @@ extension RunnerTests {
180180
count: SynthesizedDeliveryBudget.maxTextLength(delaySeconds: 0) + 1
181181
)
182182
let failureCountBefore = currentXCTestFailureCount()
183+
// The target the `type` command builds when it cannot resolve an input but the keyboard is up:
184+
// no element, no refresh point, focused-element preference.
183185
let result = typeTextReliably(
184186
app: app,
185187
target: TextEntryTarget(
@@ -197,8 +199,16 @@ extension RunnerTests {
197199
XCTAssertFalse(didRecordXCTestFailure(since: failureCountBefore))
198200
XCTAssertNil(result.failure)
199201
XCTAssertEqual(result.textEntryRoute, "xctest-application-fallback")
200-
XCTAssertNil(result.verified)
201-
XCTAssertEqual(textField.value as? String, text)
202+
// This branch has no element to read, so the value arrives unverified and the command waited for
203+
// nothing. The field is polled here, under its own deadline.
204+
let valueDeadline = Date().addingTimeInterval(appExistenceTimeout)
205+
var observed: String?
206+
while Date() < valueDeadline {
207+
observed = textField.value as? String
208+
if observed == text { break }
209+
Thread.sleep(forTimeInterval: 0.25)
210+
}
211+
XCTAssertEqual(observed, text)
202212
}
203213

204214
private struct UnavailableTextEntrySynthesizer: TextEntrySynthesizing {

‎website/docs/docs/commands.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -488,7 +488,7 @@ agent-device gesture transform 200 420 80 -40 2 35 700 # combined pan, zoom, and
488488
If `type` reports `TEXT_INPUT_NOT_FOCUSED`, focus a visible text input and retry; when accessibility does not expose the input, use a coordinate focus command before typing.
489489
On iOS, if `type "\n"` reports `TEXT_INPUT_SYNTHESIS_UNAVAILABLE` after tapping a field while the software keyboard is hidden, show the software keyboard, then retry. The runner reports this error instead of risking input through an unreliable text-entry path.
490490
On iOS, if `fill` reports `TEXT_INPUT_COMMIT_NOT_OBSERVED`, the runner could not confirm the typed text reached the field — either it did not land before the runner's deadline, or the expected final text is identical to the field's placeholder. In the latter case, accessibility cannot distinguish committed text from an empty field rendering that placeholder, even if the field held content before dispatch. The field may hold none, part, or all of the text: run `snapshot -i` and inspect it. If it already matches, continue; otherwise retry with the full text quoted and `fill --delay-ms 80`, which replaces the whole value. Do not use `type`, which appends to whatever committed. This covers the coordinate-driven `fill` route taken when the accessibility channel is under load, which observes the field after synthesizing; it is not a guarantee that every text-entry route verifies its result.
491-
On iOS, if `fill` reports `TEXT_INPUT_SYNTHESIS_BUDGET_EXCEEDED`, the text is longer than that coordinate-driven route can type inside one runner command at its bounded pace, and nothing was typed. Fill at most the number of characters the hint names and append the rest with separate `type` commands. `--delay-ms` lowers that budget, because every character then pays its own synthesize call and the delay; a longer timeout does not help.
491+
On iOS, if `fill` reports `TEXT_INPUT_SYNTHESIS_BUDGET_EXCEEDED`, the text is longer than that coordinate-driven route can type inside one runner command at its bounded pace, and nothing was typed. Fill at most the character limit the hint names for your `--delay-ms`, and append the rest with separate `type` commands: the hint gives one limit without `--delay-ms` and a lower one for the delay it recommends. `--delay-ms` lowers the budget because each character then gets its own synthesize call and each gap between characters pays that delay; a longer timeout does not help.
492492
Use plain `fill` or `type` first for ordinary login and form fields. Use `--delay-ms` on `type` or `fill` only when a debounced search field or search-as-you-type input actually misses characters, or when the app must receive incremental updates.
493493
Delayed typing intentionally prefers paced character entry over clipboard-style fallbacks so the target field receives each incremental update.
494494
On Android, `fill` also verifies text and treats IME-owned capture as a terminal failure instead of retrying against the wrong field.

0 commit comments

Comments
 (0)