Skip to content

Commit 4d5c384

Browse files
thymikeeclaude
andauthored
perf(ios): read each resolved text input through one snapshot (#3882)
* perf(ios): read each resolved text input through one snapshot On iOS every attribute read from a live XCUIElement re-runs its query, and each query snapshots the whole app (~90-110 ms on Bluesky's React Native Search screen). A fill read identity, frame, and value one attribute at a time and resolved the field again right before each post, so it made ~37 queries. - Resolution returns the snapshot it read; value, frame, identity, and the replacement baseline come from that snapshot. - The focus step hands its post-focus read to the entry, and a post reuses the last read when nothing was dispatched since it was taken. After a clear, the field resolves again under its bound identity. - The point lookup runs one query for all four text input types, reads each candidate through one snapshot, and still hands out per-type handles. Bluesky Search fill, runner type: 5.6-6.2 s -> 3.3-3.7 s. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): keep each fill read tied to the input it named Two reuses from the one-snapshot change could pair a read with the wrong input: - The point lookup counted per-type positions across candidates read from a shared query that each re-ran it. An input that left the app between two candidate reads shifted the later positions, so a handle could name another input than the snapshot it carried. Each handed-out handle now reads as the candidate's identity and frame; on a mismatch the lookup runs the per-type queries, whose handles read through their own snapshots. - The warmup kept its read across the pause it takes for an unreadable value or an expired deadline, so the next post skipped the identity check and could type into a successor. Only a read that no pause follows is reused now. Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(ios): fall back when any point-lookup position moved The point lookup validated a handle only for a candidate at the point, after the point filter. An input of another type that left the app after an earlier position's read moved a later input into a position already read, so the input at the point was never read: the next position read an input elsewhere, which the filter dropped, and the last position no longer existed and was skipped. The lookup returned no input without falling back, so a resolved-coordinate fill failed with NO_TEXT_INPUT_AT_POINT and a coordinate tap could lose its typing witness. Every position now must still exist and every handle must read as its position's input, at the point or not; otherwise the lookup runs the per-type queries. A position's existence is read only when its snapshot fails, so a lookup reads each candidate twice, as before for the input at the point. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(ios): look up text inputs through per-type queries only The shared elementType IN query saved three queries per point lookup but needed index rebinding, a validation snapshot per candidate, and a fallback for every shift race. Per-type handles never shift across types, so the lookup now reads each type's query and snapshots each candidate once, reading existence only when the snapshot fails. Bluesky and test-app fills measured no slower. An unbound target that resolves through its refresh point now keeps the snapshot the point lookup read instead of reading the input again. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(ios): let one reader own the reuse of a text input read Every resolver now returns ResolvedTextEntryInput, which owns the exists-first and snapshot-first reads. TextEntryReader hands a read out again until the next dispatch or pause, replacing the post-focus and undispatched reads tracked at each call site. The point lookup's snapshots answer identity, so its consumers stop probing candidates again. A fill reads the counted field 17 times instead of 19. Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(ios): verify and repair an append through the text entry reader Append verification, the repair decision, and the repair itself read through TextEntryReader, so a verification starts from the read its check took and the repair dispatches through the reader. The clear's caret tap takes its frame from the read it clears. A read that raises logs its exception, and every input comes from a named read. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 1778e24 commit 4d5c384

12 files changed

Lines changed: 489 additions & 236 deletions

‎apple/runner/AgentDeviceRunner/AgentDeviceRunner/AgentDeviceRunnerApp.m‎

Lines changed: 87 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ @interface AgentDeviceRunnerViewController : UIViewController
110110
@property(nonatomic, assign) NSTimeInterval textEntryBurstMinGap;
111111
@property(nonatomic, assign) NSTimeInterval textEntryAcknowledgeWindowSeconds;
112112
@property(nonatomic, strong) NSLayoutConstraint *textEntryFieldTop;
113+
@property(nonatomic, weak) UITextField *textEntryField;
113114
@property(nonatomic, assign) BOOL alertFixtureStarted;
114115
@property(nonatomic, strong) NSTimer *alertActivationBusyBackstop;
115116
@property(nonatomic, strong) NSTimer *alertBannerRepost;
@@ -130,6 +131,39 @@ - (NSString *)accessibilityValue {
130131
}
131132
@end
132133

134+
// XCTest reads every element's accessibility value each time it snapshots the app, which it does
135+
// for every element query and every attribute read from a live element. Counting this field's
136+
// value reads therefore counts the accessibility queries a command makes. The value is masked, so a
137+
// fill's read-back ends at its first read instead of polling through a stability window whose
138+
// poll count depends on how fast the host answers.
139+
static NSUInteger AgentDeviceTextEntryValueReads = 0;
140+
141+
@interface AgentDeviceValueReadCountingTextField : AgentDeviceMaskedTextField
142+
@end
143+
144+
@implementation AgentDeviceValueReadCountingTextField
145+
- (NSString *)accessibilityValue {
146+
AgentDeviceTextEntryValueReads += 1;
147+
return [super accessibilityValue];
148+
}
149+
@end
150+
151+
// Reports the read count as its accessibility label and the counted field's text as its value, so
152+
// reading either changes no view state and the masked text stays checkable.
153+
@interface AgentDeviceValueReadCountLabel : UILabel
154+
@property(nonatomic, weak) UITextField *countedField;
155+
@end
156+
157+
@implementation AgentDeviceValueReadCountLabel
158+
- (NSString *)accessibilityLabel {
159+
return [NSString stringWithFormat:@"%lu", (unsigned long)AgentDeviceTextEntryValueReads];
160+
}
161+
162+
- (NSString *)accessibilityValue {
163+
return self.countedField.text;
164+
}
165+
@end
166+
133167
// A field that reformats its own text on every edit into `## ### ## ##` digit groups, the way a
134168
// digit-grouping formatter in a Flutter or React Native input does. The accessibility value is
135169
// what the user sees — the formatted string — so it never equals the digits a `fill` sent (#2634).
@@ -513,20 +547,33 @@ - (void)agentDeviceTextEntryDidChange:(UITextField *)textField {
513547
}
514548
if ((codeComplete && [arguments containsObject:@"--agent-device-text-entry-auto-submit"]) ||
515549
(textField.text.length > 0 && [arguments containsObject:@"--agent-device-text-entry-replace-after-input"])) {
516-
UITextField *nextScreenField = [[UITextField alloc] initWithFrame:textField.frame];
517-
if (![arguments containsObject:@"--agent-device-text-entry-unnamed-input"]) {
518-
nextScreenField.accessibilityIdentifier = @"agent-device-auto-submit-next-screen-input";
519-
}
520-
nextScreenField.borderStyle = UITextBorderStyleRoundedRect;
521-
[textField.superview addSubview:nextScreenField];
522-
[textField removeFromSuperview];
523-
[nextScreenField becomeFirstResponder];
550+
[self replaceTextEntryFieldWithNextScreenInput:textField];
524551
} else if (codeComplete &&
525552
[arguments containsObject:@"--agent-device-text-entry-auto-submit-without-successor"]) {
526553
[textField removeFromSuperview];
527554
}
528555
}
529556

557+
- (void)replaceTextEntryFieldWithNextScreenInput:(UITextField *)textField {
558+
UITextField *nextScreenField = [[UITextField alloc] initWithFrame:textField.frame];
559+
if (![NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-unnamed-input"]) {
560+
nextScreenField.accessibilityIdentifier = @"agent-device-auto-submit-next-screen-input";
561+
}
562+
nextScreenField.borderStyle = UITextBorderStyleRoundedRect;
563+
[textField.superview addSubview:nextScreenField];
564+
[textField removeFromSuperview];
565+
[nextScreenField becomeFirstResponder];
566+
}
567+
568+
// The same navigation as `replace-after-input`, landing when a test presses the button instead of on
569+
// an edit, so the test decides which runner step it lands between.
570+
- (void)agentDeviceTextEntryReplaceDidPress:(UIButton *)sender {
571+
(void)sender;
572+
if (self.textEntryField.superview != nil) {
573+
[self replaceTextEntryFieldWithNextScreenInput:self.textEntryField];
574+
}
575+
}
576+
530577
// Stops answering the element-type query that found it, the moment it takes focus, under
531578
// `--agent-device-text-entry-unqueryable-on-focus`. #3060's Flutter password field reaches this shape
532579
// because its two accessibility channels disagree once focused — `TextField` through the legacy
@@ -690,6 +737,8 @@ - (void)viewDidLoad {
690737
[NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-submit-clears"];
691738
BOOL maskedValue =
692739
[NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-masked-value"];
740+
BOOL countsValueReads =
741+
[NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-counts-value-reads"];
693742
UITextField *textField;
694743
if (digitCountValue) {
695744
textField = [[AgentDeviceDigitCountTextField alloc] init];
@@ -702,6 +751,8 @@ - (void)viewDidLoad {
702751
textField = [[AgentDeviceSubmitClearingTextField alloc] init];
703752
} else if (maskedValue) {
704753
textField = [[AgentDeviceMaskedTextField alloc] init];
754+
} else if (countsValueReads) {
755+
textField = [[AgentDeviceValueReadCountingTextField alloc] init];
705756
} else {
706757
textField = [[UITextField alloc] init];
707758
}
@@ -724,6 +775,7 @@ - (void)viewDidLoad {
724775
forControlEvents:UIControlEventEditingChanged];
725776
textField.translatesAutoresizingMaskIntoConstraints = NO;
726777
[self.view addSubview:textField];
778+
self.textEntryField = textField;
727779
self.textEntryFieldTop = [textField.topAnchor constraintEqualToAnchor:label.bottomAnchor
728780
constant:AgentDeviceTextEntryFieldTopInset];
729781
[NSLayoutConstraint activateConstraints:@[
@@ -732,6 +784,18 @@ - (void)viewDidLoad {
732784
[textField.widthAnchor constraintEqualToConstant:240],
733785
[textField.heightAnchor constraintEqualToConstant:AgentDeviceTextEntryFieldHeight],
734786
]];
787+
if (countsValueReads) {
788+
AgentDeviceValueReadCountLabel *valueReads = [[AgentDeviceValueReadCountLabel alloc] init];
789+
valueReads.countedField = textField;
790+
valueReads.accessibilityIdentifier = @"agent-device-text-entry-value-reads";
791+
valueReads.text = @"value reads";
792+
valueReads.translatesAutoresizingMaskIntoConstraints = NO;
793+
[self.view addSubview:valueReads];
794+
[NSLayoutConstraint activateConstraints:@[
795+
[valueReads.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
796+
[valueReads.topAnchor constraintEqualToAnchor:textField.bottomAnchor constant:12],
797+
]];
798+
}
735799
if (currencyValue || submitClears) {
736800
UILabel *entryStatus = [[UILabel alloc] init];
737801
entryStatus.accessibilityIdentifier = @"agent-device-text-entry-events";
@@ -774,6 +838,21 @@ - (void)viewDidLoad {
774838
]];
775839
[self updateTapWitnessStatus];
776840
}
841+
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-replace-on-request"]) {
842+
// Above the title, so the software keyboard never covers it.
843+
UIButton *replace = [UIButton buttonWithType:UIButtonTypeSystem];
844+
replace.accessibilityIdentifier = @"agent-device-text-entry-replace";
845+
[replace setTitle:@"Replace input" forState:UIControlStateNormal];
846+
replace.translatesAutoresizingMaskIntoConstraints = NO;
847+
[replace addTarget:self
848+
action:@selector(agentDeviceTextEntryReplaceDidPress:)
849+
forControlEvents:UIControlEventTouchUpInside];
850+
[self.view addSubview:replace];
851+
[NSLayoutConstraint activateConstraints:@[
852+
[replace.centerXAnchor constraintEqualToAnchor:self.view.centerXAnchor],
853+
[replace.bottomAnchor constraintEqualToAnchor:label.topAnchor constant:-24],
854+
]];
855+
}
777856
if ([NSProcessInfo.processInfo.arguments containsObject:@"--agent-device-text-entry-unqueryable-on-focus"]) {
778857
self.textEntryFocusWitness = [[UILabel alloc] init];
779858
self.textEntryFocusWitness.accessibilityIdentifier = @"agent-device-text-entry-focus";

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -335,7 +335,7 @@ extension RunnerTests {
335335

336336
/// Shared ordering for point-hit candidates: smallest area wins, then top-to-bottom,
337337
/// left-to-right, then stable element-type order for ties.
338-
func smallestElementFirst(_ left: XCUIElement, _ right: XCUIElement) -> Bool {
338+
func smallestElementFirst(_ left: any XCUIElementAttributes, _ right: any XCUIElementAttributes) -> Bool {
339339
let leftArea = max(1, left.frame.width * left.frame.height)
340340
let rightArea = max(1, right.frame.width * right.frame.height)
341341
if leftArea != rightArea {
@@ -352,7 +352,7 @@ extension RunnerTests {
352352

353353
func readTextAt(app: XCUIApplication, x: Double, y: Double) -> String? {
354354
let point = CGPoint(x: x, y: y)
355-
let textInputCandidates = textInputCandidatesAt(app: app, point: point)
355+
let textInputCandidates = textInputCandidatesAt(app: app, point: point).map(\.element)
356356
for element in textInputCandidates where prefersExpandedTextRead(element) {
357357
if let text = readableText(for: element) {
358358
return text
@@ -363,7 +363,7 @@ extension RunnerTests {
363363
.filter { element in
364364
element.exists && !element.frame.isEmpty && element.frame.contains(point)
365365
}
366-
.sorted(by: smallestElementFirst)
366+
.sorted { smallestElementFirst($0, $1) }
367367

368368
for element in candidates where prefersExpandedTextRead(element) {
369369
if let text = readableText(for: element) {

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedCommitDeadline.swift‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -138,15 +138,15 @@ extension RunnerTests {
138138
observe: () -> TextValueRead,
139139
waitForNextObservation: () -> Void
140140
) {
141-
let field = resolveTextEntryElement(app: app, target: target)
141+
let field = resolveTextEntryInput(app: app, target: target)?.attributes
142142
let placeholder = field?.placeholderValue
143143
let waitStartedAt = Date()
144144
return (
145145
placeholder: placeholder,
146146
fieldIsUnreadable: textValueRead(for: field) == .unreadable,
147147
observe: {
148148
let observed = self.textValueRead(
149-
for: self.resolveTextEntryElement(app: app, target: target),
149+
for: self.resolveTextEntryInput(app: app, target: target)?.attributes,
150150
treatingPlaceholderAsEmpty: true
151151
)
152152
var observedText: String?

0 commit comments

Comments
 (0)