You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 07ee0df
Browse filesBrowse the repository at this point in the historyBrowse files
fix: guide agents past iOS keyboard dismissal and get-text guesses
Tonight's benchmark leaderboard showed two recurring agent-UX misses:
keyboard dismiss failing after fill (5x, now the #1 failed command) and
get-text guessed as a command name (1x).
- iOS keyboardDismiss now returns a hint explaining the on-screen keyboard
does not block agent-device interactions, so agents should press the next
target directly instead of retrying dismiss, and use keyboard enter only
when submission is actually wanted.
- help manual-qa and help workflow recovery text no longer teach the false
"dismiss to unblock the target" pattern.
- The command-suggestion curated map now maps get-text/gettext/get_text to
the real get text command shape; the generic edit-distance fallback did
not produce any suggestion for these guesses.
Copy file name to clipboardExpand all lines: apple-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift
+3-1Lines changed: 3 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -1411,7 +1411,9 @@ extension RunnerTests {
1411
1411
ok:false,
1412
1412
error:ErrorPayload(
1413
1413
code:"UNSUPPORTED_OPERATION",
1414
-
message:"Unable to dismiss the iOS keyboard without a safe native dismiss control"
1414
+
message:"Unable to dismiss the iOS keyboard without a safe native dismiss control",
1415
+
hint:
1416
+
"The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need."
Copy file name to clipboardExpand all lines: src/cli/parser/cli-help.ts
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -168,7 +168,7 @@ Targets:
168
168
169
169
Recovery:
170
170
Network/typeahead result missing: wait text "Expected result" or wait <selector>.
171
-
Keyboard blocks target: keyboard dismiss when supported, then snapshot -i.
171
+
Keyboard visible over the next target: the on-screen keyboard does not block agent-device interactions. Do not dismiss the keyboard; press the next target directly.
172
172
Sparse or recovered accessibility snapshot: use screenshot as visual truth, leave the bad screen if needed, then retry snapshot -i.
173
173
Non-hittable success hint: verify with the settled diff or snapshot; retarget by a better ref/selector if the UI did not change.`,
174
174
},
@@ -250,8 +250,8 @@ Text entry:
250
250
agent-device press 'id="product-note"'
251
251
agent-device type "Handle with care" --delay-ms 80
252
252
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
253
-
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. Blocked control: try keyboard dismiss when supported.
254
-
To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback.
253
+
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard does not block agent-device interactions; do not dismiss it just to reach the next control, press that target directly instead.
254
+
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback, or press the next target directly instead of retrying dismiss.
255
255
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
256
256
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.
257
257
Android Gboard handwriting/stylus UI can capture text in an IME-owned input instead of the app field. If fill reports that input was captured by the keyboard/IME, use the diagnostic targetInput/actualInput details, inspect keyboard status/get if needed, and switch or disable handwriting outside the command plan before retrying. Do not keep retrying fill/type against the same field while the IME owns focus.
test('parseRunnerResponse surfaces the keyboard-dismiss hint to press the next target directly',async()=>{
648
+
consthint=
649
+
'The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need.';
650
+
constresponse=newResponse(
651
+
JSON.stringify({
652
+
ok: false,
653
+
error: {
654
+
code: 'UNSUPPORTED_OPERATION',
655
+
message: 'Unable to dismiss the iOS keyboard without a safe native dismiss control',
0 commit comments