You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 543e9f8
Browse filesBrowse the repository at this point in the historyBrowse files
fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598) (#1606)
* fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598)
The runner already tapped a keyboard's own Hide/Dismiss/Done key when the
AX tree exposed one, but iPhone's default software keyboard has no such
key, so `keyboard dismiss` returned UNSUPPORTED_OPERATION on the common
case and agents proceeded with the keyboard (and any live QuickType
predictive-text bar) still up.
Live-validated on throwaway simulators before choosing a design: hardware
escape key (no effect without a connected hardware keyboard), swipe-down
starting on the keyboard (does not trigger UIKit's interactive dismissal
on Settings/Safari/Contacts), and a private
`performAction:onElement:value:error:` AX call (hung the runner for 90s on
a guessed action name, force-killed by the daemon timeout) were all ruled
out. The dismiss-key tap remains the primary mechanism (iPad, or any app
with an inputAccessoryView Done/Cancel button); a new snapshot-derived
safe-area tap is added as the disclosed last resort, computed to land
outside both the keyboard and every currently-hittable element so it is a
safe no-op even when it fails to dismiss.
The response now discloses which mechanism actually fired
(`mechanism: 'dismissKey' | 'safeAreaTap'`) across the CLI/daemon dispatch
path, the SDK runtime.backend surface, and session-event summaries, so
callers can tell a real dismiss-key press apart from a best-effort tap.
UNSUPPORTED_OPERATION now says both mechanisms were tried.
* fix: satisfy CI formatting and complexity gates
oxfmt on three touched files; buildKeyboardActionSummary split so the
dismiss wording (incl. the safeAreaTap mechanism disclosure) lives in its
own helper below the complexity threshold.
* fix: any-element obstacle rule for the safe-area dismiss tap (#1606 review P1)
A role allowlist cannot prove a point is AX-empty: an unlabeled RN
Pressable surfaces as a hittable Other, and a tappable parent can cover a
point its static-text child does not. Every known element frame now counts
as an obstacle regardless of role or hittability, with only ~window-sized
structural frames exempt (isStructuralRootFrame, 95% coverage) — exempting
those is what keeps the rule satisfiable, and a genuinely tappable
full-screen backdrop staying exempt is the disclosed, accepted behavior of
this fallback. One .any resolution replaces ten typed queries (single tree
snapshot, no per-element isHittable round trips), so the stricter rule is
also cheaper.
* fix: drop the safe-area tap — background-tap dismissal is unsupported (#1606 review P1, round 2)
No geometry or role query can prove a coordinate is side-effect-free: after
the any-element rule, the structural-root exemption still deliberately
removed full-screen actionable elements (RN Pressable backdrops) from the
obstacle set, so the tap could navigate or submit — and report success
because the mutation hid the keyboard. Per review, generic background-tap
dismissal is now explicitly unsupported: the dismiss key is the only
mechanism the runner vouches for, UNSUPPORTED_OPERATION says so and steers
callers to press-the-next-target / keyboard enter, and the mechanism field
narrows to 'dismissKey'. Unrecognized wire mechanisms degrade to the bare
message and are dropped from event details.
Copy file name to clipboardExpand all lines: apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift
+3-2Lines changed: 3 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -2011,7 +2011,7 @@ extension RunnerTests {
2011
2011
ok:false,
2012
2012
error:ErrorPayload(
2013
2013
code:"UNSUPPORTED_OPERATION",
2014
-
message:"Unable to dismiss the iOS keyboard without a safe native dismiss control",
2014
+
message:"Unable to dismiss the iOS keyboard: the keyboard exposes no dismiss key, and background taps are never attempted (no tap outside the keyboard can be proven side-effect-free)",
2015
2015
hint:
2016
2016
"The on-screen keyboard usually does not block agent-device interactions: press the next target directly instead of retrying dismiss. If that press fails or reports no visible effect, scroll the target into view, or use keyboard enter to press the return key when submission is wanted."
Copy file name to clipboardExpand all lines: src/cli/parser/cli-help.ts
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -296,8 +296,8 @@ Text entry:
296
296
agent-device type "Handle with care" --delay-ms 80
297
297
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
298
298
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard usually does not block agent-device interactions; press the next target directly instead of dismissing. If that press fails or reports no visible effect, scroll the target into view or use keyboard enter when submission is wanted.
299
-
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed.
300
-
On iOS, if it returns UNSUPPORTED_OPERATION, there is no generic app-agnostic blur fallback: do not assume a static text or heading is safe to press, because it can belong to a tappable parent. Use an app-provided dismiss control only when its action is explicitly intended; otherwise report that keyboard dismissal is unavailable.
299
+
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps the keyboard's own dismiss/hide key when one is exposed (common on iPad, rare on iPhone) and verifies the keyboard closed. When no dismiss key exists it reports UNSUPPORTED_OPERATION rather than tapping elsewhere — no tap outside the keyboard can be proven side-effect-free. Then prefer submitting (type "\n" on single-line fields) or pressing a known on-screen control that does not mutate state.
300
+
On iOS, if it still returns UNSUPPORTED_OPERATION, both mechanisms were exhausted: do not assume a static text or heading is safe to press, because it can belong to a tappable parent. Use an app-provided dismiss control only when its action is explicitly intended; otherwise report that keyboard dismissal is unavailable.
301
301
On Android, keyboard dismiss first avoids navigation. If it returns UNSUPPORTED_OPERATION because the current IME needs back navigation, use back only when normal back behavior is acceptable; otherwise report that keyboard dismissal is unavailable.
302
302
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
303
303
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.
'Inspect Android keyboard visibility/type or press/dismiss the device keyboard. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available, verifies the keyboard closed, and reports UNSUPPORTED_OPERATION when no safe control is available.',
184
+
'Inspect Android keyboard visibility/type or press/dismiss the device keyboard. To hide the keyboard, use keyboard dismiss. It taps the keyboard dismiss/hide key when one is exposed, verifies the keyboard closed, and reports UNSUPPORTED_OPERATION when no dismiss key exists — background taps are never attempted.',
185
185
summary: 'Inspect, press, or dismiss the device keyboard',
0 commit comments