Skip to content

Commit 543e9f8

Browse files
authored
fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598) (#1606)
* fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598) The runner already tapped a keyboard's own Hide/Dismiss/Done key when the AX tree exposed one, but iPhone's default software keyboard has no such key, so `keyboard dismiss` returned UNSUPPORTED_OPERATION on the common case and agents proceeded with the keyboard (and any live QuickType predictive-text bar) still up. Live-validated on throwaway simulators before choosing a design: hardware escape key (no effect without a connected hardware keyboard), swipe-down starting on the keyboard (does not trigger UIKit's interactive dismissal on Settings/Safari/Contacts), and a private `performAction:onElement:value:error:` AX call (hung the runner for 90s on a guessed action name, force-killed by the daemon timeout) were all ruled out. The dismiss-key tap remains the primary mechanism (iPad, or any app with an inputAccessoryView Done/Cancel button); a new snapshot-derived safe-area tap is added as the disclosed last resort, computed to land outside both the keyboard and every currently-hittable element so it is a safe no-op even when it fails to dismiss. The response now discloses which mechanism actually fired (`mechanism: 'dismissKey' | 'safeAreaTap'`) across the CLI/daemon dispatch path, the SDK runtime.backend surface, and session-event summaries, so callers can tell a real dismiss-key press apart from a best-effort tap. UNSUPPORTED_OPERATION now says both mechanisms were tried. * fix: satisfy CI formatting and complexity gates oxfmt on three touched files; buildKeyboardActionSummary split so the dismiss wording (incl. the safeAreaTap mechanism disclosure) lives in its own helper below the complexity threshold. * fix: any-element obstacle rule for the safe-area dismiss tap (#1606 review P1) A role allowlist cannot prove a point is AX-empty: an unlabeled RN Pressable surfaces as a hittable Other, and a tappable parent can cover a point its static-text child does not. Every known element frame now counts as an obstacle regardless of role or hittability, with only ~window-sized structural frames exempt (isStructuralRootFrame, 95% coverage) — exempting those is what keeps the rule satisfiable, and a genuinely tappable full-screen backdrop staying exempt is the disclosed, accepted behavior of this fallback. One .any resolution replaces ten typed queries (single tree snapshot, no per-element isHittable round trips), so the stricter rule is also cheaper. * fix: drop the safe-area tap — background-tap dismissal is unsupported (#1606 review P1, round 2) No geometry or role query can prove a coordinate is side-effect-free: after the any-element rule, the structural-root exemption still deliberately removed full-screen actionable elements (RN Pressable backdrops) from the obstacle set, so the tap could navigate or submit — and report success because the mutation hid the keyboard. Per review, generic background-tap dismissal is now explicitly unsupported: the dismiss key is the only mechanism the runner vouches for, UNSUPPORTED_OPERATION says so and steers callers to press-the-next-target / keyboard enter, and the mechanism field narrows to 'dismissKey'. Unrecognized wire mechanisms degrade to the bare message and are dropped from event details.
1 parent 65450dd commit 543e9f8

16 files changed

Lines changed: 229 additions & 20 deletions

File tree

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2011,7 +2011,7 @@ extension RunnerTests {
20112011
ok: false,
20122012
error: ErrorPayload(
20132013
code: "UNSUPPORTED_OPERATION",
2014-
message: "Unable to dismiss the iOS keyboard without a safe native dismiss control",
2014+
message: "Unable to dismiss the iOS keyboard: the keyboard exposes no dismiss key, and background taps are never attempted (no tap outside the keyboard can be proven side-effect-free)",
20152015
hint:
20162016
"The on-screen keyboard usually does not block agent-device interactions: press the next target directly instead of retrying dismiss. If that press fails or reports no visible effect, scroll the target into view, or use keyboard enter to press the return key when submission is wanted."
20172017
)
@@ -2023,7 +2023,8 @@ extension RunnerTests {
20232023
message: "keyboardDismiss",
20242024
visible: result.visible,
20252025
wasVisible: result.wasVisible,
2026-
dismissed: result.dismissed
2026+
dismissed: result.dismissed,
2027+
keyboardDismissMechanism: result.mechanism?.rawValue
20272028
)
20282029
)
20292030
case .keyboardReturn:

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift‎

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,23 +21,34 @@ private enum KeyboardDismissObservationTiming {
2121
static let settleRequiredConsecutiveMatches: Int = 3
2222
}
2323

24+
// The mechanism that actually resigned the keyboard, disclosed to the caller
25+
// (#1598) so a response never claims "dismissed" without saying how — a
26+
// safe-area tap has a very different reliability/side-effect profile than
27+
// tapping the keyboard's own Done key, and callers need to know which one
28+
// fired.
29+
enum RunnerKeyboardDismissMechanism: String {
30+
case dismissKey
31+
}
32+
2433
extension RunnerTests {
2534
func isKeyboardVisible(app: XCUIApplication) -> Bool {
2635
return visibleKeyboardFrame(app: app) != nil
2736
}
2837

29-
func dismissKeyboard(app: XCUIApplication) -> (wasVisible: Bool, dismissed: Bool, visible: Bool) {
38+
func dismissKeyboard(
39+
app: XCUIApplication
40+
) -> (wasVisible: Bool, dismissed: Bool, visible: Bool, mechanism: RunnerKeyboardDismissMechanism?) {
3041
let keyboard = app.keyboards.firstMatch
3142
let wasVisible = isKeyboardVisible(app: app)
3243
guard wasVisible else {
33-
return (wasVisible: false, dismissed: false, visible: false)
44+
return (wasVisible: false, dismissed: false, visible: false, mechanism: nil)
3445
}
3546

3647
#if os(tvOS)
3748
_ = pressTvRemote(.menu)
3849
sleepFor(0.2)
3950
let visible = isKeyboardVisible(app: app)
40-
return (wasVisible: true, dismissed: !visible, visible: visible)
51+
return (wasVisible: true, dismissed: !visible, visible: visible, mechanism: visible ? nil : .dismissKey)
4152
#else
4253
if tapKeyboardDismissControl(app: app) {
4354
_ = keyboard.waitForNonExistence(timeout: KeyboardDismissObservationTiming.timeout)
@@ -47,13 +58,20 @@ extension RunnerTests {
4758
requiredConsecutiveMatches: KeyboardDismissObservationTiming.settleRequiredConsecutiveMatches
4859
)
4960
let visible = isKeyboardVisible(app: app)
50-
return (wasVisible: true, dismissed: !visible, visible: visible)
61+
return (wasVisible: true, dismissed: !visible, visible: visible, mechanism: visible ? nil : .dismissKey)
5162
}
5263

53-
return (wasVisible: true, dismissed: false, visible: isKeyboardVisible(app: app))
64+
// #1606 review P1 (twice): generic background-tap dismissal is
65+
// deliberately UNSUPPORTED. No geometry or role query can prove a
66+
// coordinate is side-effect-free — a full-screen unlabeled Pressable is
67+
// indistinguishable from an inert backdrop, so a "safe-area" tap can
68+
// navigate or submit while reporting a successful dismiss. The dismiss
69+
// key is the only mechanism the runner can vouch for.
70+
return (wasVisible: true, dismissed: false, visible: isKeyboardVisible(app: app), mechanism: nil)
5471
#endif
5572
}
5673

74+
5775
// AX-free on purpose (screenshot bytes, not the accessibility tree) so it holds
5876
// under the same AX degradation the synthesized gesture lane is built to survive.
5977
// Bounded and self-terminating: returns as soon as `requiredConsecutiveMatches`

‎apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Models.swift‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,7 @@ struct DataPayload: Codable {
257257
let visible: Bool?
258258
let wasVisible: Bool?
259259
let dismissed: Bool?
260+
let keyboardDismissMechanism: String?
260261
let orientation: String?
261262
let gestureFallback: String?
262263
let gestureFallbackMessage: String?
@@ -297,6 +298,7 @@ struct DataPayload: Codable {
297298
visible: Bool? = nil,
298299
wasVisible: Bool? = nil,
299300
dismissed: Bool? = nil,
301+
keyboardDismissMechanism: String? = nil,
300302
orientation: String? = nil,
301303
gestureFallback: String? = nil,
302304
gestureFallbackMessage: String? = nil,
@@ -336,6 +338,7 @@ struct DataPayload: Codable {
336338
self.visible = visible
337339
self.wasVisible = wasVisible
338340
self.dismissed = dismissed
341+
self.keyboardDismissMechanism = keyboardDismissMechanism
339342
self.orientation = orientation
340343
self.gestureFallback = gestureFallback
341344
self.gestureFallbackMessage = gestureFallbackMessage

‎apple/runner/RUNNER_PROTOCOL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Successful and failed responses use the same top-level envelope:
6262
"ok": false,
6363
"error": {
6464
"code": "UNSUPPORTED_OPERATION",
65-
"message": "Unable to dismiss the iOS keyboard without a safe native dismiss control"
65+
"message": "Unable to dismiss the iOS keyboard: the keyboard exposes no dismiss key (background taps are never attempted)"
6666
}
6767
}
6868
```

‎packages/contracts/src/keyboard.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,4 +26,11 @@ export type KeyboardCommandResult = {
2626
focusedResourceId?: string;
2727
inputOwner?: 'app' | 'ime' | 'unknown';
2828
message?: string;
29+
/** iOS `dismiss` only (#1598): which mechanism actually resigned the
30+
* keyboard — 'dismissKey' (tapped the keyboard's own Hide/Dismiss/Done
31+
* key). Generic background-tap dismissal is deliberately unsupported
32+
* (#1606 review): no query can prove a coordinate is side-effect-free,
33+
* so the runner only ever taps the keyboard's own control. Absent when
34+
* the keyboard was not dismissed. */
35+
mechanism?: 'dismissKey';
2936
};

‎src/backend.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,9 @@ export type BackendKeyboardResult = {
108108
wasVisible?: boolean;
109109
dismissed?: boolean;
110110
attempts?: number;
111+
/** iOS only: which mechanism resigned the keyboard (#1598) — 'dismissKey'
112+
* (tapped the keyboard's own Hide/Dismiss/Done key); background-tap dismissal is deliberately unsupported (#1606 review). */
113+
mechanism?: string;
111114
};
112115

113116
export type BackendClipboardTextResult = {

‎src/cli/parser/__tests__/cli-help-command-usage.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ test('usageForCommand documents keyboard dismissal flow', async () => {
6262
const help = await usageForCommand('keyboard');
6363
assert.equal(help === null, false);
6464
assert.match(help ?? '', /To hide the keyboard, use keyboard dismiss/);
65-
assert.match(help ?? '', /taps safe controls like Done/);
65+
assert.match(help ?? '', /taps the keyboard dismiss\/hide key when one is exposed/);
6666
assert.match(help ?? '', /UNSUPPORTED_OPERATION/);
6767
});
6868

‎src/cli/parser/__tests__/cli-help-topics.test.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -276,9 +276,11 @@ test('usageForCommand resolves workflow help topic', async () => {
276276
assert.match(help, /Empty replacement is not a supported clear-field command/);
277277
assert.match(help, /do not plan fill <target> ""/);
278278
assert.match(help, /To hide the keyboard, use keyboard dismiss/);
279+
assert.match(help, /reports UNSUPPORTED_OPERATION rather than tapping elsewhere/);
280+
assert.match(help, /no tap outside the keyboard can be proven side-effect-free/);
279281
assert.match(
280282
help,
281-
/On iOS, if it returns UNSUPPORTED_OPERATION, there is no generic app-agnostic blur fallback/,
283+
/On iOS, if it still returns UNSUPPORTED_OPERATION, both mechanisms were exhausted/,
282284
);
283285
assert.match(help, /On Android, keyboard dismiss first avoids navigation/);
284286
assert.match(help, /use back only when normal back behavior is acceptable/);

‎src/cli/parser/cli-help.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -296,8 +296,8 @@ Text entry:
296296
agent-device type "Handle with care" --delay-ms 80
297297
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
298298
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard usually does not block agent-device interactions; press the next target directly instead of dismissing. If that press fails or reports no visible effect, scroll the target into view or use keyboard enter when submission is wanted.
299-
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed.
300-
On iOS, if it returns UNSUPPORTED_OPERATION, there is no generic app-agnostic blur fallback: do not assume a static text or heading is safe to press, because it can belong to a tappable parent. Use an app-provided dismiss control only when its action is explicitly intended; otherwise report that keyboard dismissal is unavailable.
299+
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps the keyboard's own dismiss/hide key when one is exposed (common on iPad, rare on iPhone) and verifies the keyboard closed. When no dismiss key exists it reports UNSUPPORTED_OPERATION rather than tapping elsewhere — no tap outside the keyboard can be proven side-effect-free. Then prefer submitting (type "\n" on single-line fields) or pressing a known on-screen control that does not mutate state.
300+
On iOS, if it still returns UNSUPPORTED_OPERATION, both mechanisms were exhausted: do not assume a static text or heading is safe to press, because it can belong to a tappable parent. Use an app-provided dismiss control only when its action is explicitly intended; otherwise report that keyboard dismissal is unavailable.
301301
On Android, keyboard dismiss first avoids navigation. If it returns UNSUPPORTED_OPERATION because the current IME needs back navigation, use back only when normal back behavior is acceptable; otherwise report that keyboard dismissal is unavailable.
302302
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
303303
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.

‎src/commands/system/index.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,7 @@ const orientationCliSchema = {
181181
const keyboardCliSchema = {
182182
usageOverride: 'keyboard [status|get|dismiss|enter|return]',
183183
helpDescription:
184-
'Inspect Android keyboard visibility/type or press/dismiss the device keyboard. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available, verifies the keyboard closed, and reports UNSUPPORTED_OPERATION when no safe control is available.',
184+
'Inspect Android keyboard visibility/type or press/dismiss the device keyboard. To hide the keyboard, use keyboard dismiss. It taps the keyboard dismiss/hide key when one is exposed, verifies the keyboard closed, and reports UNSUPPORTED_OPERATION when no dismiss key exists — background taps are never attempted.',
185185
summary: 'Inspect, press, or dismiss the device keyboard',
186186
positionalArgs: ['action?'],
187187
} as const satisfies CommandSchemaOverride;

0 commit comments

Comments
 (0)