Skip to content

Commit c06c8c8

Browse files
committed
fix: soften keyboard guidance for genuinely covered targets
Review on #1173 flagged the unconditional "does not block" claim as false for targets visually covered by the keyboard: direct-selector press resolution is isHittable-gated and falls through to ELEMENT_NOT_FOUND, the tree path allows non-hittable taps with only a no-visible-effect hint, and bottom-pinned-button-under-keyboard (#291/#469/#957) is a real case where dismissal was the remedy. All three strings (runner hint, help manual-qa, help workflow) now say the keyboard USUALLY does not block presses and name concrete fallbacks: scroll the target into view, or keyboard enter when submission is wanted.
1 parent 07ee0df commit c06c8c8

3 files changed

Lines changed: 9 additions & 5 deletions

File tree

‎apple-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1413,7 +1413,7 @@ extension RunnerTests {
14131413
code: "UNSUPPORTED_OPERATION",
14141414
message: "Unable to dismiss the iOS keyboard without a safe native dismiss control",
14151415
hint:
1416-
"The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need."
1416+
"The on-screen keyboard usually does not block agent-device interactions: press the next target directly instead of retrying dismiss. If that press fails or reports no visible effect, scroll the target into view, or use keyboard enter to press the return key when submission is wanted."
14171417
)
14181418
)
14191419
}

‎src/cli/parser/cli-help.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ Targets:
168168
169169
Recovery:
170170
Network/typeahead result missing: wait text "Expected result" or wait <selector>.
171-
Keyboard visible over the next target: the on-screen keyboard does not block agent-device interactions. Do not dismiss the keyboard; press the next target directly.
171+
Keyboard visible over the next target: the on-screen keyboard usually does not block presses, so press the target directly instead of dismissing. If the press fails or reports no visible effect, scroll the target into view or use keyboard enter when submission is wanted.
172172
Sparse or recovered accessibility snapshot: use screenshot as visual truth, leave the bad screen if needed, then retry snapshot -i.
173173
Non-hittable success hint: verify with the settled diff or snapshot; retarget by a better ref/selector if the UI did not change.`,
174174
},
@@ -250,7 +250,7 @@ Text entry:
250250
agent-device press 'id="product-note"'
251251
agent-device type "Handle with care" --delay-ms 80
252252
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
253-
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard does not block agent-device interactions; do not dismiss it just to reach the next control, press that target directly instead.
253+
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard usually does not block agent-device interactions; press the next target directly instead of dismissing. If that press fails or reports no visible effect, scroll the target into view or use keyboard enter when submission is wanted.
254254
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback, or press the next target directly instead of retrying dismiss.
255255
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
256256
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.

‎src/platforms/apple/core/__tests__/runner-client.test.ts‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -646,7 +646,7 @@ test('parseRunnerResponse preserves runner unsupported-operation codes', async (
646646

647647
test('parseRunnerResponse surfaces the keyboard-dismiss hint to press the next target directly', async () => {
648648
const hint =
649-
'The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need.';
649+
'The on-screen keyboard usually does not block agent-device interactions: press the next target directly instead of retrying dismiss. If that press fails or reports no visible effect, scroll the target into view, or use keyboard enter to press the return key when submission is wanted.';
650650
const response = new Response(
651651
JSON.stringify({
652652
ok: false,
@@ -665,8 +665,12 @@ test('parseRunnerResponse surfaces the keyboard-dismiss hint to press the next t
665665
assert.ok(error instanceof AppError);
666666
assert.equal(error.code, 'UNSUPPORTED_OPERATION');
667667
assert.equal(error.details?.hint, hint);
668-
assert.match(String(error.details?.hint), /does not block agent-device interactions/i);
668+
assert.match(
669+
String(error.details?.hint),
670+
/usually does not block agent-device interactions/i,
671+
);
669672
assert.match(String(error.details?.hint), /press the next target directly/i);
673+
assert.match(String(error.details?.hint), /scroll the target into view/i);
670674
assert.match(String(error.details?.hint), /keyboard enter/i);
671675
return true;
672676
},

0 commit comments

Comments
 (0)