Skip to content

Commit 07ee0df

Browse files
committed
fix: guide agents past iOS keyboard dismissal and get-text guesses
Tonight's benchmark leaderboard showed two recurring agent-UX misses: keyboard dismiss failing after fill (5x, now the #1 failed command) and get-text guessed as a command name (1x). - iOS keyboardDismiss now returns a hint explaining the on-screen keyboard does not block agent-device interactions, so agents should press the next target directly instead of retrying dismiss, and use keyboard enter only when submission is actually wanted. - help manual-qa and help workflow recovery text no longer teach the false "dismiss to unblock the target" pattern. - The command-suggestion curated map now maps get-text/gettext/get_text to the real get text command shape; the generic edit-distance fallback did not produce any suggestion for these guesses.
1 parent 5c08f16 commit 07ee0df

5 files changed

Lines changed: 50 additions & 4 deletions

File tree

‎apple-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1411,7 +1411,9 @@ extension RunnerTests {
14111411
ok: false,
14121412
error: ErrorPayload(
14131413
code: "UNSUPPORTED_OPERATION",
1414-
message: "Unable to dismiss the iOS keyboard without a safe native dismiss control"
1414+
message: "Unable to dismiss the iOS keyboard without a safe native dismiss control",
1415+
hint:
1416+
"The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need."
14151417
)
14161418
)
14171419
}

‎src/cli/parser/__tests__/command-suggestions.test.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,18 @@ test('input and settext suggest fill', () => {
9797
}
9898
});
9999

100+
test('get-text, gettext, and get_text suggest get text', () => {
101+
for (const guess of ['get-text', 'gettext', 'get_text']) {
102+
assert.throws(
103+
() => parseArgs([guess]),
104+
(error: unknown) =>
105+
error instanceof AppError &&
106+
error.code === 'INVALID_ARGS' &&
107+
error.message === `Unknown command: ${guess}. Did you mean get text?`,
108+
);
109+
}
110+
});
111+
100112
test('screencap and capture suggest screenshot', () => {
101113
for (const guess of ['screencap', 'capture']) {
102114
assert.throws(

‎src/cli/parser/cli-help.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ Targets:
168168
169169
Recovery:
170170
Network/typeahead result missing: wait text "Expected result" or wait <selector>.
171-
Keyboard blocks target: keyboard dismiss when supported, then snapshot -i.
171+
Keyboard visible over the next target: the on-screen keyboard does not block agent-device interactions. Do not dismiss the keyboard; press the next target directly.
172172
Sparse or recovered accessibility snapshot: use screenshot as visual truth, leave the bad screen if needed, then retry snapshot -i.
173173
Non-hittable success hint: verify with the settled diff or snapshot; retarget by a better ref/selector if the UI did not change.`,
174174
},
@@ -250,8 +250,8 @@ Text entry:
250250
agent-device press 'id="product-note"'
251251
agent-device type "Handle with care" --delay-ms 80
252252
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
253-
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. Blocked control: try keyboard dismiss when supported.
254-
To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback.
253+
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard does not block agent-device interactions; do not dismiss it just to reach the next control, press that target directly instead.
254+
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback, or press the next target directly instead of retrying dismiss.
255255
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
256256
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.
257257
Android Gboard handwriting/stylus UI can capture text in an IME-owned input instead of the app field. If fill reports that input was captured by the keyboard/IME, use the diagnostic targetInput/actualInput details, inspect keyboard status/get if needed, and switch or disable handwriting outside the command plan before retrying. Do not keep retrying fill/type against the same field while the IME owns focus.

‎src/cli/parser/command-suggestions.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@ const COMMAND_ALIAS_SUGGESTIONS: Record<string, CommandAliasSuggestion> = {
3535
screencap: { command: 'screenshot', example: 'screenshot' },
3636
capture: { command: 'screenshot', example: 'screenshot' },
3737
dismiss: { command: 'keyboard', example: 'keyboard dismiss' },
38+
'get-text': { command: 'get', example: 'get text' },
39+
gettext: { command: 'get', example: 'get text' },
40+
get_text: { command: 'get', example: 'get text' },
3841
};
3942

4043
export function listCommandAliasSuggestionEntries(): Array<[string, CommandAliasSuggestion]> {

‎src/platforms/apple/core/__tests__/runner-client.test.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -644,6 +644,35 @@ test('parseRunnerResponse preserves runner unsupported-operation codes', async (
644644
);
645645
});
646646

647+
test('parseRunnerResponse surfaces the keyboard-dismiss hint to press the next target directly', async () => {
648+
const hint =
649+
'The on-screen keyboard does not block agent-device interactions: snapshot refs and presses still work through it. Press the next target directly instead of retrying dismiss; use keyboard enter to press the return key if submission is what you actually need.';
650+
const response = new Response(
651+
JSON.stringify({
652+
ok: false,
653+
error: {
654+
code: 'UNSUPPORTED_OPERATION',
655+
message: 'Unable to dismiss the iOS keyboard without a safe native dismiss control',
656+
hint,
657+
},
658+
}),
659+
);
660+
const session = { ready: false };
661+
662+
await assert.rejects(
663+
() => parseRunnerResponse(response, session, '/tmp/runner.log'),
664+
(error: unknown) => {
665+
assert.ok(error instanceof AppError);
666+
assert.equal(error.code, 'UNSUPPORTED_OPERATION');
667+
assert.equal(error.details?.hint, hint);
668+
assert.match(String(error.details?.hint), /does not block agent-device interactions/i);
669+
assert.match(String(error.details?.hint), /press the next target directly/i);
670+
assert.match(String(error.details?.hint), /keyboard enter/i);
671+
return true;
672+
},
673+
);
674+
});
675+
647676
test('parseRunnerResponse preserves iOS AX snapshot failure code and hint', async () => {
648677
const hint =
649678
'Try a smaller read such as snapshot -s <visible label or id> -d 8, or use direct selector commands such as find id <value> click.';

0 commit comments

Comments
 (0)