Skip to content

Commit 73b492e

Browse files
thymikeeclaude
andcommitted
feat: add Android test IME helper for deterministic text entry (#1198)
Ships a headless InputMethodService (android-ime-helper) as a third Android helper APK, replacing the visible system keyboard during automated sessions. Renders zero accessibility nodes and accepts Unicode/CJK/emoji text over a base64-encoded broadcast channel, fixing both the settle-diff IME-chrome flood and the ASCII-only adb-shell text entry limit in one structural fix. - android-ime-helper/: InputMethodService + build/package scripts on the existing helper-APK toolchain (javac+d8+aapt2+zipalign+apksigner). - src/platforms/android/ime-helper.ts, ime-lifecycle.ts: install/version lifecycle (shared with the other two helpers via the new helper-package-install.ts), activation on session open, and on-device restore-hygiene (previous IME persisted to a device settings key so any daemon/state-dir can recover it; restored on close, daemon teardown, and daemon startup for orphans left by a crashed run). - input-actions.ts: fill/type route through the helper's broadcast channel when active, unicode-safe; unchanged ASCII-shell fallback otherwise. - doctor: new android-test-ime check flags a stuck helper IME with a copy-pasteable `adb shell ime set` remediation command. - Gating: default-on for emulators, opt-in via `open --test-ime` on real devices. - Dead-weight: rewrote the manual ADBKeyBoard workaround doc, dropped the now-provably-live skillgym non-ASCII eval case, updated the ASCII fallback's error message to point at the helper instead of dead-ending. 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>
1 parent ee117c4 commit 73b492e

33 files changed

Lines changed: 1841 additions & 204 deletions

‎.github/workflows/android.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ jobs:
5555
run: |
5656
pnpm package:android-snapshot-helper:npm
5757
pnpm package:android-multitouch-helper:npm
58+
pnpm package:android-ime-helper:npm
5859
5960
- name: Run Android smoke checks
6061
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0

‎.github/workflows/perf-nightly.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ jobs:
102102
run: |
103103
pnpm package:android-snapshot-helper:npm
104104
pnpm package:android-multitouch-helper:npm
105+
pnpm package:android-ime-helper:npm
105106
106107
- name: Run Android command perf benchmark
107108
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,5 @@ android-snapshot-helper/build/
3838
android-snapshot-helper/dist/
3939
android-multitouch-helper/build/
4040
android-multitouch-helper/dist/
41+
android-ime-helper/build/
42+
android-ime-helper/dist/
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
<manifest
2+
xmlns:android="http://schemas.android.com/apk/res/android"
3+
package="com.callstack.agentdevice.imehelper">
4+
<uses-sdk android:minSdkVersion="23" android:targetSdkVersion="36" />
5+
6+
<application
7+
android:debuggable="false"
8+
android:label="Agent Device Test IME"
9+
android:testOnly="true"
10+
android:theme="@android:style/Theme.NoDisplay">
11+
12+
<service
13+
android:name=".TestInputMethodService"
14+
android:label="Agent Device Test IME"
15+
android:permission="android.permission.BIND_INPUT_METHOD"
16+
android:exported="true"
17+
android:directBootAware="true">
18+
<intent-filter>
19+
<action android:name="android.view.InputMethod" />
20+
</intent-filter>
21+
<meta-data
22+
android:name="android.view.im"
23+
android:resource="@xml/method" />
24+
</service>
25+
</application>
26+
</manifest>

‎android-ime-helper/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Android Test IME Helper
2+
3+
A minimal headless `InputMethodService` used as a deterministic, Unicode-safe text-entry backend
4+
for local Android sessions. `onEvaluateInputViewShown()` returns `false` and
5+
`onCreateInputView()` returns `null`, so activating this IME contributes zero accessibility nodes
6+
to the UI tree -- no per-key `[group]+[text]` chrome, no clipboard/translate/voice-dictation
7+
buttons -- unlike the real system keyboard (Gboard et al.), which pulls its visible keyboard into
8+
the UIAutomator tree the moment a field is focused.
9+
10+
Text is injected through a dynamically-registered `BroadcastReceiver` carrying base64-encoded
11+
UTF-8 Intent extras, the same `payloadBase64` convention `android-multitouch-helper` uses for the
12+
same reason: `adb shell` re-tokenizes raw spaces, and base64 sidesteps that entirely. This makes
13+
Unicode/CJK/emoji round-trip exactly, unlike `adb shell input text`, which is ASCII-only.
14+
15+
The helper is a service, not an instrumentation, so it cannot use `android-snapshot-helper`'s and
16+
`android-multitouch-helper`'s `am instrument` invocation shape. It is installed and its version
17+
verified the same way (bundled npm-packaged dist + version-keyed manifest.json), but invoked with
18+
`adb shell ime enable`/`ime set` (lifecycle) and `adb shell am broadcast` (text entry) instead.
19+
20+
## Build
21+
22+
```sh
23+
VERSION="$(node -p 'require("./package.json").version')"
24+
sh ./scripts/build-android-ime-helper.sh "$VERSION" .tmp/android-ime-helper
25+
```
26+
27+
## Run
28+
29+
```sh
30+
PACKAGE="com.callstack.agentdevice.imehelper"
31+
SERVICE="$PACKAGE/.TestInputMethodService"
32+
VERSION="$(node -p 'require("./package.json").version')"
33+
34+
adb install -r -t ".tmp/android-ime-helper/agent-device-android-ime-helper-$VERSION.apk"
35+
36+
# Record the current default IME before switching, so it can be restored exactly.
37+
PREVIOUS_IME="$(adb shell settings get secure default_input_method)"
38+
39+
adb shell ime enable "$SERVICE"
40+
adb shell ime set "$SERVICE"
41+
42+
# Focus a text field, then inject Unicode-safe text via base64 extras.
43+
TEXT_B64="$(printf '%s' '你好世界' | base64)"
44+
adb shell am broadcast -p "$PACKAGE" \
45+
-a com.callstack.agentdevice.imehelper.ACTION_INPUT_TEXT_B64 \
46+
--es text "$TEXT_B64"
47+
48+
# Clear the focused field.
49+
adb shell am broadcast -p "$PACKAGE" -a com.callstack.agentdevice.imehelper.ACTION_CLEAR_TEXT
50+
51+
# Restore the previous IME (critical -- do not skip on a real device).
52+
adb shell ime set "$PREVIOUS_IME"
53+
```
54+
55+
## Broadcast actions
56+
57+
All actions are dynamically registered (`RECEIVER_EXPORTED` on API 33+, matching the receiver fix
58+
ADBKeyBoard itself needed and doesn't reliably have). Every extra is treated as untrusted input:
59+
bounded length, defensive base64 decoding, and the whole handler is wrapped so a malformed
60+
broadcast can never crash the IME process and strand a field without an active input method.
61+
62+
- `ACTION_INPUT_TEXT_B64` (`--es text <base64 utf-8>`) -- commit decoded text at the cursor.
63+
- `ACTION_INPUT_TEXT` (`--es text <string>`) -- commit text directly (subject to `adb shell`'s own
64+
tokenization; prefer the base64 variant for anything with spaces or non-ASCII).
65+
- `ACTION_CLEAR_TEXT` -- select-all and commit an empty string.
66+
- `ACTION_ENTER` -- `performEditorAction(IME_ACTION_DONE)`.
67+
68+
An optional `--es protocol android-ime-helper-v1` extra is a defensive sanity check (not a
69+
security boundary): if present and it doesn't match, the broadcast is dropped and logged.
70+
71+
## Restore hygiene
72+
73+
Switching the active IME is machine-global, not session-scoped. A crashed daemon that switched the
74+
IME and never restored it leaves the device with an invisible keyboard -- on a real device, this
75+
means the user cannot type anywhere until they manually reset their IME in Settings. The TS-side
76+
lifecycle (`src/platforms/android/ime-lifecycle.ts`) persists the previously active IME to disk
77+
before switching, restores it on session close/teardown, and best-effort restores any orphaned
78+
switch left behind by a previous crashed daemon on startup.
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<input-method xmlns:android="http://schemas.android.com/apk/res/android" />
Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
package com.callstack.agentdevice.imehelper;
2+
3+
import android.content.BroadcastReceiver;
4+
import android.content.Context;
5+
import android.content.Intent;
6+
import android.content.IntentFilter;
7+
import android.inputmethodservice.InputMethodService;
8+
import android.os.Build;
9+
import android.util.Base64;
10+
import android.util.Log;
11+
import android.view.View;
12+
import android.view.inputmethod.EditorInfo;
13+
import android.view.inputmethod.InputConnection;
14+
import java.nio.charset.StandardCharsets;
15+
16+
/**
17+
* Minimal headless test IME for agent-device: renders no input view, injects text over a
18+
* base64-encoded broadcast channel. Treats every broadcast extra as untrusted input.
19+
*/
20+
public class TestInputMethodService extends InputMethodService {
21+
private static final String TAG = "AgentDeviceTestIME";
22+
private static final String PROTOCOL = "android-ime-helper-v1";
23+
24+
public static final String ACTION_INPUT_TEXT =
25+
"com.callstack.agentdevice.imehelper.ACTION_INPUT_TEXT";
26+
public static final String ACTION_INPUT_TEXT_B64 =
27+
"com.callstack.agentdevice.imehelper.ACTION_INPUT_TEXT_B64";
28+
public static final String ACTION_CLEAR_TEXT =
29+
"com.callstack.agentdevice.imehelper.ACTION_CLEAR_TEXT";
30+
public static final String ACTION_ENTER =
31+
"com.callstack.agentdevice.imehelper.ACTION_ENTER";
32+
public static final String EXTRA_TEXT = "text";
33+
public static final String EXTRA_PROTOCOL = "protocol";
34+
35+
// Upper bound on a single broadcast payload.
36+
private static final int MAX_TEXT_LENGTH = 32_000;
37+
38+
private BroadcastReceiver receiver;
39+
40+
@Override
41+
public void onCreate() {
42+
super.onCreate();
43+
receiver =
44+
new BroadcastReceiver() {
45+
@Override
46+
public void onReceive(Context context, Intent intent) {
47+
try {
48+
handleAction(intent);
49+
} catch (Throwable error) {
50+
// Never let untrusted extras crash the IME process.
51+
Log.w(TAG, "handleAction failed for " + intent.getAction(), error);
52+
}
53+
}
54+
};
55+
IntentFilter filter = new IntentFilter();
56+
filter.addAction(ACTION_INPUT_TEXT);
57+
filter.addAction(ACTION_INPUT_TEXT_B64);
58+
filter.addAction(ACTION_CLEAR_TEXT);
59+
filter.addAction(ACTION_ENTER);
60+
if (Build.VERSION.SDK_INT >= 33) {
61+
registerReceiver(receiver, filter, Context.RECEIVER_EXPORTED);
62+
} else {
63+
registerReceiver(receiver, filter);
64+
}
65+
Log.i(TAG, "onCreate: receiver registered");
66+
}
67+
68+
@Override
69+
public void onDestroy() {
70+
if (receiver != null) {
71+
unregisterReceiver(receiver);
72+
receiver = null;
73+
}
74+
super.onDestroy();
75+
}
76+
77+
private void handleAction(Intent intent) {
78+
String action = intent.getAction();
79+
if (action == null) return;
80+
if (!isKnownAction(action)) return;
81+
if (!isProtocolAcceptable(intent)) {
82+
Log.w(TAG, "handleAction: rejected " + action + ", protocol mismatch");
83+
return;
84+
}
85+
86+
InputConnection ic = getCurrentInputConnection();
87+
if (ic == null) {
88+
Log.w(TAG, "handleAction: no current input connection for " + action);
89+
return;
90+
}
91+
92+
if (ACTION_INPUT_TEXT.equals(action)) {
93+
commitValidatedText(ic, intent.getStringExtra(EXTRA_TEXT));
94+
} else if (ACTION_INPUT_TEXT_B64.equals(action)) {
95+
commitValidatedText(ic, decodeBase64Text(intent.getStringExtra(EXTRA_TEXT)));
96+
} else if (ACTION_CLEAR_TEXT.equals(action)) {
97+
clearText(ic);
98+
} else if (ACTION_ENTER.equals(action)) {
99+
ic.performEditorAction(EditorInfo.IME_ACTION_DONE);
100+
Log.i(TAG, "enter");
101+
}
102+
}
103+
104+
private static boolean isKnownAction(String action) {
105+
return ACTION_INPUT_TEXT.equals(action)
106+
|| ACTION_INPUT_TEXT_B64.equals(action)
107+
|| ACTION_CLEAR_TEXT.equals(action)
108+
|| ACTION_ENTER.equals(action);
109+
}
110+
111+
// Optional sanity check, not a security boundary; missing extra is accepted.
112+
private static boolean isProtocolAcceptable(Intent intent) {
113+
if (!intent.hasExtra(EXTRA_PROTOCOL)) return true;
114+
return PROTOCOL.equals(intent.getStringExtra(EXTRA_PROTOCOL));
115+
}
116+
117+
private static String decodeBase64Text(String encoded) {
118+
if (encoded == null || encoded.isEmpty()) return null;
119+
if (encoded.length() > MAX_TEXT_LENGTH * 2) {
120+
Log.w(TAG, "decodeBase64Text: payload too large, dropping");
121+
return null;
122+
}
123+
try {
124+
byte[] decoded = Base64.decode(encoded, Base64.DEFAULT);
125+
return new String(decoded, StandardCharsets.UTF_8);
126+
} catch (IllegalArgumentException error) {
127+
Log.w(TAG, "decodeBase64Text: malformed base64 payload", error);
128+
return null;
129+
}
130+
}
131+
132+
private void commitValidatedText(InputConnection ic, String text) {
133+
if (text == null) return;
134+
String bounded = text.length() > MAX_TEXT_LENGTH ? text.substring(0, MAX_TEXT_LENGTH) : text;
135+
ic.commitText(bounded, 1);
136+
Log.i(TAG, "commitText length=" + bounded.length());
137+
}
138+
139+
private void clearText(InputConnection ic) {
140+
ic.beginBatchEdit();
141+
ic.performContextMenuAction(android.R.id.selectAll);
142+
ic.commitText("", 1);
143+
ic.endBatchEdit();
144+
Log.i(TAG, "clearText");
145+
}
146+
147+
@Override
148+
public boolean onEvaluateInputViewShown() {
149+
return false;
150+
}
151+
152+
@Override
153+
public boolean onEvaluateFullscreenMode() {
154+
return false;
155+
}
156+
157+
@Override
158+
public View onCreateInputView() {
159+
return null;
160+
}
161+
}

‎package.json‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,10 @@
9595
"build:android-multitouch-helper": "sh ./scripts/build-android-multitouch-helper.sh $(node -p \"require('./package.json').version\") .tmp/android-multitouch-helper",
9696
"package:android-multitouch-helper": "sh ./scripts/package-android-multitouch-helper.sh $(node -p \"require('./package.json').version\") .tmp/android-multitouch-helper",
9797
"package:android-multitouch-helper:npm": "rm -rf android-multitouch-helper/dist && sh ./scripts/package-android-multitouch-helper.sh $(node -p \"require('./package.json').version\") android-multitouch-helper/dist",
98-
"build:android": "pnpm package:android-snapshot-helper:npm && pnpm package:android-multitouch-helper:npm",
98+
"build:android": "pnpm package:android-snapshot-helper:npm && pnpm package:android-multitouch-helper:npm && pnpm package:android-ime-helper:npm",
99+
"build:android-ime-helper": "sh ./scripts/build-android-ime-helper.sh $(node -p \"require('./package.json').version\") .tmp/android-ime-helper",
100+
"package:android-ime-helper": "sh ./scripts/package-android-ime-helper.sh $(node -p \"require('./package.json').version\") .tmp/android-ime-helper",
101+
"package:android-ime-helper:npm": "rm -rf android-ime-helper/dist && sh ./scripts/package-android-ime-helper.sh $(node -p \"require('./package.json').version\") android-ime-helper/dist",
99102
"build:macos-helper": "swift build -c release --package-path macos-helper",
100103
"build:all": "pnpm build:node && pnpm build:xcuitest",
101104
"ad": "node bin/agent-device.mjs",
@@ -125,7 +128,7 @@
125128
"check:tooling": "pnpm lint && pnpm typecheck && pnpm check:layering && pnpm check:production-exports && pnpm check:mcp-metadata && pnpm build",
126129
"check:unit": "pnpm test:unit && pnpm test:smoke",
127130
"check": "pnpm check:tooling && pnpm check:fallow && pnpm check:unit",
128-
"prepack": "pnpm check:mcp-metadata && pnpm build:all && pnpm package:apple-runner:npm && pnpm package:android-snapshot-helper:npm && pnpm package:android-multitouch-helper:npm",
131+
"prepack": "pnpm check:mcp-metadata && pnpm build:all && pnpm package:apple-runner:npm && pnpm package:android-snapshot-helper:npm && pnpm package:android-multitouch-helper:npm && pnpm package:android-ime-helper:npm",
129132
"typecheck": "tsc -p tsconfig.json",
130133
"test-app:install": "pnpm install --dir examples/test-app --ignore-workspace",
131134
"test-app:start": "pnpm --dir examples/test-app start",
@@ -167,6 +170,9 @@
167170
"android-multitouch-helper/dist",
168171
"!android-multitouch-helper/dist/*.idsig",
169172
"!android-multitouch-helper/README.md",
173+
"android-ime-helper/dist",
174+
"!android-ime-helper/dist/*.idsig",
175+
"!android-ime-helper/README.md",
170176
"src/platforms/linux/atspi-dump.py",
171177
"skills",
172178
"server.json",

0 commit comments

Comments
 (0)