Repository navigation
feat(android): experimental direct Android engine over the engine protocol - #1234
Closed
okwasniewski wants to merge 5 commits into
Closed
okwasniewski wants to merge 5 commits into
okwasniewski wants to merge 5 commits into
Conversation
…tocol A Java engine server that runs on the device as an instrumentation and drives it through UiAutomation, without agent-device: the tree from accessibility nodes, touches and keys as injected events, one connection at a time, cancellation honored, secure fields masked from a full scan. The host half installs the APK, starts it, and forwards a port. The mobile benchmark's e2e.android.config.ts runs the deterministic suite on it.
Contributor
🟢 e2e web: 91 passedAll 91 tests in 41 files
e2e 0.21.2 · 1m 6s · web · run artifacts |
Contributor
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 18 model calls · 72.0k tokens (91% cached) · $0.0035 All 33 tests in 5 files
e2e 0.21.2 · 2m 12s · web · run artifacts |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 21 files
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Turn on auto-fix | Re-trigger cubic
Contributor
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 36 model calls · 191.2k tokens (84% cached) · $0.0126 All 20 tests in 1 file
e2e 0.21.2 · 6m 56s · android-emulator · run artifacts |
Contributor
🟢 e2e android: 55 passed, 1 skippedAll 56 tests in 7 files
e2e 0.21.2 · 7m 48s · android-emulator · run artifacts |
Contributor
🟢 e2e ios: 1 flaky, 52 passed, 1 skipped
|
| Test | Time | |
|---|---|---|
| tests/device-fixture.e2e.ts · 1 flaky, 9 passed | 1m 33s | |
| 🟢 | device fixture › setAppearance flips the color scheme the app reads | 14.9s |
| 🟢 | device fixture › setOrientation rotates the window the app measures | 11.8s |
| device fixture › the clipboard reads back what was written (1 failed attempt first) | 9.0s | |
| 🟢 | device fixture › home leaves the app, openApp brings it back where it was | 5.9s |
| 🟢 | device fixture › closeApp then openApp relaunches on the home list | 6.9s |
| 🟢 | device fixture › closeApp leaves the next commands on the same device | 10.8s |
| 🟢 | device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN | 6.0s |
| 🟢 | device fixture › device.back pops the scenario | 6.8s |
| 🟢 | device fixture › setLocation and clearLocation change what the app reads | 10.9s |
| 🟢 | device fixture › enrollBiometrics and setBiometrics answer a biometric prompt | 10.2s |
| ⏭️ | tests/controls.e2e.ts · 23 passed, 1 skipped | 3m 17s |
| 🟢 | control inventory › a label query resolves a labeled text field | 11.6s |
| 🟢 | control inventory › a labeled field answers name, display value, value reads, fill, and clear | 23.4s |
| 🟢 | control inventory › a placeholder query and attribute reads find the field | 6.5s |
| 🟢 | control inventory › a secure field takes a fill and denies value reads | 10.2s |
| 🟢 | control inventory › press sends a character and Enter through the soft keyboard | 7.9s |
| 🟢 | control inventory › a field autofocuses on mount and focus() moves the focus | 10.0s |
| 🟢 | control inventory › toBeFocused follows the focus | 9.0s |
| 🟢 | control inventory › a switch checks, unchecks, and reports its state | 10.0s |
| 🟢 | control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap | 8.4s |
| 🟢 | control inventory › checkbox and radio roles check, uncheck, and report their state | 13.0s |
| 🟢 | control inventory › a radio group reports the checked option | 9.1s |
| 🟢 | control inventory › buttons report disabled, enabled, and a state that arrives later | 8.0s |
| 🟢 | control inventory › a display: none element is absent rather than hidden, its visible twin attached | 7.7s |
| 🟢 | control inventory › a header answers its accessible name | 4.4s |
| 🟢 | control inventory › a tab row answers count, all, texts, filters, positions, and switching | 6.9s |
| 🟢 | control inventory › a list answers counts, list-form text, reads, and boxes in order | 8.2s |
| ⏭️ | control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…) | |
| 🟢 | control inventory › a long press lands as one | 10.1s |
| 🟢 | control inventory › pointer verbs land at the point asked for | 11.0s |
| 🟢 | control inventory › an ambiguous locator fails at once | 6.2s |
| 🟢 | control inventory › screenshot, restart, and clearState reopen the app on its home list | 13.2s |
| 🟢 | refusals › verbs a device never declares are refused before the engine | 3ms |
| 🟢 | refusals › keys the soft keyboard cannot send are refused by the engine | 492ms |
| 🟢 | refusals › a missing locator fails as not found once its timeout passes | 1.3s |
| 🟢 | tests/device.e2e.ts · 2 passed | 54.2s |
| 🟢 | sequential onboarding echoes the exact values | 40.5s |
| 🟢 | onboarding rejects a malformed email before advancing | 13.7s |
| 🟢 | tests/flows-lists.e2e.ts · 7 passed | 3m 58s |
| 🟢 | infinite scroll reaches the target item | 27.2s |
| 🟢 | product catalog adds exactly two of the right variant | 11.2s |
| 🟢 | gestures: long-press, then swipe left | 9.7s |
| 🟢 | sticky chrome: accept below the fold, then continue | 12.1s |
| 🟢 | async states: load, pull to refresh, claim through the toast | 13.4s |
| 🟢 | huge virtualized list reaches row 512 | 2m 28s |
| 🟢 | web view coupon form applies the code shown on the page | 16.9s |
| 🟢 | tests/flows.e2e.ts · 7 passed | 1m 22s |
| 🟢 | text inputs unlock the submit button | 15.3s |
| 🟢 | filled text inputs report their values | 11.3s |
| 🟢 | modal flow confirms through the native alert | 10.5s |
| 🟢 | bottom tabs act inside the last tab | 9.6s |
| 🟢 | error recovery retries, then confirms the delete | 11.4s |
| 🟢 | choice controls place the exact order | 10.5s |
| 🟢 | debounced search selects the target once results arrive | 13.3s |
| 🟢 | tests/login-form.e2e.ts · 3 passed | 57.3s |
| 🟢 | login form › rejects a malformed email | 16.0s |
| 🟢 | login form › rejects an unknown account | 15.3s |
| 🟢 | login form › signs in and logs out | 26.0s |
| 🟢 | tests/smoke.e2e.ts · 1 passed | 15.0s |
| 🟢 | home lists the scenarios and opens one | 15.0s |
e2e 0.21.2 · 6m 53s · ios-simulator · run artifacts
Contributor
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 10 replayed from cache · 70 model calls · 441.8k tokens (79% cached) · $0.0342 All 22 tests in 1 file
e2e 0.21.2 · 13m 17s · ios-simulator · run artifacts |
…the SDK debug key The server masks fields found before and after a capture, refuses an id without a method, and waits for a vanished session's work however long it takes. The build checks against the platform API and signs with the SDK's debug keystore, replacing a server signed with another key. Device cleanup gets its own budget and runs when startup fails.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Turn on auto-fix | Re-trigger cubic
…engine Modifiers and named keys beyond Enter and Space are refused with the mobile engine's reasons, and scrollIntoView is no longer declared, so the mobile benchmark's selection passes whole on the direct engine.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Turn on auto-fix | Re-trigger cubic
This was referenced Oct 19, 2026
Member
Author
|
Closing: experiment done, it validated the engine protocol (#1232) with no protocol changes needed. Branch kept for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1232. An Android engine that is an engine protocol server written in Java, running on the device: the proof that an engine can live outside the runner's process and language, and a measure of what driving the device directly buys over agent-device.
@e2e-dev/android(private, experimental)A Java engine server (
packages/android/server/java/) that runs on the device as an instrumentation. It listens on an abstract socket and drives the device in process throughUiAutomation: the tree comes from accessibility nodes, and touches and keys are injected events. There is no adb process, no daemon hop, and no XML dump per call. The host half (src/transport.ts) installs the APK, starts the instrumentation, and forwards a port. The APK builds in seconds withjavac,d8,aapt2, andapksigner, no Gradle.Mobile benchmark, deterministic Android suite (
apps/mobile-benchmark/e2e.android.config.ts, 42 tests, Pixel 7 API 36 emulator, one worker):Engine calls on the same screens (
packages/android/bench/bench.ts, nearest-rank p50, 5-20 samples per row, init one cold sample):It wins wherever agent-device spends an adb process or a daemon hop per call. It loses where the app sets the pace:
swipewaits for the list to come to rest.backis held up by the pop animation, which blocks accessibility queries for about 450 ms.keyboard.typesends one key event per character, and React Native re-renders on each. agent-device commits the whole string through its IME.fillsets text withACTION_SET_TEXT; agent-device types it and reads it back.Both engines pass all 42, twice in a row. The direct engine declares the same device surface as the mobile engine:
presstakes Enter, Space, or one character, the keys a soft keyboard has, and refuses modifiers and other named keys with the mobile engine's reasons.Not there yet (in its README): one device per run, a
devicefixture withback()only, no video or state capture, noselectorlocators, nolaunchArgumentsorpermissions(refused at config load).Verified
Ran it locally: yes
node node_modules/e2e/dist/cli/bin.js run --config e2e.android.config.ts, withBENCH_ENGINE=agent-devicefor the baseline) andnode bench/bench.ts --rounds 20.