You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cf6be5b
Browse filesBrowse the repository at this point in the historyBrowse files
Apply the writing-user-docs house style across every website docs page,
the root README, and the TestMu provider README: second person, active
voice, no time-relative or marketing wording, prerequisites up front,
goal-oriented headings, and less internal implementation detail.
Also close the unterminated selector-helpers snippet in client-api.md and
map agent-device/selectors in examples/sdk/tsconfig.json so the snippet
compile test covers it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: README.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,7 @@ agent-device doctor
31
31
agent-device help workflow
32
32
```
33
33
34
-
Run `doctor` yourself before handing the CLI to an agent;`help workflow` links to the guides for debugging, replay, and profiling, and the installed help always matches the installed version.
34
+
Run `doctor` yourself before you hand the CLI to an agent.`help workflow` links to the guides for debugging, replay, and profiling; installed help always matches the installed version.
Refs are only valid from the latest output: after a `--settle` command, use the refs in its diff, and take a new snapshot only if the diff omits what you need. Snapshots come from the app's accessibility tree, so clear labels, roles, and test IDs make agent runs more reliable; use screenshots and video as evidence or when accessibility data is poor.
64
+
Refs are valid only in the latest output: after a `--settle` command, use the refs in its diff, and take a new snapshot only if the diff omits what you need. Snapshots come from the app's accessibility tree, so clear labels, roles, and test IDs make agent runs more reliable; use screenshots and video as evidence or when accessibility data is poor.
65
65
66
66

67
67
@@ -143,9 +143,9 @@ The same session and evidence model works at every step: the agent explores the
143
143
144
144
`agent-device` keeps device state in sessions. It uses a local accessibility bridge for iOS Simulator snapshots and XCTest for iOS interactions, physical iOS, and tvOS; ADB and the snapshot helper on Android; HDC and ArkUI `uitest` on HarmonyOS; Vega CLI/VDA on the Vega Virtual Device; a local helper on macOS; and AT-SPI on Linux.
145
145
146
-
Support depth varies by target. Newer backends such as HarmonyOS and Vega OS cover a subset of commands; run `agent-device capabilities --platform <platform>` to see what a target supports.
146
+
Support depth varies by target. Some backends, such as HarmonyOS and Vega OS, cover a subset of commands; run `agent-device capabilities --platform <platform>` to see what a target supports.
147
147
148
-
Sessions are scoped to the caller's git worktree, and host-local device claims stop parallel agents from taking over each other's simulators and emulators. Inspect ownership without a daemon via `agent-device device status`, and settle provably dead owners with `agent-device device release --stale`. The same commands drive hosted devices on [BrowserStack, AWS Device Farm, TestMu AI, and Limrun](https://oss.callstack.com/agent-device/docs/device-clouds).
148
+
Sessions are scoped to the caller's git worktree, and host-local device claims stop parallel agents from taking over each other's simulators and emulators. Run `agent-device device status` to inspect ownership without a daemon, and `agent-device device release --stale` to release claims whose owner is provably no longer running. The same commands drive hosted devices on [BrowserStack, AWS Device Farm, TestMu AI, and Limrun](https://oss.callstack.com/agent-device/docs/device-clouds).
149
149
150
150
`agent-device` uses the inspect-act-verify process from Vercel's [agent-browser](https://github.com/vercel-labs/agent-browser) for mobile, TV, and desktop apps. Basic `--platform web` support runs `agent-browser` in the same session and replay system.
See the [TestMu AI guide](https://agent-device.dev/docs/testmu) for setup and supported operations.
15
15
To update the plugin, run `agent-device plugins update @agent-device/testmu`.
16
16
17
-
After adding or updating a plugin, close your sessions and run `agent-device daemon stop` before reconnecting.
17
+
After you add or update the plugin, close your sessions and run `agent-device daemon stop` before reconnecting.
Copy file name to clipboardExpand all lines: website/docs/docs/ai-sdk.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ title: AI SDK
4
4
5
5
# AI SDK
6
6
7
-
Use `agent-device/ai-sdk` to give an [AI SDK](https://ai-sdk.dev/) agent a typed set of tools for navigating and inspecting an app. The tools run in-process, share one named session, and default to the focused perceive-and-act surface most agents need.
7
+
Use `agent-device/ai-sdk` to give an [AI SDK](https://ai-sdk.dev/) agent a typed set of tools for navigating and inspecting an app. The tools run in-process, share one named session, and by default cover the core loop of reading the screen and acting on it.
8
8
9
9
```bash
10
10
pnpm add agent-device ai
@@ -48,13 +48,13 @@ try {
48
48
}
49
49
```
50
50
51
-
Set `AI_MODEL` to a model available through your configured AI SDK provider. The agent sees the available device tools and chooses the calls needed to complete the prompt. The returned `client` targets the same session; keep cleanup in `finally` so the device is released even if generation fails.
51
+
Set `AI_MODEL` to a model available through your configured AI SDK provider. The agent decides which device tools to call to complete the prompt. The returned `client` targets the same session; close it in `finally` so the device is released even if generation fails.
52
52
53
-
## Options
53
+
## Choose tools and approvals
54
54
55
55
`set: 'core'` is the default. It exposes the perceive-and-act loop: open, close, snapshot, click, press, fill, type, get, is, find, wait, back, scroll, swipe, alert, and screenshot.
56
56
57
57
- Pass `set: 'all'` when the agent also needs device-management or observability commands.
58
-
- Pass `approval: { close: 'user-approval' }` to require approval for a command. `createAgentDeviceTools()` returns the map as `toolApproval`, ready to pass to [`ToolLoopAgent`](https://ai-sdk.dev/docs/agents/tool-approvals).
58
+
- Pass `approval: { close: 'user-approval' }` to require user approval before a command runs. `createAgentDeviceTools()` returns the map as `toolApproval`, ready to pass to [`ToolLoopAgent`](https://ai-sdk.dev/docs/agents/tool-approvals).
59
59
60
-
See the [Node.js API](/docs/client-api) when the host application needs deterministic setup or other direct device control outside the agent loop. See the AI SDK reference for [`ToolLoopAgent`](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool-loop-agent).
60
+
Use the [Node.js API](/docs/client-api) when your application needs deterministic setup or other direct device control outside the agent loop. See the AI SDK reference for [`ToolLoopAgent`](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool-loop-agent).
Use AWS Device Farm for hosted Android and iOS remote-access WebDriver sessions. The adapter does not route Vega OS or accept a Vega Fire TV ARN. The initial Vega workflow uses a local VVD.
8
+
Use AWS Device Farm to run agent-device on hosted Android and iOS devices through remote-access WebDriver sessions. You need an AWS account with a Device Farm project and the AWS CLI installed. AWS Device Farm does not cover Vega OS: agent-device rejects a Vega Fire TV ARN. For Vega, use a local Vega Virtual Device (VVD).
9
9
10
-
## Credentials and connection
10
+
## Set credentials and connect
11
11
12
-
AWS Device Farm uses the AWS CLI credential provider chain. `agent-device` runs `aws devicefarm ...`, so it works with any non-interactive AWS CLI credential source available in CI. It does not require`aws login`. See the [AWS CLI environment variable reference](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for supported credential sources.
12
+
agent-device runs `aws devicefarm ...`, so it uses the AWS CLI credential provider chain and works with any non-interactive AWS CLI credential source in CI. You do not need`aws login`. See the [AWS CLI environment variable reference](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for supported credential sources.
13
13
14
-
Use short-lived CI credentials instead of long-lived IAM user keys. In GitHub Actions, use OIDC to assume an IAM role and let the action export standard AWS environment variables. AWS documents [IAM OIDC providers](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html), and the official [`configure-aws-credentials` action](https://github.com/aws-actions/configure-aws-credentials) documents the GitHub Actions setup.
14
+
Use short-lived CI credentials instead of long-lived IAM user keys. In GitHub Actions, use OIDC to assume an IAM role and let the action export the standard AWS environment variables. AWS documents [IAM OIDC providers](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html), and the official [`configure-aws-credentials` action](https://github.com/aws-actions/configure-aws-credentials) documents the GitHub Actions setup.
`--aws-app-arn`is optional when the remote-access session does not need an uploaded app. You can also provide the ARNs through environment variables:
43
+
Omit `--aws-app-arn` when the remote-access session does not need an uploaded app. You can also pass the ARNs through environment variables:
44
44
45
45
```bash
46
46
export AWS_DEVICE_FARM_PROJECT_ARN=...
47
47
export AWS_DEVICE_FARM_DEVICE_ARN=...
48
48
export AWS_DEVICE_FARM_APP_ARN=...
49
49
```
50
50
51
-
`AGENT_DEVICE_AWS_DEVICE_FARM_PROJECT_ARN`, `AGENT_DEVICE_AWS_DEVICE_FARM_DEVICE_ARN`, and `AGENT_DEVICE_AWS_DEVICE_FARM_APP_ARN`are accepted as agent-device-specific aliases.
51
+
agent-device also accepts `AGENT_DEVICE_AWS_DEVICE_FARM_PROJECT_ARN`, `AGENT_DEVICE_AWS_DEVICE_FARM_DEVICE_ARN`, and `AGENT_DEVICE_AWS_DEVICE_FARM_APP_ARN`as aliases.
52
52
53
-
`connect` makes read-only `get-project`, `get-device`, and, when supplied, `get-upload` calls. It rejects a device or app for the wrong platform, and an app upload that is not ready. AWS Device Farm does not support app installation after remote-access session allocation. When an app is required, run the printed reconnect command, including `--session <name> --force`, before `open`.
53
+
`connect` makes read-only `get-project`, `get-device`, and, when you pass an app, `get-upload` calls. It rejects a device or app for the wrong platform, and an app upload that is not ready. You cannot install an app once the remote-access session is allocated. When an app is required, run the printed reconnect command, including `--session <name> --force`, before `open`.
54
54
55
-
## CLI workflow
55
+
## Run a session from the CLI
56
56
57
-
Every unscoped `connect` creates a fresh connection. The printed next steps include its generated `--session`. Keep that flag on every command when multiple processes or CI jobs share a host. The active connection is only safe for one sequential workflow. To replace a named connection, run `connect ... --session <name> --force`. An unscoped`--force` creates a new connection and leaves existing sessions untouched.
57
+
Every `connect`without `--session`creates a new connection, and the printed next steps include its generated `--session`. Keep that flag on every command when several processes or CI jobs share a host; the active connection is safe only for one sequential workflow. To replace a named connection, run `connect ... --session <name> --force`. `--force` without`--session` creates a new connection and leaves existing sessions untouched.
58
58
59
59
```bash
60
60
export AWS_REGION=us-west-2
@@ -76,11 +76,11 @@ agent-device artifacts --json
76
76
agent-device disconnect
77
77
```
78
78
79
-
For MCP-only use, run `connect` in the same effective state directory before starting `agent-device mcp`. MCP exposes operational tools but not provider `connect` commands.
79
+
To use AWS Device Farm only through MCP, run `connect` in the same effective state directory before you start `agent-device mcp`. MCP exposes device commands such as `open`, `snapshot`, `close`, and `artifacts`, but not provider `connect` commands.
80
80
81
-
## Node.js client
81
+
## Use the Node.js client
82
82
83
-
Use direct client configuration when the Node process manages AWS credentials and selectors:
83
+
Configure the client directly when your Node process manages the AWS credentials and selectors:
After `close`, AWS Device Farm can return remote-access video and log artifacts after the provider finalizes them. Run `agent-device artifacts --json`, or look up a previous session explicitly:
103
+
After `close`, AWS Device Farm returns remote-access video and log artifacts once it finalizes them. Run `agent-device artifacts --json`, or look up an earlier session by its ARN:
If `connect` fails, use the reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. The provider has not allocated a device yet. If artifacts are pending immediately after `close`, retry the lookup.
109
+
If `connect` fails, use the reported `aws devicefarm get-*` error to check the credential chain, ARN, region, resource platform, or upload readiness. A failed `connect`has not allocated a device yet. If artifacts are still pending right after `close`, retry the lookup.
110
110
111
-
On hosted WebDriver sessions, `fill` checks that the field received focus before it sends keys. If it cannot confirm focus, it fails without typing. Use `snapshot -i` to confirm the target. If the driver cannot expose focus at all, use `press <target>` followed by `type <text>`. That sends text without confirming the destination.
111
+
On hosted WebDriver sessions, `fill` checks that the field received focus before it sends keys. If it cannot confirm focus, it fails without typing. Use `snapshot -i` to confirm the target. If the driver cannot report focus at all, use `press <target>` followed by `type <text>`, which sends text without confirming where it lands.
112
112
113
-
A screen that never goes still — a looping video, a live ticker, continuous animation — gives the provider's driver no quiet moment to read the UI tree, so `snapshot -i` can run out of its read budget on a rented device while `screenshot` of the same screen still returns. Snapshots covers what that failure means and what to do instead; on a metered device the difference matters, because every second of the walk is billed. Take the screenshot, drive from `@refs` an earlier snapshot captured, and remember that `--depth` trims a tree after it arrives, so it cannot shorten a read that never returned.
113
+
On a screen that never goes still, such as a looping video, a live ticker, or continuous animation, `snapshot -i` can time out while `screenshot` of the same screen still returns. On a metered device every second of that read is billed, so do not retry the snapshot in a loop. Take a screenshot instead and drive from `@refs` an earlier snapshot captured. `--depth` trims the tree after it arrives, so it cannot shorten a read that never finishes. See [Snapshots](/docs/snapshots) for details.
-`input` is required and uses the same fields as the matching MCP/Node command.
55
-
-Unknown top-level step fields are rejected. Supported keys are `command`, `input`, and `runtime`.
56
-
-The legacy `positionals`/`flags`step shape was removed in 0.21. Migrate each step to structured input, for example `{"command":"open","input":{"app":"settings","platform":"ios"}}`.
57
-
-nested `batch` and `replay` steps are rejected.
58
-
-`--on-error stop` is the supported behavior.
53
+
-`input` is required and takes the same fields as the matching MCP or Node client command.
54
+
-A step can only have `command`, `input`, and `runtime` keys; any other key is rejected.
55
+
-Steps written as `positionals`/`flags`are rejected (that shape was removed in 0.21). Rewrite each step with structured input, for example `{"command":"open","input":{"app":"settings","platform":"ios"}}`.
56
+
-`batch` and `replay` steps can't be nested inside a batch.
57
+
-`--on-error stop` is the only error mode: the batch stops at the first failing step.
59
58
60
59
## Response shape
61
60
@@ -78,7 +77,7 @@ Success:
78
77
}
79
78
```
80
79
81
-
In non-JSON mode, `batch`also prints a short per-step summary after the overall completion line.
80
+
Without `--json`, `batch` prints the overall completion line followed by a short summary of each step.
82
81
83
82
Failure:
84
83
@@ -102,16 +101,16 @@ Failure:
102
101
}
103
102
```
104
103
105
-
## Agent best practices
104
+
## Tips for agents
106
105
107
-
-Batch only one related screen flow at a time.
108
-
- After mutating steps (`open`, `click`, `fill`, `swipe`), add a sync guard (`wait`, `is exists`) before critical reads.
109
-
- Treat prior refs/snapshots as stale after UI changes.
106
+
-Keep each batch to one screen flow.
107
+
- After steps that change the UI (`open`, `click`, `fill`, `swipe`), add a `wait` or `is exists` step before reads that matter.
108
+
- Treat earlier refs and snapshots as stale after the UI changes.
110
109
- Prefer `--steps-file` over inline JSON.
111
-
- Keep batches moderate (about 5-20 steps).
112
-
-Replan from the failing step using `details.step` and `details.partialResults`.
110
+
- Keep batches to about 5-20 steps.
111
+
-When a batch fails, use `details.step` and `details.partialResults` to replan from the failing step.
113
112
114
-
## Canonical recipes
113
+
## Example recipes
115
114
116
115
Open app -> open thread -> type -> send
117
116
@@ -150,9 +149,10 @@ Open app -> open action menu -> choose option -> verify
150
149
]
151
150
```
152
151
153
-
## Stale accessibility tree risk
152
+
## Avoid acting on a stale accessibility tree
154
153
155
-
Rapid UI changes can outpace accessibility tree updates. Mitigate by inserting explicit waits and splitting long workflows into phases:
154
+
The accessibility tree can lag behind fast UI changes. Add explicit waits, and split long workflows
0 commit comments