title	Replay & E2E Testing

Replay & E2E Testing

Agents use refs for exploration and authoring. Replay scripts are deterministic runs that can be used for E2E testing.

Core model

Two-pass workflow:

1. Agent pass: discover and interact with refs (snapshot -> click @e.. / fill @e..).
2. Deterministic pass: run recorded .ad script with replay.

Record a replay script

Enable recording during a session:

agent-device open Settings --platform ios --session e2e --save-script
agent-device snapshot -i --session e2e
agent-device click @e13 --session e2e
agent-device close --session e2e

By default, on close, a replay script is written to:

~/.agent-device/sessions/<session>-<timestamp>.ad

You can also provide a custom output file path:

agent-device open Settings --platform ios --session e2e --save-script ./workflows/e2e-settings.ad

• --save-script value is treated as a file path.
• Parent directories are created automatically when they do not exist.
• For ambiguous bare values, use --save-script=workflow.ad or a path-like value such as ./workflow.ad.

.ad line grammar

A .ad line is the CLI spelling of one command: <command> [positional ...] [flag ...]. Whitespace separates tokens, so a value with a space needs quotes.

open "com.example.app" --relaunch
scroll down --until 'id="far-button"'
press id="far-button"
wait 'label="Order summary"' 5000
close

• A token quoted with " or ' is one argument. Single quotes keep a " literal, so 'id="far-button"' and "id=\"far-button\"" are the same selector — write whichever matches how you typed the command at the shell.
• Values in double quotes are JSON strings, so escape \\, \", \t, and \n. Values in single quotes are literal except for \' and \\.
• A command accepts only its own flags. A flag the CLI accepts is available in a script when it belongs to the step; per-request options are not part of a step. --settle, --verify, and the device-selection flags (--platform, --serial, --device) are the common ones a script does not carry.
• help <command> prints the flags each command accepts. help scripting prints this grammar.

Reaching an off-screen element is viewport-independent in a script exactly as it is at the CLI. Prefer the stop condition over a fixed amount, which passes on one screen size and fails on another:

# repeats until the element is on screen
scroll down --until 'id="checkout-submit"'
# one gesture, viewport-relative
scroll down 0.8
# run to the end of the content
scroll bottom

A # only starts a comment at the beginning of a line. A scroll line carries --until and keeps its distance as a positional; --pixels and --duration-ms belong to the CLI invocation, not to a step. wait carries --raw, --depth <n>, and --scope <selector|@ref> to choose the capture its target is read from.

Run replay

agent-device replay ~/.agent-device/sessions/e2e-2026-02-09T12-00-00-000Z.ad --session e2e-run

• Replay reads .ad scripts.

• A script without a terminal close already leaves its session active. For an existing script that does end in close, pass --keep-session to suppress only that final action and continue with interactive commands in the same session:

  agent-device replay ./checkout.ad --session e2e-run --keep-session
  agent-device snapshot -i --session e2e-run

  Interior close actions still run. The flag is intentionally unavailable to test because suite attempts own cleanup, and it is rejected for Maestro YAML because that runtime owns its lifecycle.

• press, click, and longpress steps wait up to 2 seconds for their target to appear before they fail. A step recorded against a screen that was still loading passes on replay once the target shows up. The wait covers only a target that is not on screen yet: a target that is covered, off-screen, or matched by more than one element fails at once, as it does live. A step that waited and then passed shows no trace of the wait in the replay output; run with --debug to see it in the diagnostics.

• When the target never appears, replay stops with REPLAY_DIVERGENCE, and error.details.readiness says how long the step waited and how many times it looked (waitedMs, polls, end). For a step recorded with a target annotation (the # agent-device:target-v1 line above it), error.details.divergence.kind is selector-miss and the step is never sent. For a step without an annotation, error.details.reason is selector_not_found, as for a live command. If the app shows an empty accessibility tree during that wait, error.details.reason is capture_sparse instead; take a snapshot to see where the app is.

Run Maestro compatibility flows

Agent Device can run a supported subset of Maestro YAML through its typed Maestro compatibility runtime:

agent-device replay ./flow.yaml --maestro --platform ios --session e2e-run
agent-device test ./maestro-flows --maestro --platform android --artifacts-dir ./tmp/maestro-artifacts

Supported subset:

• Flows: launchApp (with clearState, permissions, and Apple-only launch arguments; permissions apply after state clearing but before launch, and a launchApp without permissions touches nothing — there is no silent all: allow default); setPermissions (mid-flow permission grants, denials, and resets; all resolves in the backend — one simctl call on iOS, the declared permissions on Android — with specific entries overriding after it); runFlow file/inline with platform, visibility, and limited boolean conditions; onFlowStart/onFlowComplete; repeat.times and retry.
• Interactions: tapOn, doubleTapOn, longPressOn, inputText on the focused element, eraseText, openLink, hideKeyboard, basic pressKey, and back; selector targets poll until available and support recursive index, childOf, above, below, leftOf, rightOf, containsChild, containsDescendants, points, and optional; outer command labels are metadata, not target selectors.
• Assertions and navigation: assertVisible, assertNotVisible, assertTrue (literal values and ${VAR} lookups only; "", "false", "0", "null", and "undefined" are falsy, everything else is truthy), extendedWaitUntil, scroll, scrollUntilVisible, absolute/percentage/target swipe, takeScreenshot, waitForAnimationToEnd, clearState, and stopApp.
• Scripts: ordered runScript file/env scripts with http.post, json, and output variables; evalScript inline expressions run flow-scoped JavaScript and write output.* leaves for later steps.

Boundaries:

• Permissions: every entry is one settings permission call, applied in order with all first; the step stops at the first entry the selected platform refuses, earlier entries stay applied, and the error names what landed. Android’s only allow level is while-in-use, so location: inuse and location: never mean allow and deny there, while location: always and photos: limited are Apple-only and fail. On iOS, which service a runtime changes is simctl privacy’s own verdict: current runtimes refuse a targeted notifications change and leave notifications untouched under all.
• Runtime: iOS and Android only; launchApp.clearState and standalone clearState support Android and iOS simulators, launch arguments are Apple-only, and other standalone device utility/state commands are unsupported.
• Expressions: evalScript is the only command whose payload is evaluated as JavaScript (flow env and prior output leaves are string-typed); with that exception, fields stay literal or ${VAR} lookup-only — assertTrue supports literals and bare lookups, repeat.while is unsupported, and other expression-shaped payloads fail loud.
• Environment: flow env is the default, AD_VAR_* overrides it, and CLI -e KEY=VALUE wins over both.
• Failure diagnostics: resolved targets and runFlow paths are rendered, while inputText payloads remain hidden; do not place secrets in diagnostic identifiers.
• Trust: runScript and evalScript execute flow scripts in-process via node:vm, which is not a security sandbox; runScript may make http.post network requests and its output keys cannot contain a dot. evalScript is refused outright for a flow accepted over the daemon’s remote HTTP surface, since that context can escape to the host.
• Errors and tracking: unsupported commands and fields fail with source context when available; open a focused issue only when implementation work is planned.
• Session takeover: --keep-session is a native .ad replay option and is rejected for Maestro YAML.

See ADR 0015 for architecture, performance tradeoffs, and deliberate deviations. If a missing feature matters for your suite, open a focused issue with a small flow snippet.

Export .ad scripts to Maestro YAML

Replay scripts can be exported to a Maestro YAML subset when you need to hand a recorded Agent Device flow to a Maestro runner:

agent-device replay export ./workflows/checkout.ad --out ./maestro/checkout.yaml

replay export is a local file transform. It does not start the daemon or contact a device. If --out is omitted, the YAML is printed to stdout.

Each open <appId> exports with an explicit launchApp.appId, so a flow can switch between apps and return to the original app. The first app remains the flow's default appId; relaunch options and app-specific deep links stay attached to their authored targets.

Deep links, including schemes without // such as tel: and mailto:, export as openLink. A standalone open tel:+15551234567 emits only the link command; open com.example.app mailto:[email protected] emits the app launch followed by the link.

The exporter is intentionally strict. It writes Maestro YAML for compatible flow actions such as app launch, taps, long press, text input, keyboard dismiss/enter, back, home, text visibility assertions, coordinate swipes, basic scroll, screenshots, and .ad env directives. home exports as pressKey: Home, so flows that visit the home screen and reopen the app can be exported. Agent-only inspection or maintenance actions such as snapshot, get, record, trace, settings, and unsupported selector shapes fail with the source line and action instead of being silently dropped. Known semantic differences are reported as warnings; for example, .ad fill exports as tapOn plus inputText, which may append text in Maestro rather than replacing existing field contents. Native .ad label= selectors export as Maestro text: selectors and warn because Maestro text matching is broader than label-only matching.

Run a lightweight .ad suite

agent-device test ./workflows
agent-device test "./workflows/**/*.ad" --platform android
agent-device test ./workflows --timeout 60000 --retries 1
agent-device test ./workflows --artifacts-dir ./tmp/agent-device-artifacts
agent-device test ./workflows --reporter default --reporter junit:./tmp/junit.xml

• test discovers .ad files from files, directories, or globs and runs them serially.
• Quote relative globs to expand them on the caller from its working directory, including when the directory name contains glob characters such as [ or {. A missing file input without glob characters reports an error.
• context platform=... inside each .ad file is the target source of truth for suite execution.
• --platform is a filter for suite discovery; files without platform metadata are skipped when a filter is present, and the run reports how many sources were skipped for having no context platform= header versus how many declared another platform. Add the header to run a file under a filter, or omit --platform and let the selected device decide.
• context timeout=... and context retries=... can be declared per script; CLI flags override metadata. Retries are capped at 3, and duplicate keys in the context header fail fast instead of silently overriding each other.
• By default, suite artifacts are written under .agent-device/test-artifacts/<run-id>/.... Each attempt writes replay.ad, result.txt, and replay-timing.ndjson. Failed attempts also keep copied logs and artifact files when the replay produced them.
• Copied diagnostic artifacts receive numbered filenames when their names collide with another artifact, a replay source, timing trace, or attempt manifest. result.txt lists the retained names in copiedArtifacts.
• replay-timing.ndjson records attempt, cleanup, and per-step start/stop events with durations. Upload it from CI even for passing runs when comparing local and CI performance.
• Timeouts are cooperative: the runner marks the attempt failed at the timeout boundary, then gives the underlying replay a short grace period to stop before session cleanup.
• The default text reporter streams live progress on stderr while a suite runs, then prints the final summary, failed tests, and passed-on-retry flaky tests. Use --verbose to include step traces in completed-test progress output.
• The default reporter prints a Warnings: section after the summary when any test accumulated composable warnings — for example a Maestro step with optional: true that was skipped — whether that test passed or failed. A failing replay run repeats the warnings it accumulated as Warning: lines after the error. --json carries the same strings in each test result's warnings array.
• --reporter is repeatable. Built-ins are default for the console summary and junit:<path> for JUnit XML. Passing any explicit reporter list replaces the implicit default reporter, so include --reporter default when you also want terminal output. --report-junit <path> remains a compatibility alias for --reporter junit:<path>.
• JUnit reports preserve legal Unicode and whitespace, and replace characters forbidden by XML 1.0 (such as terminal ESC or NUL) with U+FFFD (�) so CI parsers can read the report. JSON and other reporters retain the original suite values.
• When --fail-fast and retries are both set, the current test still consumes its retries before the suite stops.

Custom test reporters

Custom reporters are CLI-only presentation adapters. The daemon streams progress and returns the structured replay suite result; reporters run in the local CLI process and can render both live progress and final output.

agent-device test ./workflows --reporter ./scripts/replay-reporter.mjs

Reporter modules can export a reporter object, reporter, createReporter, or a default factory. Factories receive load context. Reporter hooks receive replay test domain objects and an IO context with stdout and stderr streams:

// scripts/replay-reporter.mjs
import fs from 'node:fs';
import path from 'node:path';

export default function createReporter(loadContext) {
  return {
    name: 'summary-file',
    onTestStep(test, context) {
      context.stderr.write(`running ${test.file} ${test.stepIndex}/${test.stepTotal}\n`);
    },
    onSuiteEnd(suite, context) {
      context.stdout.write(`finished ${suite.total} tests\n`);
      fs.mkdirSync('./tmp', { recursive: true });
      fs.writeFileSync(
        path.join('./tmp', 'report.txt'),
        JSON.stringify(
          {
            total: suite.total,
            passed: suite.passed,
            failed: suite.failed,
            modulePath: loadContext.modulePath,
          },
          null,
          2,
        ),
        'utf8',
      );
    },
    getExitCode(suite) {
      return suite.failed > 0 ? 1 : 0;
    },
  };
}

For a live terminal reporter that prints each completed test as an emoji, title, and duration:

// scripts/emoji-reporter.mjs
export default {
  name: 'emoji-status',
  onTestResult(test, context) {
    const icon = test.status === 'pass' ? '✓' : test.status === 'fail' ? '⨯' : '-';
    const title = test.title?.trim() || test.file;
    const duration =
      typeof test.durationMs === 'number' ? ` ${(test.durationMs / 1000).toFixed(2)}s` : '';

    context.stderr.write(`${icon} ${title}${duration}\n`);
  },
};

TypeScript reporters use the same object shape; compile them to JavaScript before passing them to --reporter:

const createReporter = () => ({
  name: 'typed-reporter',
  onSuiteStart(suite, context) {
    context.stderr.write(`starting ${suite.runnable} tests\n`);
  },
  onTestResult(test, context) {
    context.stderr.write(`${test.status} ${test.title ?? test.file}\n`);
  },
  onSuiteEnd(suite) {
    // Write artifacts, annotations, or summaries from suite.
  },
});

export default createReporter;

The CLI loads reporter modules with Node dynamic import(). Use .mjs or .js files at runtime; for TypeScript, compile the reporter to JavaScript before passing it to --reporter. Loading .ts files directly depends on Node's type-stripping behavior and is not part of the supported reporter contract.

Live reporter hooks are semantic: onSuiteStart, onTestStart, onTestStep, and onTestResult run while the daemon request is active; generic command progress frames are not exposed to test reporters. These live hooks are synchronous — they run from the progress stream as events arrive and are not awaited, so keep their work synchronous and defer anything async to onSuiteEnd, which the CLI awaits before exiting. onSuiteEnd receives the final suite result. getExitCode can only raise the suite exit code, never lower it: the highest reporter-provided code wins and failed tests still exit with 1 when no reporter raises it further, so a reporter cannot mask a failing suite. Return an integer from 0 to 255, or undefined to leave the exit code unchanged. Other values fail with INVALID_ARGS; in particular, codes such as 256 are rejected before they can wrap to a successful process exit.

Parametrise .ad scripts

Substitute ${VAR} tokens in .ad scripts using values from the CLI, shell env, script-local env directives, or built-ins.

context platform=android
env APP_ID=settings
env WAIT_SHORT=500

open ${APP_ID} --relaunch
wait ${WAIT_SHORT}
click "label=${APP_ID}"

Precedence

Source	Priority	Example
CLI -e KEY=VALUE	highest	agent-device test flow.ad -e APP_ID=demo
Shell env prefixed AD_VAR_		AD_VAR_APP_ID=demo agent-device test flow.ad (imported as APP_ID)
Script env KEY=VALUE		env APP_ID=settings in header
Built-ins	runtime	AD_PLATFORM, AD_SESSION, AD_FILENAME, AD_DEVICE, AD_ARTIFACTS

Built-ins

Built-ins are provided by replay/test runtime and use the reserved AD_* namespace.

• AD_PLATFORM - matches context platform=... or the selected platform when available
• AD_SESSION - active session name
• AD_FILENAME - path of the running .ad file
• AD_DEVICE - device identifier (when --device is set)
• AD_ARTIFACTS - attempt artifacts directory (when running under test)

User-defined keys starting with AD_ are rejected in env, -e, and shell imports such as AD_VAR_AD_FOO, so built-ins cannot be overridden.

Substitution happens inside parsed string values. It does not create extra arguments, so quote selectors or text values that contain spaces:

env SETTINGS="label=Account || label=Profile"
click "${SETTINGS}"

Fallback and escape

wait ${WAIT_MS:-500}

${VAR:-default} yields default when VAR is unset.

echo "Price: \${APP}"

\${APP} emits a literal ${APP} with no substitution.

Recipes

Run one flow against two app variants in CI:

agent-device test ./flows/login.ad -e APP_ID=com.example.debug
agent-device test ./flows/login.ad -e APP_ID=com.example.release

Tune timings locally without editing the script:

AD_VAR_WAIT_SHORT=2000 agent-device replay ./flow.ad

Extract a reusable selector. Before:

click "label=Account || label=Profile || label=User"
wait 500
click "label=Account || label=Profile || label=User"

After:

env SETTINGS="label=Account || label=Profile || label=User"

click "${SETTINGS}"
wait 500
click "${SETTINGS}"

Quote ${VAR} inside selector expressions so the whole expression is treated as a single argument.

Notes

• Shell env (AD_VAR_*) is collected on the CLI/client side at request time, so the same values are seen whether the daemon runs locally or remotely.
• No nested fallback. ${A:-${B}} is not supported.
• Unresolved ${VAR} fails with a file:line reference. Typos are loud.

Replay divergence and resume

A failing replay/test step returns a structured REPLAY_DIVERGENCE error instead of a bare failure. The report carries, bounded and redacted:

• step — the 1-based plan index and its source file/line (through Maestro runFlow includes).
• screen — a fresh post-failure snapshot digest with actionable refs, or unavailable with a reason/hint when capture failed or was sparse (never a stale tree).
• suggestions — up to 5 ranked, re-resolved candidates for the failing selector (id match ranks above role+label, which ranks above label-only), each with a basis you can inspect before acting.
• resume — whether and how to continue without re-running the script from the top.

{
  "code": "REPLAY_DIVERGENCE",
  "details": {
    "divergence": {
      "step": { "index": 4, "source": { "path": "flow.ad", "line": 6 } },
      "screen": {
        "state": "available",
        "refsGeneration": 3,
        "refs": [
          /* ... */
        ],
      },
      "suggestions": [{ "selector": "id=\"auth_continue\"", "basis": "id" }],
      "resume": { "allowed": true, "from": 4, "planDigest": "…64 hex chars…" },
    },
  },
}

Text output prints a compact summary of the same fields; --json/MCP carry the full object.

Resuming a failed replay

replay --from <n> --plan-digest <sha256> resumes at plan step n, not after it, skipping 1..n-1 without executing them. Both flags come from a divergence report's resume field — from is the failed step, planDigest is the digest of the exact unchanged plan that produced it.

Choose one recovery workflow:

1. Change the replay script. Review the suggestion, edit the selector or include, then run a fresh full replay ./flow.ad. The old digest is intentionally invalid after any plan edit; do not combine it with the edited script. A later divergence supplies a new digest.
2. Keep the replay plan unchanged. Repair app/device state so the reported failed step can succeed when retried, then resume with the report's unchanged from and planDigest. If you manually complete the failed action itself, the reported from will execute it again; only do that when repeating the action is safe.

The unchanged-plan resume loop is:

1. Run replay ./flow.ad. On failure, read resume from the divergence.
2. Leave the script, includes, platform, and target unchanged. Repair app state yourself so the failed step can be retried safely. The daemon never infers or reconstructs app state — it only skips execution of the earlier steps.
3. replay ./flow.ad --from <resume.from> --plan-digest <resume.planDigest>.

agent-device replay ./flow.ad
# ... REPLAY_DIVERGENCE, resume: { allowed: true, from: 4, planDigest: "ab12...“ }
# (repair app state on the device)
agent-device replay ./flow.ad --from 4 --plan-digest ab12...

For Maestro flows, --from addresses the immutable top-level typed plan. Compact runtime control nodes remain single plan steps and nested commands are not independently addressable. As with generic .ad replay, the caller is responsible for restoring any state and environment values established by skipped steps before resuming.

Passing --plan-digest that no longer matches the current script — because you edited it, an include changed, or platform-conditioned expansion differs — fails INVALID_ARGS before any action; run a fresh full replay to get a new digest. --from is replay-only; test rejects it (a suite run must stay full and deterministic).

--update/-u (retired)

--update/-u no longer rewrites .ad files. Historically it retried a failing step against the recorded selector's candidate material and rewrote the line in place; the audit behind ADR 0012 found that mechanism rarely able to act (it can only recover drift the original selector still matches, never a rename) and a silent rewrite is a target-binding risk on its own. The flag is kept, accepted, and is a complete no-op: every replay divergence already carries the same ranked suggestions the old heal path used to apply blind, whether or not --update is passed. Review a suggestion, then edit the .ad file yourself if it's right.

Troubleshooting

• Replay fails after UI/layout changes:
  • Read the divergence report's suggestions and repair the selector by hand; there is no automated rewrite. Because the edit changes the plan digest, run a fresh full replay instead of using the old resume flags.
• Repeated re-runs are slow or the app is stateful, but the script is still correct:
  • Leave the replay plan unchanged, repair app state so the reported failed step can be retried, then use its --from/--plan-digest. Resume starts at --from; it does not skip that step.
• Replay file parse error:
  • Validate quoting in .ad lines (unclosed double quotes are rejected). A selector with a space needs quotes, and help scripting states how each quote form decodes.
• A replay passes on one device size and fails on another because a target was off screen:
  • The script used a fixed scroll amount. Replace it with the stop condition, scroll down --until <selector>, which repeats until the element is actually on screen.
• A press or click step fails because its target was not found, but the element is on the screenshot:
  • A selector-miss divergence, or error.details.readiness.end: expired, means the element was not in the accessibility tree for the whole 2-second wait: the selector is wrong for this build, or the element is not exposed to accessibility. readiness.end: sparse means the app showed an empty tree: the screen was mid-transition or the app had left. Add a wait step for a landmark on the new screen before the press.
• Maestro compatibility flow fails on unsupported syntax:
  • Check ADR 0015. If the missing feature matters to your suite, open a focused issue with a small flow snippet.