skip to content
e2e raised $8m to build the software verification layer for the agentic coding era
13.8kstar on github

docs / Deep dive

[ view as markdown ]

Telemetry

What the CLI sends, what it never sends, and how to turn it off.

The e2e CLI sends anonymous usage data: which commands, engines, and features people use, and where runs fail. Telemetry is on by default. It sends counts and versions, never test content, app content, or credentials.

Opt out

Any one of these turns telemetry off:

npx e2e telemetry disable   # saved in ~/.config/e2e/telemetry.json
export E2E_TELEMETRY_DISABLED=1          # for a shell or a CI job
export DO_NOT_TRACK=1                    # the cross-tool convention

The environment variables win over the saved setting. e2e telemetry enable reverses a saved disable. e2e telemetry prints the status and, when off, the reason. The preferences file is under $XDG_CONFIG_HOME/e2e/ when that is set, and %APPDATA%\e2e\ on Windows.

A disable also stops commands already running in another terminal. Nothing is sent when the preferences directory isn’t writable, or from a source checkout of the e2e repository.

See what would be sent

E2E_TELEMETRY_DEBUG=1 npx e2e run

The CLI prints the exact payloads to stderr as [telemetry] {...} and sends nothing.

What is sent

Each command sends one e2e_cli_session event. run and explore also send e2e_run_completed, init sends e2e_init_completed (and so does explore when it runs the init wizard first), upgrade sends e2e_upgrade_completed, and mcp sends one e2e_mcp_session per MCP session. --help and --version send nothing, on any command. run() and explore() from e2e/runner send no session event, only one e2e_run_completed per call that returns a report.

The events go in one request when the command, run(), or explore() ends, with a two-second deadline. e2e mcp sends each session event in its own request, and e2e run --watch sends each run’s event when that run ends.

Events hold only the properties below. An error code that isn’t an upper-case token is sent as OTHER. An engine, platform, model, fleet, or MCP client name, or an MCP client version, that isn’t a plain token of letters, digits, dots, dashes, and underscores is sent as other. An engine version that isn’t semver is sent as unversioned. In action and tool call counts, a project’s own tools count together as tool, and any other name as other.

Every event
PropertyValue
distinct_idA random id stored in the preferences file. CI uses ci:<vendor> instead, and a fleet uses fleet:<name> (see In a fleet)
project_idA hash of the repository’s root commit, shared across full clones. null outside git and in a shallow clone
session_idA random id for this one invocation
first_run, days_since_first_runWhether this command created the preferences file, and whole days since then. null in CI, in a fleet, and for a preferences file from before the time was recorded
os, os_release, arch, cpu_count, memory_gb, docker, wsl, ttyThe machine class
sandboxThe microVM the kernel announces itself as (firecracker), else null
ci, ci_nameWhether CI is set, and the vendor when one announces itself (github-actions, gitlab, circleci, ...)
fleetThe value of E2E_TELEMETRY_FLEET, null when unset
coding_agentThe coding agent whose shell ran the command, when one announces itself (claude-code, cursor, codex, ...)
e2e_version, package_managerThe e2e version, and the package manager that started the CLI
runtime, runtime_version, node_versionnode, bun, or deno, and its version. node_version is Node’s own version, or the Node compatibility version under Bun or Deno
$lib, $lib_version, $process_person_profile, $geoip_disablee2e, its version, false, and true: PostHog keeps no person profile and derives no location
e2e_cli_session, once per command
PropertyValue
commandinit, run, list, cache ls, guide, telemetry, ...
flagsThe names of the flags given, such as --headed or --tag. Never their values
exit_code, duration_msHow the command ended and how long it took
error_codeThe runner code of the failure that stopped the command before it could run (CONFIG_NOT_FOUND, INVALID_CONFIG, ...), CLI_USAGE for a flag or argument the CLI rejected, else null
e2e_init_completed, once per init wizard
PropertyValue
resultscaffolded, already-initialized, cancelled, install-failed, not-interactive, invalid-project, or example-failed
yes, existing_configWhether --yes was given, and whether a config already existed, so no engine or gateway was asked
exampleThe --example name (with-vite, with-expo, ...), null without the flag
engine, gatewayThe ids of the engines and the model gateway chosen for a new config (web, mobile, web,mobile, none; vercel, openrouter, openai-compatible, chatgpt, copilot, opencode-console, grok, none), null when not asked. Never the endpoint typed for an OpenAI-compatible gateway
skill, mcp, installWhether the agent skill, the MCP registration, and the dependency install were chosen
e2e_upgrade_completed, once per e2e upgrade
PropertyValue
resultupgraded, up-to-date, dry-run, no-project, invalid-project, linked, unknown-target, incompatible, registry-failed, or install-failed
targetversion for a named version, else the dist-tag: latest, nightly, canary, beta, or other
from_version, to_version, prereleaseThe e2e version installed before, the one the target resolved to, and whether that one is a prerelease. null when unknown
managerThe package manager the command picked for the project: npm, pnpm, yarn, or bun. It installs only on upgraded. null when the command stopped before picking one
siblingsThe @e2e-dev packages in the upgrade plan, without the scope: ["web", "kernel"]. Linked packages and ones the registry doesn’t have are left out
packages_moved, packages_linkedHow many e2e and @e2e-dev entries in package.json the upgrade rewrote, and how many it left alone because they aren’t registry versions (workspace:, file:, catalog:, a path, git, an alias). Not the lockfile’s package count
e2e_run_completed, once per e2e run, e2e explore, or completed run() or explore()
PropertyValue
commandrun, explore, runner.run for run() from e2e/runner, or runner.explore for explore()
status, exit_code, duration_msThe run’s outcome
flagsThe names of the flags given to the run command, never their values
tests_discovered, tests_selected, tests_executed, tests_passed, tests_failed, tests_interrupted, tests_flaky, tests_skippedThe summary counts
tests_failed_as_expectedTests whose failure test.fail covered, counted as passed
attempts_total, tests_retriedHow many attempts the run made across tests and serial groups, and how many of these needed more than one
filters_run, filters_timed_out, filters_failed, filters_skipped, filters_changed_files, tests_left_out_by_filtersHow many test filters the run applied, how many ran out of time, and how many failed, whether a run with --changed-since called none of them, such as when no file changed, how many files the change had, and how many tests the filters left out. Never a filter’s name, reason, or note, a path, or a commit
targets, platforms, enginesHow many targets, on which platforms, and with which engines and versions: web, ios, [email protected], or your own engine’s name and version
steps_total, steps_agent, steps_locator, steps_assertion, steps_screen, steps_app, steps_session, steps_resourceStep counts by kind
steps_agent_act, steps_agent_assert, steps_agent_extract, steps_agent_waitforThe agent steps by entry point
tests_with_agent_steps, tests_with_actHow many executed tests ran any agent step, and how many ran agent.act
agent_steps_replayed, agent_steps_partial, agent_steps_missed, agent_steps_uncached, agent_steps_visionHow the replay cache served the agent steps, how many agent.act steps ran with the cache off, and how many used pixel evidence
agent_miss_reasons, agent_handoff_reasonsWhy steps missed the cache and why replays handed off to the agent, by the runner’s own reason names: {"no-entry": 3, "retry": 1}
tests_code_only, tests_replayed, tests_assisted, tests_liveThe executed tests by how the cache carried them: no agent step, every agent step replayed from the cache, the agent ran but the cache carried part of a step, the agent ran every agent step
agent_actionsHow often the agent took each action, by the runner’s own names: {"tap": 12, "typeText": 3, "scroll": 2}
model_gateway, model_provider, model_id, model_calls, model_tokens, model_cached_tokens, estimated_cost_usdFor the first model-backed step: the AI SDK provider that served it (gateway for the Vercel AI Gateway, openrouter.chat, openai.chat, ...), the vendor (the first segment of a vendor/model id, else the provider), and the public model id. Then the run’s totals
model_calls_act, model_calls_assert, model_calls_extract, model_calls_waitforThe model calls by agent entry point
cost_sourceprovider when the provider priced the calls, none when nobody priced them, null for a run without model calls
artifact_bytesThe size of the artifacts the run wrote
errors, error_codesHow many run-level errors, and the distinct error codes seen anywhere in the run (LOCATOR_NOT_FOUND, APP_UNREACHABLE, ...), at most 20
primary_error_codeThe first error the run recorded, which decided its status: a run-level error, else the failed attempt’s verdict, else a step’s. null for a clean run
error_kindsFor ENGINE_FAILURE, MODEL_PROVIDER_FAILED, and ERROR: the code and a kind read off the message against a closed list (auth, rate-limit, overloaded, timeout, network, server, bad-request, device, daemon, browser, permission, not-found, memory, other), as MODEL_PROVIDER_FAILED:rate-limit. Only the kind’s name leaves the machine, never the message

Config features, absent when the config didn’t load. Each is a count, a boolean, or one of the runner’s own option values, never text you wrote:

PropertyValue
config_workers, config_retriesThe worker count and retries the run resolved
config_agents, config_custom_executor, config_separate_judge, config_project_toolsHow many agents are configured, whether any has a custom executor or a judge model of its own, and how many distinct project tools they declare
config_credentials, config_secretsHow many credentials and secrets are configured
config_cache_mode, config_cache_store, config_cache_strictoff, read-only, or read-write; file or custom; whether strict replay is on
config_reporters, config_custom_reporters, config_artifact_storeThe built-in reporter ids (list, json, junit, markdown), how many reporter objects the config adds, and whether an artifact store is set
config_filtersHow many test filters the config declares
config_trace_modes, config_video_modesThe distinct recording modes across targets (off, on, retain-on-failure, on-first-retry, on-all-retries)
config_app_commands, config_environmentsHow many targets start an app command, and the distinct app environments (test, staging, production)

On e2e explore and explore() only, counts of what the exploration was given and found:

PropertyValue
explore_endedWhy it stopped: finished, skipped, step-limit, time, stuck, or aborted
explore_max_steps, explore_timeout_msThe budgets it ran with
explore_steps, explore_steps_passed, explore_steps_failed, explore_steps_blocked, explore_steps_exhaustedExploration steps, by how each ended
explore_assessedWhether the agent gave a closing assessment
explore_issues, explore_warnings, explore_findings_by_severityFindings by kind, and by severity: {"1": 0, "2": 1, "3": 0, "4": 0, "5": 1}
e2e_mcp_session, once per MCP session

An e2e mcp server sends one for each open_session it receives, when the session closes or the open fails. A session that is still open when the client kills the server isn’t sent.

PropertyValue
client, client_versionThe name and version the MCP client gives itself (claude-code, cursor-vscode, ...). null before the client introduced itself
outcome, error_codeclosed for a session that opened, open-failed with the runner code of the failure (APP_UNREACHABLE, ENGINE_FAILURE, ...)
ended_byagent for close_session, idle, step for the session’s time limit, server when the server shut down. null for a failed open
platform, engine, headedThe target’s platform and engine with its version, as for a run, and whether the UI was shown
duration_ms, sessions_openHow long the session lived, and how many other sessions were open when it opened
calls_total, calls_failed, callsTool calls, failed calls, and calls by the session’s own tool names: {"observe": 4, "tap": 9, "locate": 2, "screenshot": 1}
error_codesHow often each runner code failed a call: {"LOCATOR_NOT_FOUND": 2}, at most 20 codes

What is never sent

Telemetry never sends test titles, file paths, URLs, origins, instructions, observations, page content, screenshots, error messages, stack traces, environment variable values, credentials, or file contents. It never sends the names of targets, agents, credentials, secrets, or project tools, MCP tool arguments or results, an exploration’s goal or charters, or its findings’ words.

In CI

CI runs count as the CI vendor, not as a machine, with no preferences file and no notice. A coding agent that sets CI in its shell, with no CI vendor around it, counts as the machine it runs on.

In CI, only the environment variables turn telemetry off. Set E2E_TELEMETRY_DISABLED: '1' in the job’s env.

In a fleet

A platform that runs e2e for its users on new machines would count each machine as a new user. Set E2E_TELEMETRY_FLEET=<name> on those machines. Their events count as fleet:<name>, with no preferences file and no notice, as in CI. Or set E2E_TELEMETRY_DISABLED=1 to send nothing.

Where it goes

Events go to eu.i.posthog.com, a PostHog project that the e2e maintainers operate. PostHog keeps no person profile, stores no request address, and skips GeoIP for each event. The data is read in aggregate.

Telemetry falls under the security policy. Outbound connections lists every other connection a run can make.

Feedback

e2e feedback isn’t telemetry. It sends the words in its flags as one e2e_feedback event to the same PostHog project, with the machine facts above. When telemetry is on, it has the same anonymous id as your telemetry events.

A saved e2e telemetry disable doesn’t stop it. E2E_TELEMETRY_DISABLED=1 and DO_NOT_TRACK=1 do. The CLI reference covers redaction, --dry-run, and bugs without --repro.

The notice

The first command, or the first run() or explore() from e2e/runner, prints a notice on stderr that tells how to opt out. It leaves --reporter json output intact. e2e init and e2e telemetry don’t print it, so it waits for the next command. The notice shows again after a large change to the data collected.

was this page helpful?