# Sandbox bench

> Sandbox bench: cloud sandbox providers for AI agents, timed through one SDK (Sandbox SDK, built by us; 0.2.0 is public on npm and the 1.0 used here will be published soon) from one server in Florida, US, on two tracks: the same 4 vCPU, 8 GB machine for every provider (the fair comparison, and the one the site opens on), and each provider's default machine. Medians, p95s and 95% confidence intervals per provider; every provider-side failure kept, failures caused by our own code set aside.

Providers (id → name): e2b → E2B, freestyle → Freestyle, tenki → Tenki, boxd → boxd, superserve → Superserve, upstash → Upstash Box, daytona → Daytona, upstash-ephemeral → Upstash Ephemeral Box, boat → Boat, vercel → Vercel Sandbox, modal → Modal.

## Tracks

- `same-spec`: Same machine: 4 vCPU, 8 GB. Every provider asked for Boat's machine, 4 vCPU and 8 GB, or its nearest size.
- `default`: Default machines. Each provider's default sandbox, so machine sizes differ.

The page shows one track at a time: `/` opens the same-machine track and `/?track=default` the default-machines track.

## What a task costs (realistic workloads)

Pilot: 5 runs per provider; most costs are provider meters or estimates not yet checked against settled bills. The full 10-batch run and recorded agent sessions come later.

Real work instead of the quick task, each in a fresh sandbox from the create request to confirmed deletion, setup included. Fix and test applies a real Vue bug, runs the type checker and Vue's test suite, applies the fix and runs them again. Build is a production build of the Jitsi Meet web app. Each runs on the same-machine track (4 vCPU, 8 GB) and on each provider's default machine. An idle test leaves the sandbox open after fix and test. Full numbers: `whatATaskCosts` in /results.json and the "What a task costs" section of /results.md.

- Fix and test, Same machine: Lowest verified cost among eligible providers: Upstash Box $12.06 per 1,000 (compute verified (billed, settled); disk estimated). Eligible: Upstash Box, Modal. No certified winner: with 5 pilot runs there is no statistical test, so the result is descriptive. Lower but not eligible: Boat $0.95 (metered, unsettled, unreconciled); boxd $3.80 (metered, unreconciled, wrong size); Freestyle $9.58 (estimate); Tenki $9.79 (estimate); E2B $9.99 (metered, unreconciled); Daytona $10.49 (billed, unsettled, provisional); Upstash Ephemeral Box $12.05 (billed, provisional).
- Build, Same machine: Lowest verified cost among eligible providers: Upstash Box $16.07 per 1,000 (compute verified (billed, settled); disk estimated). Eligible: Upstash Box, Modal. No certified winner: with 5 pilot runs there is no statistical test, so the result is descriptive. Lower but not eligible: Boat $2.10 (metered, unsettled, unreconciled); boxd $7.57 (metered, unreconciled, wrong size); Daytona $11.24 (billed, unsettled, provisional).
- Fix and test, Default machines: Lowest verified cost among eligible providers: Upstash Box $5.94 per 1,000 (compute verified (billed, settled); disk estimated). Eligible: Upstash Box, Modal. No certified winner: with 5 pilot runs there is no statistical test, so the result is descriptive. Lower but not eligible: Boat $0.88 (metered, unsettled, unreconciled); boxd $3.99 (metered, unreconciled).
- Build, Default machines: Lowest verified cost among eligible providers: Upstash Box $16.68 per 1,000 (compute verified (billed, settled); disk estimated). Eligible: Upstash Box, Modal. No certified winner: with 5 pilot runs there is no statistical test, so the result is descriptive. Lower but not eligible: Boat $2.19 (metered, unsettled, unreconciled); boxd $8.27 (metered, unreconciled); Upstash Ephemeral Box $11.23 (billed, provisional).

Cost scope: every charge from the create request through confirmed deletion, setup and failed attempts included: compute, memory, disk and standby storage. Upstash Box's billing leaves disk out, so disk is added at list price, an estimated part of its cost, marked "disk estimated"; only the billed part counts as verified. For Upstash Ephemeral Box and Daytona it isn't known whether the billing source includes disk, so nothing is added and the cost is marked provisional. Freestyle and Tenki are estimates: the machine's list price, disk included, over the billed time. Retained snapshots after deletion and data transfer are outside the task cost; this pilot doesn't measure them, so they aren't reported yet.

Usage only: Boat's $20 monthly plan minimum isn't in these costs (from its pricing notes, https://docs.boat.dev/machines). Upstash Box's costs include an estimated part: $0.036 per 1,000 of it (0.30%) is disk added at list price, an estimate; the rest is billed compute.

Idle test (same machine, fix and test, then 10 or 60 minutes open, 3 runs per provider and wait): Upstash Ephemeral Box: the resume step failed after the hour in all 3 runs, with no stop reported; Boat: stopped the sandbox during the hour, so the work couldn't resume (0 of 3 resumed); Daytona: stopped the sandbox during the hour, so the work couldn't resume (0 of 3 resumed). Not tested: Freestyle (it has no billing data we can read, so an idle run's cost couldn't be tied to the sandbox); Superserve (its billing reports account totals over a time window, not per sandbox, so an idle run's cost couldn't be tied to the sandbox); Tenki (it has no billing data we can read, so an idle run's cost couldn't be tied to the sandbox).

## Quick picks

### Same machine: 4 vCPU, 8 GB

- Fastest cold start: boxd 0.30 s (runner-up Modal 0.38 s)
- Fastest agent work (tool replay): Modal 12 s (runner-up Daytona 12 s)
- Fastest 25-sandbox fan-out: Modal 0.62 s (runner-up Superserve 0.79 s)
- Fastest reconnect: Modal 0.19 s (runner-up boxd 0.30 s)
- Quick task: lowest verified cost among eligible providers: Upstash Ephemeral Box $0.00026 per task, $0.26 per 1,000 (billed, verified) (runner-up Upstash Box $0.00034 per task, $0.34 per 1,000 (billed, verified)) Eligible: Upstash Ephemeral Box, Upstash Box, Superserve, Modal. Lower but not eligible: boxd $0.00013 per task, $0.13 per 1,000 (metered, provisional), because its metered figure isn't reconciled with settled charges, and its per-hour charge for the same task fell about 6x between Oct 22 (Frankfurt) and Nov 4 (New York), unexplained.
- Most reliable: E2B, Tenki, Superserve, Upstash Box, Daytona, Upstash Ephemeral Box, Boat, Modal 100.00% (tied) (runner-up Freestyle 99.91%)

### Default machines

- Fastest cold start: boxd 0.28 s (runner-up Modal 0.38 s)
- Fastest agent work (tool replay): Modal 11 s (runner-up boxd 13 s)
- Fastest 25-sandbox fan-out: Modal 0.56 s (runner-up Superserve 0.75 s)
- Fastest reconnect: Modal 0.18 s (runner-up boxd 0.30 s)
- Quick task: lowest verified cost among eligible providers: Upstash Ephemeral Box $0.00013 per task, $0.13 per 1,000 (billed, verified) (runner-up Upstash Box $0.00017 per task, $0.17 per 1,000 (billed, verified)) Eligible: Upstash Ephemeral Box, Upstash Box, Modal, Superserve. Lower but not eligible: boxd $0.000097 per task, $0.097 per 1,000 (metered, provisional), because its metered figure isn't reconciled with settled charges, and its per-hour charge for the same task fell about 6x between Oct 22 (Frankfurt) and Nov 4 (New York), unexplained.
- Most reliable: E2B, Tenki, Superserve, Upstash Box, Daytona, Boat, Vercel Sandbox, Modal 100.00% (tied) (runner-up Freestyle 99.81%)

## Coming in the next update

Any snapshot or burst a chart marks as coming. Every result, cold start, SDK setup, snapshots, bursts and cost included, comes from the runs on our faster SDK (commit 1185278); Superserve now runs the same-machine track at 4 vCPU. Pause and resume are from the previous run (SDK a2316c3), because the new main runs skipped them. boxd is the exception: every boxd result, pause and resume included, comes from a full re-run on its New York region on Nov 4 (commit d8844f0). Daytona was re-collected on Oct 22 after its account quota was raised from 10 vCPU / 10 GiB; earlier Daytona numbers were limited by that quota. Upstash appears twice because Box (long-lived, can pause) and Ephemeral Box (short-lived, for one-off agent jobs) are separate products. A provider gets a second row only for a product it sells separately, not for a setting on the same product. Ephemeral Box has no pause, so it has no pause result. There is no overall cheapest pick: the cost pick is the lowest verified cost among eligible providers, and only a cost that is billed and settled, and not provisional, is eligible. Metered costs stay unreconciled until checked against the provider's settled charges, which none has been yet. Shown but not eligible yet: boxd's metered cost is provisional: its metered figure isn't reconciled with settled charges, and its per-hour charge for the same task fell about 6x between Oct 22 (Frankfurt) and Nov 4 (New York), unexplained; Daytona's billed cost is unsettled: Daytona has not settled this charge yet, so it can still change; E2B's metered cost is unreconciled: E2B's usage records at its published rates haven't been checked against the charges it settled yet; Vercel Sandbox's metered cost is unreconciled: Vercel Sandbox's usage records at its published rates haven't been checked against the charges it settled yet. Daytona settles up to 48 hours after a run. Freestyle and Tenki costs are list-price estimates on our measured sandbox lifetime, because neither has a billing API. Boat's is an estimate from its docs: its documented price on our measured sandbox lifetime, because its per-sandbox usage wasn't pulled for that run. Boat can't be a pick yet because no settled-charge source for Boat is available. Estimates are shown, never picked. Vercel's snapshot() also stops the sandbox it snapshots. Modal is priced at Modal's us-east rate, 1.75× its base price. Destroy on Modal times Modal's terminate request, the same call the other providers' delete times. Modal reports the sandbox stopped later, sometimes about 30 s after (2 of 18 sandboxes in a probe), and may bill until then. Modal's snapshots keep the file system only, not running processes or memory. Modal's hourly price on its default machine is a lower bound for the same reason. Billing models: E2B, Boat: bills the whole machine while it exists; Freestyle, Tenki, Superserve, Daytona: billing basis not confirmed; boxd: bills active CPU and memory in use; Upstash Box, Upstash Ephemeral Box: bills active CPU time, with no memory charge listed; Vercel Sandbox: bills active CPU time and the memory it allocates; Modal: bills per second on the larger of its request and what it uses, counting a CPU as a physical core. Whole machine at list price: Shown per hour for the allocation: a per-task figure needs each task's sandbox lifetime, which the harness doesn't export yet. A burst size is published only when all three bursts finished. Failures caused by our own code are set aside, not counted. 23 Modal main-suite runs (11 same machine, 12 default) were set aside as our-side failures and rerun: Modal's 200-deployed-App limit (one App per iteration) refused the App lookup, so no create reached Modal.

## Data

- [results.json](/results.json): both tracks. `whatATaskCosts` holds the realistic workloads (pilot): per track and workload the pick, its eligible set, lower costs that aren't eligible, and every provider's cost, kind, status and completion count; plus the idle test. Per provider: what we asked for and measured (allocation, equalSpec), cost per task with its kind (billed, metered, estimate), source and confidence, SDK-ready time, prices, regions and failures. Per metric: median, p95, attempts, failed and status. Start here.
- [results.md](/results.md): the same numbers as Markdown tables, one section per track.

## How to read it

Pick a track first: `tracks["same-spec"]` asks every provider for 4 vCPU and 8 GB, so prefer it for comparisons; `tracks.default` compares each provider's default sandbox, so sizes differ. In same-spec, `allocation.equalSpec: false` means the machine measured inside was not 4 vCPU, 8 GB, so compare that provider with care. median and p95 are in `unit`; `better` says which direction wins; results are sorted best first. attempts counts every try, failed included. A null median means the provider does not support the test, every attempt failed, or the result is excluded (see `status`). Cold start is the provider's own time, create() until the first command; Our SDK's setup after create is not included; on the faster SDK it measured 0.1 to 0.3 ms median per provider (20 creates each, separate run). `costPerTask.kind` is billed (provider charge), metered (provider usage at published rates) or estimate (list price, where billing gives no amount); the quick task is one sandbox that is created, runs the 100-call tool replay and is deleted, from a separate cost run (real workloads are in `whatATaskCosts`): never compare them as if they were the same kind without saying so. `costPerTask.status` comes from one eligibility table (`costEligibility`): verified (billed and settled, and not provisional) can be the cost pick; unreconciled (metered, not yet checked against settled charges), provisional, unsettled and estimate are shown but never picked. There is no overall cheapest pick: the cost pick is the lowest verified cost among eligible providers (`costPick`), with the eligible set and every lower cost that isn't eligible, with its reason. `billing` says what each provider bills on, or that its billing basis is not confirmed. `wholeMachineAtListPrice` is the hourly list price of the allocation, a counterfactual for providers that bill usage; it has no per-task figure until the harness exports each task's sandbox lifetime. `bound: "lower"` on a cost or price marks a lower bound (status `lowerBound`): it counts only the resources requested and the provider also bills use above them, so the charge can be higher. Read it as at least that amount; it is never eligible and never compared with exact costs. `price.regionMultiplier` is the pinned region's price multiplier, already included in every price and cost for that provider.

Prices cover compute only at list rates; disk, transfer, subscriptions and credits are excluded. Every call starts in Florida, so a provider hosted in Europe (Boat, in France) pays a transatlantic round trip per call. In the same-machine track, usable memory measured inside sits a little under 8 GB; equalSpec allows 7.6 to 8.4 GiB. Amounts are US dollars unless marked EUR.

## Metrics

- Start-up: Cold start (`cold`)
- Agent work: Agent tool replay (`tool-replay`), npm install (`npm-install`), pip install (`pip-install`), git clone (`git-clone`), Dev server ready (`dev-server`), Preview URL first byte (`preview-first-byte`), Edit, rebuild, reload (`edit-loop`)
- Everyday calls: Run a command (`command`), Write + read 1 KB (`files-1kb`), Write + read 10 MB (`files-10mb`), Reconnect (`reconnect`), Destroy (`destroy`)
- Pause and snapshots: Pause + resume (`pause-resume`), Take snapshot (`snapshot`), Start from snapshot (`create-from-snapshot`)
- Bursts: 10 at once, all ready (`burst-10`), 25 at once, all ready (`burst-25`)
- Machine speed: CPU, one core (`cpu-single`), CPU, all cores (`cpu-all`), Memory bandwidth (`memory`), Disk write (`disk-write`), Disk read (`disk-read`), 4K random write (`iops-write`), 4K random read (`iops-read`)

## Provider pages

- [Sandbox providers we benchmark](/providers/): every provider in one table, with cold start on both tracks and cost per task, and the providers not tested yet.
- [E2B sandbox benchmark](/providers/e2b/): E2B's results on both tracks, its ranks, cost per task and caveats.
- [Daytona sandbox benchmark](/providers/daytona/): Daytona's results on both tracks, its ranks, cost per task and caveats.
- [Superserve sandbox benchmark](/providers/superserve/): Superserve's results on both tracks, its ranks, cost per task and caveats.
- [Upstash Box sandbox benchmark](/providers/upstash/): Upstash Box's results on both tracks, its ranks, cost per task and caveats.
- [Upstash Ephemeral Box sandbox benchmark](/providers/upstash-ephemeral/): Upstash Ephemeral Box's results on both tracks, its ranks, cost per task and caveats.
- [Boat sandbox benchmark](/providers/boat/): Boat's results on both tracks, its ranks, cost per task and caveats.
- [Freestyle sandbox benchmark](/providers/freestyle/): Freestyle's results on both tracks, its ranks, cost per task and caveats.
- [Tenki sandbox benchmark](/providers/tenki/): Tenki's results on both tracks, its ranks, cost per task and caveats.

## Comparison pages

- [E2B vs Daytona](/compare/e2b-vs-daytona/): both on both tracks, test by test, with which one is ahead and where the gap is a tie inside the 95% intervals.
- [Upstash Box vs E2B](/compare/upstash-box-vs-e2b/): both on both tracks, test by test, with which one is ahead and where the gap is a tie inside the 95% intervals.
- [Upstash Box vs Daytona](/compare/upstash-box-vs-daytona/): both on both tracks, test by test, with which one is ahead and where the gap is a tie inside the 95% intervals.

## Alternatives pages

- [E2B alternatives](/alternatives/e2b/): every other provider we tested, set against E2B on both tracks: tests ahead, behind and tied, cold start, tool replay, cost per task and pause support.