We simulated an e-commerce platform, Tidewick Commerce, with AI agents role-playing
its people: the CEO, data analysts, engineers and more. This report shows how fast databases answer
the queries they asked.
CategoryCloud
Scale factor
Machines
Figure 1.Up and to the left is better.
Queries per second against what it costs to run
all 409 queries.
Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.Each system on its most efficient configuration; in grey, its other configurations on the Pareto front (dashed).Each system on every configuration, joined by a line.
Pricing based on AWS.
Timeouts and errors are penalized.
1Performance
System
QPS
Cost
Slowdown, relative to the fastest
Correct
Load
Table 1. Systems ordered by cost, the cheapest first. Slowdown is the geometric mean, over all
409 queries, of each query's time relative to the fastest correct system (plus 10 ms), the bar on a log scale. A timed-out query counts as taking the full timeout
(5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000); a failed or wrong one as taking twice the slowest correct time of that query.
QPS (queries per second) and Cost as in Figure 1. Correct is out of 409 after the last load. Load is the
total of all five loads.
There are no reference results at this scale factor yet: every result that came back counts
as correct, without being checked.
2Queries
Order queries by
Figure 2. One row per system, one column per query, colored by the time relative to the fastest
correct system. Select a column to see the query.
3Methodology
The data
Tidewick Commerce in 251 tables, at scale factors 1, 10, 100 and 1000: from 6.1 million to 4.1 billion
rows.
The run
A fresh system loads everything through 2024, then 2025 one quarter at a time. After every load it answers
all 409 queries once, and it is never restarted. The figures show the run after the last load.
Correctness
After the last load, every result is compared by value with a reference result. Integers must match exactly;
other numbers may differ by at most 0.0001 + 0.0001 × |expected|. A wrong result counts as a
failed query.
Failures
A query still running after its timeout (5 s at SF1 and SF10, 10 s at SF100, 30 s at SF1000) is aborted. In every measure of speed, a timed-out query
counts as taking the full timeout, and a failed or wrong one as taking twice the slowest correct time of that
query.
Figure 1
Queries per second is 409 over the time all queries take together after the last load. The cost is
that time at the machine's price per hour (methodology, Machines).
The full methodology: machines, systems, isolation, how
results are checked, and how to run it yourself.