Repository navigation
Release of DB Olympics #2
Description
Hello,
We have been working on a new database benchmark meant for agentic workflows. The benchmark is inspired by ClickBench, but the queries are written by agents (Claude Sonnet) that try to answer business questions for a simulated e-commerce platform. The benchmark contains 409 queries targeting the SQL standard with common extensions. The dataset is scalable, and verified results from scale factor 1 to 1000 are available. The schema is based on the open-source Heron e-commerce schema (251 tables). The data generator, designed by Claude Opus, uses causal simulation, so skewness, correlations, and messy, incident-laden data arise as they do in a real business. By using agents, we have tried to reduce the impact of our own biases and produce a realistic workload.
Right now, we are running the evaluation locally for cost reasons, but the performance is comparable to c8a instances on AWS, so we price the runs according to the instance's on-demand prices (we do not consider data storage cost for now).
We want to announce this benchmark publicly in the first week of November. Before that, we wanted to notify you, and we'd deeply value any thoughts and suggestions you have to help improve this benchmark. We have done our best to ensure each system is measured accurately. If you would like to check your system's connectors, they are available at https://github.com/dbolympics/dbolympics/tree/main/systems. We want to make this comparison as fair as possible to everyone.
cedardb: @lukasvogel, @pfent, @ChrisWint
clickhouse: @alexey-milovidov, @rschu1ze
datafusion: @alamb
duckdb: @Mytherin, @hannes, @peterboncz
firebolt: @hall-alex, @moshap-firebolt, @wagjamin
gizmosql: @prmoore77
hyper: @vogelsgesang, @caetanosauer
In this benchmark, wrong results and errors are penalized. All systems except Postgres, Hyper, DuckDB, GizmoSQL, and ClickHouse_brew (ClickHouse + Umbra Query Optimizer) return wrong results for some queries. We allow simple query rewrites to support various dialects, but unfortunately, not everything can be covered. You can visit https://dbolympics.com/ for an overview of the results.
Some of the issues may, of course, be our errors. We are happy to work with you to merge in fixes before or after the announcement. We intend to keep the repo and website up for the foreseeable future, so there is no deadline for improvements.
Best
Altan, Tobias
Activity
Mytherin commentedon Oct 22, 2026
Looks very interesting - I like the inclusion of cost which I think is very relevant.
I wonder how you are thinking about adding systems to the benchmark? One issue I see with ClickBench is that a lot of systems have been getting added that are not relevant which dilutes the benchmark and creates confusion more than anything. For example, one of the systems at the top of ClickBench right now is a vibe-coded project that is itself stated as not being meant for production use. Especially in todays age of AI it seems risky to allow for this - it's trivial to fire up an AI to generate a "system" that performs well at a specific benchmark, especially if reliability or performance on general-purpose queries is of no concern.
There's also the related issue of derivative systems - we've seen this a lot with both ClickHouse and DuckDB where a new system pops up that is wrapping one of the two. Since ClickHouse and DuckDB do well on the benchmark the derivative system also does well on the benchmark - but this can also of course lead to a bunch of systems flooding the list as anyone can just fork an existing system, slap a new name on it and submit a PR to add it to the benchmark. ClickBench has added labels for these (
X derivative) but it seems to add noise more than anything else unless there is something fundamentally different about the system that impacts query performance for the benchmark.prmoore77 commentedon Oct 22, 2026
hi @toschmidt - I'm working on a submission for Hardcharger now! (Successor to GizmoSQL)...
prmoore77 commentedon Oct 22, 2026
Done - please see: #4
toschmidt commentedon Oct 22, 2026
That is a good point. We hope to avoid the problem for now by using a large query set with higher complexity than ClickBench and an evolving dataset. Of course, this might not be enough, and AI-generated systems could still be tuned for this benchmark. An alternative we discussed is to keep the query set secret and publish only query samples. We decided against this for now, since we want to publish Umbra results as well, and this might lead to conflicts if Altan or I have access to the queries. A secret query set also complicates submitting results for cloud offerings like Snowflake, Databricks, Redshift, ClickHouse, or MotherDuck. Another option is to regenerate the query set every 3 or 6 months, thereby discouraging tuning. @Mytherin, what is your take?
Yeah, that is true. Personally, I would exclude systems that just change the system name without significant modifications. But we should establish a clear rule here, since we still want to include hard forks of existing systems (e.g., Firebolt, CedarDB). Also, should we include systems like pg_clickhouse and pg_duckdb? These are usually slower than DuckDB and ClickHouse on their own, but could be seen as derivations of PostgreSQL since they change its query engine. Similarly, clickhouse_brew combines Umbra's query optimizer with ClickHouse to provide Postgres compatibility.
Would you consider Hardcharger/GizmoSQL as different enough from DuckDB? It seems like they run plain DuckDB internally, but have made some changes to it.
tmuehlbauer commentedon Oct 22, 2026
This looks very promising! The inclusion of QPS and cost are important dimensions. I also like that wrong results get penalized in scoring.
Directionally I'd like to see some form of detection of intentional benchmark overfitting. I understand it's hard to quantify or detect but it is increasingly becoming a bigger issue with vibe coding. If you let a coding agent optimize it'll happily build things that break general applicability for a real productized database.
Iceberg/Parquet will be a good addition. While arguably not the best format, it has basically become the new CSV. It's a good insight into how all the engines can handle the common sharing format used today.
Mytherin commentedon Oct 23, 2026
Perhaps a good metric is to either (1) look at how relevant a system is in the real world (looking at e.g. github stars,
db-enginesranking, etc), or (2) have the system prove that is general purpose enough by completing some sort of stress test successfully, e.g. successfully running some combination of TPC-H, TPC-DS, TPC-C, TPC-E, ClickBench, SQLStorm.That's fair - it can definitely be interesting to include them, but maybe it would be stylistically better to hide these derivative engines by default to prevent clogging up the results too much to the point of unreadability.
peterboncz commentedon Oct 23, 2026
toschmidt commentedon Oct 23, 2026
Okay, for now, we will hide systems that repackage existing systems. Hard forks of systems are allowed, but it should be clear how the fork differs from the original system (e.g., by replacing relevant parts of the engine or changing the data layout).
prmoore77 commentedon Oct 23, 2026
Not cool, bro
hbirler commentedon Oct 23, 2026
Hello,
Thanks for all the comments! With this benchmark, we have multiple goals.
Firstly, we want to reward the admirable work modern production systems have been doing in bringing performance improvements to everyone. DuckDB, Hyper, ClickHouse and similar systems have shown people orders of magnitude improvements in real world workloads are possible.
Another goal is to discover the “speed of light.” Given a relatively nontrivial benchmark, what is the fastest possible way of running it?
ClickBench has been incredible impactful in both. It showed how fast great production systems had become. And it also pushed everyone towards the speed of light. There have been many days we saw another system beating our system Umbra, and did day long discussions on how that could be possible and what generically useful techniques we could implement to achieve such performance. That experience was super rewarding.
So I believe we should allow for derivative/frankenstein systems. But we can easily tag them as such and put them behind a selector. We might need some time to think about a good definition of derivative though, feel free to make suggestions.
tmuehlbauer commentedon Oct 23, 2026
More thoughts after skimming through the query set:
caetanosauer commentedon Oct 23, 2026
Thank you for this open effort and let's keep collaborating!
I'd like to +1 @Mytherin 's comment:
I think each system could be assigned a global score as a metric of how general-purpose it is. I think just SQLStorm would suffice for that since it is comprehensive enough and a lot of thought was put into that benchmark for exactly this purpose. I would even support calling it just the "SQLStorm score".
alamb commentedon Oct 23, 2026
Thank you for this benchmark -- it is interesting and I am sure it will help us improve performance across the ecosystem
I have one request: Could you please either
My reasoning
This benchmark appears to measure (and prominently report) query performance, and the scripts talk about "parquet" data so I assumed this chart was running against Parquet:
But looking more carefully, it appears that DataFusion is the only system actually running against the parquet input files, while the others loads the files into a native format of some type and that load time is not accounted for in the headline performance chart.
For example, I could spin this image as "DataFusion has 100x faster load time than the nearest other system" which doesn't seem like a useful conclusion to draw as it is benchmarking a different usecase (ad hoc queries on data lakes, for example)
toschmidt commentedon Oct 23, 2026
We are working on a comparison using Parquet as the primary storage format. We will release the results next week. Right now, systems like Firebolt, Datafusion, and ClickHouse are penalized for incorrect results or query failures (2x the execution time of the slowest system with a correct result).
alamb commentedon Oct 23, 2026
Thank you
I believe more that half of the reported wrong results for DataFusion are due to a dialect difference (this took my agent 5 minutes to find, BTW)
to_chardialect differences #7I recommend at the minimum you do the same thing (have an agent tell you why the results are "wrong") for ClickHouse before publishing these results
I am working on the other reported errors (which do indeed appear to be incorrect/inconsistent behavior in DataFusion FWIW)
hbirler commentedon Oct 23, 2026
@alamb most ClickHouse issues are due to correlated queries that they cannot execute and not supporting division on arbitrary numeric inputs. Unnesting is not a simple textual rewrite. And division requires semantic analysis to correctly determine cast targets (not really a generic rewrite or a simple issue of a function having a different name). We looked for something like a setting to allow arbitrary divisions but did not see anything (we already have various settings on in multiple databases to make them behave closer to the standard).
hbirler commentedon Oct 23, 2026
On the suggestions to use SQLStorm: It is practically impossible to determine correct results for SQLStorm queries. Those queries are generated by agents that were explicitly instructed to make queries complex. So they generated complex patterns that are not necessarily realistic (such as LIKE and IN expressions within join conditions), and various other constructs that are not supported by the SQL standard. Such a suite of queries is great for testing databases and algorithms such as unnesting to make sure they do not break, but hard to compare multiple databases on.
The queries on DBOlympics are generated by agents that were instructed to keep to the standard. Regardless, many of the queries used either features not defined by the standard or things that the standard leaves implementation-defined. We had to rewrite a large amount of queries to bring them closer to the standard (or consensus), solve a bunch of issues with various dialects, and make sure all databases can return correct^1 results. We even have tried using the Umbra query parser and rewriter as a frontend for ClickHouse to prove their execution engine could run all queries. We still seem to have missed things though (see the pull requests for datafusion which we are looking into).
1: On correctness, we know of no database that follows the standard when it comes to numerics. So we have allowed for noncompliant behavior when it comes to numerics (we just check whether the result is close enough).
alamb commentedon Oct 23, 2026
I think you will be disappointed if you expect the SQL standard to define semantics precisely. As you seem to have been re-discovering, the term "standard" is a misnomer for what "NCITS/ISO/IEC 9075-2:2011 [2012] -- (ISO/IEC 9075-2:2011, IDT)" is; Despite the name, it is many things, but it is not a precise technical standard followed by database management systems.
caetanosauer commentedon Oct 24, 2026
Just to clarify: I proposed SQLStorm as a tool to assess how general-purpose a system is, not its correctness. Some of those overfitted ClickBench contestants are likely to error, crash, or time out on a large share of SQLStorm queries, and a simple metric could capture that.
Correctness is a whole can of worms as the discussion already reveals. In general, I do like the idea of admitting different results as long as they are reasonable (e.g., numeric ranges or rounding), as well as relying more on consensus than strict semantics.