Repository navigation
DataFusion 1TB single machine benchmark #19815
Description
Is your feature request related to a problem or challenge?
There was a challenge recently https://www.confessionsofadataguy.com/duckdb-beats-polars-for-1tb-of-data/ and I think it would be beneficial also to run the same for DataFusion
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Activity
GaneshPatil7517 commentedon Jan 14, 2026
take
comphead commentedon Jan 14, 2026
Thanks @GaneshPatil7517 I think we can try to reuse the same data generator proposed in the article https://github.com/danielbeach/rustGenerate1TB and query generator https://github.com/danielbeach/duckdbAndDaftEat1TB
comphead commentedon Jan 16, 2026
danielbeach/duckdbAndDaftEat1TB#1
GaneshPatil7517 commentedon Jan 17, 2026
great @comphead ....
comphead commentedon Nov 6, 2026
DataFusion TPC-H SF1000 on
main(a616831)Setup
57.0.0-dev,mainata616831(2026-11-06)target_partitions = 32datafusion.runtime.memory_limit = 105G(greedy pool)datafusion.runtime.temp_directory = /data,max_temp_directory_size = 900Gdatafusion.runtime.metadata_cache_limit = 2G,list_files_cache_limit = 256M,file_statistics_cache_limit = 256MThis commit already includes the revert of #25790 (#27145), so the Q18 failure below is not the #27108 regression.
Results
19 of 22 queries passed. The sum of their means is 593.6 s and the geometric mean is 24.4 s.
Failures
Q09, Q18 and Q21 failed on all 3 iterations with
Resources exhaustedwhile building hash join input.HashJoinInputreportscan spill: false, so the 105G pool fills up with build-side reservations of similar size, one per partition.Failed to allocate additional 763.0 MB for HashJoinInput[24] with 763.0 MB already allocated for this reservation - 361.1 MB remain available for the total memory pool: greedy(used: 104.6 GB, pool_size: 105.0 GB). The top consumers areHashJoinInputpartitions at 2.1 GB each.Failed to allocate additional 1440.0 KB for HashJoinInput[4] with 2.5 GB already allocated for this reservation - 247.2 KB remain available for the total memory pool: greedy(used: 105.0 GB, pool_size: 105.0 GB). The top consumers areHashJoinInputpartitions at 2.5 GB each.Failed to allocate additional 4.7 GB for HashJoinInput[24] with 0.0 B already allocated for this reservation - 1591.6 MB remain available for the total memory pool: greedy(used: 103.4 GB, pool_size: 105.0 GB). The top consumers areHashJoinInputpartitions at 6.5 GB each.comphead commentedon Nov 9, 2026
DataFusion vs Polars, TPC-H SF1000
This follows up on the DataFusion SF1000 numbers above. Polars ran on the same hardware against the same data.
Setup
57.0.0-dev,mainata616831,target_partitions = 32,memory_limit = 105G(greedy pool)2.1.0at7f902bd, streaming engine,POLARS_TEMP_DIR=/data, no explicit memory limitResults
Both engines finished 18 queries. Over those 18, DataFusion took 556.4 s and Polars took 422.7 s (1.32x). The geometric means are 23.9 s and 16.8 s (1.42x).
%%{init: {"themeVariables": {"xyChart": {"plotColorPalette": "#2a78d6, #8a8984"}}}}%% xychart-beta title "DataFusion time / Polars time, SF1000 (above 1 = Polars faster)" x-axis ["Q01", "Q02", "Q03", "Q04", "Q05", "Q06", "Q07", "Q08", "Q11", "Q12", "Q13", "Q14", "Q15", "Q16", "Q17", "Q19", "Q20", "Q22"] y-axis "DF / Polars" 0 --> 6 bar [1.74, 5.42, 1.33, 0.55, 1.56, 1.01, 1.60, 1.35, 1.46, 0.60, 0.75, 1.24, 5.43, 1.25, 1.26, 1.10, 2.42, 1.21] line [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]Observations:
comphead commentedon Nov 11, 2026
superseded by #26169