Name: Vortex benchmark results
Creator: Vortex

Compression26 chartsencode 2.7× · decode 6.8× faster than Parquet

Write Throughput

#1vortex2.70x2067 MB/s

#2parquet1.00x1099 MB/s

#3lance0.74x680 MB/s

Scan Throughput

#1vortex6.77x11.34 GB/s

#2lance1.37x3.13 GB/s

#3parquet1.00x2.99 GB/s

Geomean throughput ratio to Parquet | Aggregate throughput (higher is better)

Compression Size35 charts11% larger than Parquet

TotalMBvs Arrow

#1parquet1,546.65.92x

#2vortex1,736.75.31x

#3lance3,325.22.93x

#4arrow-ipc9,561.61.09x

vs ParquetMinGeoMeanMax

Total compressed size, and geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)

SQL File Size16 charts

TotalMBvs Arrow

#1vortex-compact22,553.3—

#2vortex31,477.2—

#3parquet30,955.9—

#4duckdb60,934.6—

#5lance67,351.1—

vs vortexMinGeoMeanMax

Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)

Random Access (NVMe)9 chartshot 94× · cold 28× faster than Parquet

Hot

#1vortex1.00x848.05 us

#2litdata2.24x1.72 ms

#3lance2.88x3.06 ms

#4arrow-ipc5.23x15.79 ms

#5parquet93.83x77.83 ms

Cold

#1vortex1.30x3.15 ms

#2lance1.85x4.60 ms

#3arrow-ipc2.19x16.55 ms

#4litdata20.21x46.75 ms

#5parquet36.41x78.27 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Random Access (S3)9 chartshot 78× · cold 2.2× faster than Parquet

Hot

#1vortex1.00x23.84 ms

#2lance32.29x209.24 ms

#3parquet78.37x487.47 ms

Cold

#1vortex1.00x283.18 ms

#2lance2.16x625.14 ms

#3parquet2.19x621.85 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Clickbench43 chartsDuckDB 1.9× · DataFusion 1.6× faster than Parquet

Performance Summary

#1duckdb:vortex1.11x3.59 s

#2datafusion:vortex1.22x3.78 s

#3duckdb:vortex-compact1.32x4.13 s

#4datafusion:vortex-compact1.51x4.45 s

#5datafusion:parquet2.00x5.92 s

#6duckdb:duckdb2.04x5.37 s

#7duckdb:parquet2.07x5.67 s

#8datafusion:lance17.78x49.48 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=1)22 chartsDuckDB 3.6× · DataFusion 3.9× faster than Parquet

Performance Summary

#1duckdb:duckdb1.12x335.84 ms

#2duckdb:vortex1.30x381.59 ms

#3duckdb:vortex-compact1.44x416.96 ms

#4datafusion:vortex1.70x522.98 ms

#5datafusion:vortex-compact1.80x549.99 ms

#6datafusion:arrow3.31x1.08 s

#7duckdb:parquet4.72x1.39 s

#8datafusion:lance6.44x2.04 s

#9datafusion:parquet6.56x1.86 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=1)22 chartsDuckDB ≈ same · DataFusion 1.1× slower vs Parquet

Performance Summary

#1datafusion:parquet1.13x5.43 s

#2datafusion:vortex1.25x6.02 s

#3datafusion:vortex-compact1.31x6.53 s

#4duckdb:vortex-compact1.48x7.00 s

#5datafusion:lance1.53x7.69 s

#6duckdb:vortex1.58x7.57 s

#7duckdb:parquet1.60x7.66 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=10)22 chartsDuckDB 2.5× · DataFusion 2.4× faster than Parquet

Performance Summary

#1duckdb:duckdb1.15x1.01 s

#2duckdb:vortex1.31x1.16 s

#3duckdb:vortex-compact1.48x1.28 s

#4datafusion:vortex1.75x1.78 s

#5datafusion:vortex-compact1.97x1.89 s

#6duckdb:parquet3.26x2.89 s

#7datafusion:parquet4.18x3.60 s

#8datafusion:arrow14.70x16.39 s

#9datafusion:lance24.65x23.51 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=10)22 chartsDuckDB 1.7× · DataFusion 1.2× slower than Parquet

Performance Summary

#1datafusion:parquet1.14x12.09 s

#2datafusion:vortex-compact1.22x14.13 s

#3duckdb:parquet1.29x13.29 s

#4datafusion:vortex1.33x13.88 s

#5duckdb:vortex-compact2.10x22.58 s

#6duckdb:vortex2.16x23.11 s

#7datafusion:lance3.49x43.02 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=100)22 chartsDuckDB 1.7× · DataFusion 2.0× faster than Parquet

Performance Summary

#1duckdb:vortex1.14x7.11 s

#2datafusion:vortex1.56x12.51 s

#3duckdb:parquet1.92x11.21 s

#4datafusion:parquet3.15x19.81 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=100)22 chartsDuckDB 3.9× slower · DataFusion 1.1× faster vs Parquet

Performance Summary

#1duckdb:parquet1.03x36.73 s

#2datafusion:vortex1.28x47.10 s

#3datafusion:parquet1.42x54.31 s

#4duckdb:vortex4.05x167.04 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=1)99 chartsDuckDB 1.3× · DataFusion 1.5× faster than Parquet

Performance Summary

#1duckdb:duckdb1.02x1.69 s

#2duckdb:vortex1.36x2.12 s

#3duckdb:vortex-compact1.57x2.41 s

#4duckdb:parquet1.81x2.84 s

#5datafusion:vortex2.20x4.19 s

#6datafusion:vortex-compact2.49x4.80 s

#7datafusion:parquet3.26x6.03 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=10)99 chartsDuckDB 1.4× · DataFusion 1.6× faster than Parquet

Performance Summary

#1duckdb:vortex1.05x5.02 s

#2datafusion:vortex1.40x9.65 s

#3duckdb:parquet1.53x6.96 s

#4datafusion:parquet2.28x13.94 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=100)99 chartsDuckDB 1.2× · DataFusion 1.9× faster than Parquet

Performance Summary

#1duckdb:vortex1.18x22.96 s

#2duckdb:parquet1.41x27.06 s

#3datafusion:vortex1.45x120.70 s

#4datafusion:parquet2.81x148.40 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=10)99 chartsDuckDB 1.1× slower · DataFusion 1.1× faster vs Parquet

Performance Summary

#1datafusion:vortex1.27x61.38 s

#2datafusion:parquet1.35x64.74 s

#3duckdb:parquet1.40x58.83 s

#4duckdb:vortex1.53x69.02 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=100)99 chartsDuckDB 1.7× slower · DataFusion 1.7× faster vs Parquet

Performance Summary

#1datafusion:vortex1.22x235.37 s

#2duckdb:parquet1.32x156.28 s

#3datafusion:parquet2.10x350.80 s

#4duckdb:vortex2.31x320.87 s

Geomean of query time ratio to fastest (lower is better)

Statistical and Population Genetics11 chartsDuckDB 2.1× · DataFusion 2.1× faster than Parquet

Performance Summary

#1datafusion:vortex1.24x3.27 s

#2datafusion:vortex-compact1.41x3.73 s

#3datafusion:parquet2.55x4.11 s

#4duckdb:vortex3.09x8.14 s

#5duckdb:vortex-compact3.44x8.60 s

#6duckdb:parquet6.61x7.40 s

Geomean of query time ratio to fastest (lower is better)

Polar Signals Profiling13 chartsVortex only

Performance Summary

#1datafusion:vortex1.00x523.30 ms

Geomean of query time ratio to fastest (lower is better)

fineweb [nvme]9 chartsDuckDB 4.9× · DataFusion 3.4× faster than Parquet

Performance Summary

#1duckdb:vortex1.05x224.53 ms

#2datafusion:vortex1.28x220.14 ms

#3duckdb:vortex-compact3.96x948.54 ms

#4datafusion:parquet4.38x621.51 ms

#5duckdb:parquet5.15x779.21 ms

#6datafusion:vortex-compact5.23x1.01 s

Geomean of query time ratio to fastest (lower is better)

fineweb [s3]9 chartsDuckDB 1.2× · DataFusion 2.5× faster than Parquet

Performance Summary

#1datafusion:vortex1.03x3.08 s

#2datafusion:vortex-compact1.14x3.40 s

#3duckdb:vortex1.65x5.48 s

#4duckdb:vortex-compact1.71x5.46 s

#5duckdb:parquet2.04x5.37 s

#6datafusion:parquet2.59x6.61 s

Geomean of query time ratio to fastest (lower is better)

appian [nvme]8 chartsDuckDB ≈ same · DataFusion 1.2× faster vs Parquet

Performance Summary

#1duckdb:duckdb1.13x1.62 s

#2duckdb:parquet1.24x1.68 s

#3duckdb:vortex1.30x1.83 s

#4duckdb:vortex-compact1.31x1.84 s

#5datafusion:vortex1.89x11.01 s

#6datafusion:vortex-compact1.90x11.06 s

#7datafusion:parquet2.35x11.43 s

Geomean of query time ratio to fastest (lower is better)

clickbench-sorted [nvme]10 chartsDuckDB 2.1× · DataFusion 3.1× faster than Parquet

Performance Summary

#1duckdb:vortex1.06x210.55 ms

#2datafusion:vortex1.13x250.33 ms

#3duckdb:vortex-compact1.39x265.35 ms

#4datafusion:vortex-compact1.69x495.04 ms

#5duckdb:duckdb1.97x336.20 ms

#6duckdb:parquet2.21x446.96 ms

#7datafusion:parquet3.49x1.26 s

#8datafusion:lance77.75x20.50 s

Geomean of query time ratio to fastest (lower is better)

job [nvme]113 chartsDuckDB 3.8× faster than Parquet

Performance Summary

#1duckdb:duckdb1.08x4.82 s

#2duckdb:vortex1.27x5.50 s

#3duckdb:parquet4.88x20.13 s

Geomean of query time ratio to fastest (lower is better)

tpch-nested/list sf=1 [nvme]22 chartsDuckDB 7.7× · DataFusion 7.8× faster than Parquet

Performance Summary

#1duckdb:vortex1.13x739.84 ms

#2duckdb:duckdb1.24x787.68 ms

#3duckdb:vortex-compact1.46x940.80 ms

#4datafusion:vortex1.50x1.02 s

#5datafusion:vortex-compact1.75x1.16 s

#6datafusion:lance5.60x5.04 s

#7duckdb:parquet8.69x5.58 s

#8datafusion:parquet11.61x7.60 s

Geomean of query time ratio to fastest (lower is better)

tpch-nested/list sf=100 [nvme]22 chartsDuckDB 1.5× · DataFusion 1.9× faster than Parquet

Performance Summary

#1duckdb:vortex1.07x26.17 s

#2datafusion:vortex1.42x41.89 s

#3duckdb:parquet1.65x37.54 s

#4datafusion:parquet2.72x67.05 s

Geomean of query time ratio to fastest (lower is better)

vortex [nvme]3 chartsDuckDB 1.7× faster · DataFusion 1.4× slower vs Parquet

Performance Summary

#1datafusion:parquet1.15x497.61 ms

#2duckdb:vortex1.24x478.75 ms

#3datafusion:vortex1.65x652.63 ms

#4duckdb:parquet2.07x789.56 ms

Geomean of query time ratio to fastest (lower is better)

String Encoding24 charts

Write Throughput

#1fsst1.00x224 MB/s

#2onpair-120.65x155 MB/s

#3onpair-160.63x150 MB/s

Scan Throughput

#1onpair-161.24x4.56 GB/s

#2onpair-121.24x4.61 GB/s

#3fsst1.00x3.69 GB/s

Geomean throughput ratio to FSST | Aggregate throughput (higher is better)

String Encoding Size24 charts

TotalMBvs Arrow

#1onpair-16245.42.82x

#2onpair-12243.82.82x

#3fsst322.32.13x

vs FSSTMinGeoMeanMax

Total file size, and geometric means of file sizes versus Arrow (higher is better) and versus FSST (lower is better)

Benchmarks

Write Throughput

Scan Throughput

Hot

Cold

Hot

Cold

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Performance Summary

Write Throughput

Scan Throughput