Repository navigation
Conversation
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.988x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 1↑ 0↓)
duckdb / duckdb / ns (0.991x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 1↑ 2↓)
datafusion / vortex-compact / ns (1.003x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.001x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.959x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.872x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.845x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.061x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.005x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.981x ➖, 1↑ 0↓)
|
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 1↓)
duckdb / duckdb / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.976x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.983x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.978x ➖, 1↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.043x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.988x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.850x ✅, 5↑ 0↓)
datafusion / vortex-compact / ns (0.878x ✅, 4↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.006x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 1↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.003x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.891x ✅, 39↑ 6↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.891x ➖, 3↑ 1↓)
datafusion / vortex-compact / ns (0.943x ➖, 2↑ 3↓)
datafusion / parquet / ns (1.031x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.836x ➖, 4↑ 0↓)
duckdb / vortex-compact / ns (1.059x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.012x ➖, 1↑ 4↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.908x ➖, 5↑ 1↓)
datafusion / vortex-compact / ns (0.953x ➖, 3↑ 2↓)
datafusion / parquet / ns (0.919x ➖, 5↑ 2↓)
duckdb / vortex-file-compressed / ns (0.889x ➖, 3↑ 0↓)
duckdb / vortex-compact / ns (0.924x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.942x ➖, 3↑ 2↓)
|
Signed-off-by: Matt Katz <[email protected]>
3fccd52 to
34f556e
Compare
Summary
A bounded list read whose selection has holes no longer reads every element between the first and last selected list and filters afterwards. The elements read is masked to the elements the selected lists reference, so the elements reader skips the rest, whole chunks included, and the lists are rebuilt over only what was read.
Changes
select_listsandkept_elements(list/reader.rs): the element mask and the kept lists' offsets from the cropped offsets, fetched concurrently with the validity.lists_over: checks the elements read against the last offset before building the list.CountingSegmentSourcerecords the requested segment ids.list<u64>(4k × 2,500), raw, half the rowsTesting
sparse_mask_skips_element_chunks_between_selected_lists: a mask selecting the first and last list requests two element chunks.random_list_uses_match_memory: random ranges and masks over nullable lists match the in-memory result.Follow-up
list_transform's lambda body intoListLayoutelement projection #10948 pusheslist_transformbodies into the same read.