Repository navigation
Preserve DuckDB dictionaries across run-end chunks - #11378
Merged
Merged
Conversation
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
robert3005
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
robert3005
approved these changes
Nov 10, 2026
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.039x ➖, 0↑ 2↓)
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 1↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.914x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.935x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.929x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.948x ➖, 2↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
ChristopherWirt
added a commit
to quantcast/vortex
that referenced
this pull request
Nov 11, 2026
## Summary QC `develop` is missing 13 upstream commits. Merge `vortex-data/vortex:develop` at `cb613ac1c0adc56cbe63dfc79e4bbc431ad0bc52` into QC `develop` at `b2c429ac2830b883cb37c1ff9262dc9e3cb7e483`. Merge this PR with a merge commit to preserve upstream ancestry for future syncs. ## Changes - Import typed Arrow extension plugins and round-trip metadata preservation, run-end export fixes, and DuckDB dictionary reuse across chunks ([vortex-data#11360](vortex-data#11360), [vortex-data#11375](vortex-data#11375), [vortex-data#11378](vortex-data#11378)). - Bring in improved string dictionary selection, spatial contains/within pruning, and safe PCO decoding ([vortex-data#9762](vortex-data#9762), [vortex-data#11372](vortex-data#11372), [vortex-data#11363](vortex-data#11363)). - Include CUDA Widen execution, bounded dictionary gathers, encoded-child host copies, and string dictionaries in the CUDA compression preset. - Reconcile the upstream Java I/O ports with their existing QC implementations, preserving read metrics, Kryo-safe Hadoop broadcasts, lifecycle handling, session defaults, and regression coverage. Correct the coalescing default's Javadoc to 1 MiB. ## Testing - Pinned `nightly-2026-09-10` Rust formatting for all affected crates, root/subproject `./gradlew spotlessApply`, and clang-format for the changed CUDA kernels passed. Reviewed and included the formatter diff. - `git diff --check` passed; no unresolved merge entries or conflict markers remain. The worktree is clean and both remote tips are parents of the signed-off merge commit. - All 13 imported commits have sign-off trailers and retain their original upstream metadata. - Tests, builds, lint, and benchmarks were not run locally, following repository guidance. --------- Signed-off-by: Daniel King <[email protected]> Signed-off-by: Tanish Gudise <[email protected]> Signed-off-by: Alexander Droste <[email protected]> Signed-off-by: Christopher Wirt <[email protected]> Signed-off-by: Nemo Yu <[email protected]> Signed-off-by: Connor Tsui <[email protected]> Signed-off-by: Will Manning <[email protected]> Signed-off-by: Robert Kruszewski <[email protected]> Co-authored-by: Dan King <[email protected]> Co-authored-by: Tanish Gudise <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Alexander Droste <[email protected]> Co-authored-by: Connor Tsui <[email protected]> Co-authored-by: Nemo Yu <[email protected]> Co-authored-by: Will Manning <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #11375 and the DuckDB exporter review.
Chunked run-end inputs currently lose their cached dictionary when an output batch crosses a chunk boundary. Exposing the existing batch codes through
DictionaryPartslets those ranges reuse the values dictionary, including ranges shared with ordinary dictionary chunks.Changes
Keep the run-end specialization because routing through
to_dict()would make the generic exporter materialize the entire codes array. The existing fill loop also remains: usingslice_relative_endsincreased standalone single-row-run export time in all five trial pairs (median paired ratio 1.064).Empty inputs now avoid run lookup and values conversion, and zero-length requests avoid scalar access. Constant batches include requests ending exactly at a run boundary. Compact inline regressions cover empty requests, slice offsets, exact run boundaries, nulls, successive batches, and shared or different dictionaries across chunks. The existing null and chunked-values tests remain.
Measurements on
702fd58dea, before the test and module cleanup, showed about 15% lower batch-export time for single-row runs and 16% for long runs. Rust allocations per export loop fall from 146 to 112 and from 212 to 188, respectively. Standalone export and construction timings show no clear material change. Native DuckDB allocations were not measured.Validation
cargo nextest run -p vortex-duckdb -E 'test(exporter::run_end::)': 7 passed, 265 outside the filter.cargo clippy -p vortex-duckdb --all-targets --all-features -- -D warnings: passed.cargo +nightly-2026-09-10 fmt -p vortex-duckdb: passed.Benchmark methodology and full results
Baseline:
b367941e9d490b4a19ef9bdd6653f63d712c581fwith the same benchmark harness. Candidate:702fd58dea3c0eeb214d590b3cc6d6b5b3c89c9b. This comparison isolates the follow-up from the Arrow extension/property stack.The measurement harness and its Cargo target remain available in that measured commit. The permanent benchmark target was removed during review, along with duplicate test setup and the module split. The production algorithm is unchanged, but these measurements were not rerun after the file move.
macOS 27.0.1, aarch64, rustc 1.99.1, LLVM 23.1.1, Cargo bench profile, and DuckDB 2.0.0. Build both revisions with
cargo bench -p vortex-duckdb --bench run_end_export --no-run. Run their saved executables with--bench --sample-count 300 --sample-size 20 --color neverin five pairs, alternating baseline/candidate order. The table shows the median of each executable's five run medians, plus the median of the five paired candidate/baseline ratios. These two aggregations need not agree exactly.Inputs are non-null i64 run values with run lengths 1 or 8192. Short-run sources have 16,383 runs, and long-run sources have eight. Chunked inputs repeat the same source four times. Every input is sliced from row 17 to 13 rows before its end, so exports span multiple batches and chunk boundaries fall inside output batches. Fixtures execute the slice to its encoded representation before exporter construction. Construction uses a fresh conversion cache per iteration. Batch export uses a warm cache and excludes exporter construction and data-chunk creation.
AllocProfiler<MiMalloc>counts Rust allocations in the measured region, including its timing overhead. It does not count native DuckDB allocations. Neitherxctracenorheaptrackis available on this machine. Allocation figures below are allocation counts and requested bytes, not peak resident memory.The shared-chunk batch timing improved in every pair. Standalone controls and construction moved much less, and the standalone long-run ratios varied in direction. No compiler-output analysis was performed, so those small movements are not attributed to a particular code-generation change. These microbenchmarks do not establish an end-to-end query speedup.