Repository navigation
Blocked BitPacked: Add Compressor support via BitPackingScheme mode - #10245
Conversation
Merging this PR will improve performance by 18.8%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | threshold_avx512[8] |
2.1 µs | 1.8 µs | +18.8% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/bitpacked-v2-scheme (3711acb) with develop (d42e7c9)
c98cb99 to
f638117
Compare
4c72f98 to
d0d2cc6
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (8)
Powered by Polar Signals Cloud |
d0d2cc6 to
20c3fb4
Compare
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.093x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.022x ➖, 0↑ 4↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 5↑ 4↓)
duckdb / parquet / ns (0.998x ➖, 5↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.012x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.986x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.957x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.032x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.047x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.972x ➖, 1↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.984x ➖, 1↑ 0↓)
vortex / parquet / ns (0.995x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.995x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
20c3fb4 to
f3fad6b
Compare
BitPackingScheme to per-block bit widths when fastlanes.bitpacked.v2 is allowedBitPacked: Add Compressor support via BitPackingScheme mode
…ked.v2 is allowed Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Signed-off-by: Matt Katz <[email protected]>
Select block widths before encoding, retain primitive arrays when all widths are native, and use global bitpacking for uniform widths. Declare both serialized IDs for the V2 scheme and share patch handling. Cover the fallback and wire-format choices, register Patched in the test session, and update stock-ticker size expectations for the verified saving from blocked delta bases. Signed-off-by: Matt Katz <[email protected]>
8da23f9 to
3711acb
Compare
Tracking Issue: #10167 ## Summary `FoR` now decodes a blocked `BitPacked` child in one pass, the same as a global-width child. Before, the child was decoded to a primitive array first and the references were added in a second pass. A block and a FoR chunk are both 1024 values, so each block unpacks at its own width and adds its chunk's reference directly. ## Changes - `unpack_chunks` walks the blocks of a blocked child, each at its own width. - `execute_block_offsets` and `for_each_packed_block` are shared with the blocked `BitPacked` decoder. - Block offsets widen to `u64` directly instead of through a lazy `cast`. ## Benchmarks `for_decode`, per-chunk references, local M5 Max: | | Global width | Blocked, before | Blocked, after | | --- | --- | --- | --- | | u32 | 9.62 µs | 18.54 µs | 9.87 µs | | i64 | 11.95 µs | 20.04 µs | 12.29 µs | ## Testing - `fused_decode_is_wrapping_add` draws a bit width per block. - `for_fuses_blocked_child` covers partial blocks, zero and native widths, and constant and per-chunk references. ## Follow-up `FoRScheme` still bit-packs its child with `BITPACKING_V1`, so the compressor doesn't write this combination yet. That needs #10245. --------- Signed-off-by: Matt Katz <[email protected]>
Tracking Issue: #10167
Summary
Support blocked
BitPackedencoding in the compressor by adding a newV2mode toBitPackingScheme. LikeFoRScheme V2andDecimalScheme V2, the new versions use the currentV1by default and are only "refined" toV2when the corresponding edition is enabled.The new scheme mode is quite naive for now. Compression estimate for V2 is the same as for V1. In general, blocked BP will compress an array at least as much as regular BP, so this estimate is a decent approximate lower bound on compression ratio.
CUDA
The CUDA preset excludes
fastlanes.bitpacked.v2, as it doesfastlanes.for.v2. CUDA decoding returns an error for per-block bit widths.