Skip to content

Commit 0767966

Browse files
TanishGudiseclaude0ax1
authored
feat(btrblocks): allow string dictionaries in the CUDA compression preset (#10917)
## Summary - Tracking Issue: #10914 - Depends on #10915 and #10916; this branch contains both. The diff of its own is one line plus a test. `CompressionMode::Cuda` excludes `StringDictScheme`, but `execute_dict_varbinview` already decodes string dictionaries on the GPU. Keeping the scheme makes GPU-mode files 16.0% smaller across compress-bench's 17-dataset GPU suite (3,351.3 → 2,813.9 MB). Per-dataset gains: CMSprovider 45.7%, Food 52.9%, HashTags 33.2%, both TPC-H l_comment variants 12.2%, Euro2016 10.7%. The issue has the full table. ## Changes - `vortex-btrblocks/src/builder.rs`: drops `StringDictScheme` from the Cuda exclusions and updates the comment. A new test checks that the preset keeps string dictionaries but not binary ones. This is the only change to CPU-compressor code. Evidence with this branch (A40, CUDA 13.0): - `--gpu-verify` passes on all 17 datasets. develop plus only this line fails 5: taxi, CMSprovider, Euro2016, Food, HashTags. - The CPU-fallback census is identical to develop's: 59 constant-array fallbacks, none from dictionaries. - The dictionary gather takes at most 6.1% of decode time and reaches 62-85% of the measured 563 GB/s. - `vortex-cuda` lib suite on this branch: 723 passed, 0 failed. - `compress_gpu` fuzz (CI flags, ASAN): 100,000 runs in 256 s, 0 crashes. ## Benchmarks Setup: - `compress-bench --gpu-decompress --iterations 5`, a fresh process per run, `taskset -c 2-7`, 3 ABBA rounds. - Speedup = develop time / branch time. - The same-design A/A floor is 0.14 in |log|, so a "faster" claim needs a 95% CI above 1 and a speedup above 1.32×. | Dataset | File MB, develop → branch | Speedup [95% CI] | | |---|---|---:|---| | Food | 84.7 → 39.9 | 1.55× [1.41, 1.71] | faster | | CMSprovider | 771.0 → 418.5 | 1.37× [0.66, 2.86] | within noise | | Euro2016 | 163.3 → 145.9 | 1.46× [0.92, 2.32] | within noise | | TPC-H l_comment chunked | 169.6 → 148.8 | 1.36× [0.55, 3.34] | within noise | | taxi | 52.0 → 51.5 | 1.28× [0.32, 5.11] | within noise | | HashTags | 235.2 → 157.0 | 1.22× [0.89, 1.69] | within noise | | TPC-H l_comment canonical | 169.6 → 148.8 | 1.18× [0.71, 1.96] | within noise | | Arade | 144.2 → 141.7 | 1.05× [0.42, 2.68] | within noise | The nine datasets whose files don't change got one AB/BA round each. Eight came out at 1.01-1.18×, and one (wide table cols=1000 chunks=50) at 0.91×, inside the A/A floor. The measurements are from an A40 (GA102); the repo's GPU benchmarks run on A10G. ## API changes `only_cuda_compatible()`, and therefore `cuda_write_strategy`, can now emit string dictionaries. - An `ArrowDeviceArrayStream` with the default `DictionaryExport::Preserve` rejects a batch whose Arrow schema differs from the first. - With this change, 6 of CMSprovider's 14 string columns mix dictionary and constant chunks. - Today's Cuda files already do this in numeric columns: 7 of taxi's 18 numeric columns and 5 of CMSprovider's 12. - Streams with `DictionaryExport::Decode`, which the FFI scan uses, are unaffected. Overlap: #10425 adds a line next to the removed one, so whichever merges second needs a one-line rebase. #9762 changes dictionary selection, so the sizes are as of a59294e. ## AI assistance Developed with Claude Code (Claude Opus 5.5), which wrote most of the code, tests and this description under my direction. I reviewed the diff and the measurements. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Tanish Gudise <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Alexander Droste <[email protected]>
1 parent 30446ea commit 0767966

1 file changed

Lines changed: 16 additions & 5 deletions

File tree

‎vortex-btrblocks/src/builder.rs‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,10 +63,12 @@ impl CompressionMode {
6363
excluded.push(binary::ZstdBuffersScheme.id());
6464
}
6565
Self::Cuda => {
66-
// Keep FSST, which has a CUDA decoder and direct Arrow offset-based export. Other
67-
// string fragmentation and dictionary schemes still require unsupported decode
68-
// paths. Delta has a CUDA decode kernel, but stays excluded until GPU delta
69-
// decode is benchmarked against the schemes it would displace.
66+
// Keep FSST, which has a CUDA decoder and direct Arrow offset-based export, and
67+
// string dictionaries, which the CUDA dictionary gather decodes into views over
68+
// the decoded values. Other string fragmentation schemes and binary dictionaries
69+
// still require unsupported decode paths. Delta has a CUDA decode kernel, but
70+
// stays excluded until GPU delta decode is benchmarked against the schemes it
71+
// would displace.
7072
excluded.extend([
7173
integer::DeltaScheme::default().id(),
7274
integer::SparseScheme.id(),
@@ -76,7 +78,6 @@ impl CompressionMode {
7678
float::NullDominatedSparseScheme.id(),
7779
string::NullDominatedSparseScheme.id(),
7880
fixed_size_list::FixedSizeListSparseScheme.id(),
79-
string::StringDictScheme.id(),
8081
binary::BinaryDictScheme.id(),
8182
]);
8283
// Both binary Zstd schemes are kept. Buffer-level compression preserves binary
@@ -321,4 +322,14 @@ mod tests {
321322
assert!(has_delta(builder.clone().with_compact()));
322323
assert!(!has_delta(builder.only_cuda_compatible()));
323324
}
325+
326+
#[test]
327+
fn cuda_keeps_string_dictionaries_but_not_binary_ones() {
328+
let cuda = BtrBlocksCompressorBuilder::from_session(&SESSION)
329+
.unrestricted()
330+
.only_cuda_compatible();
331+
let has = |id: SchemeId| cuda.allowed_schemes().iter().any(|s| s.id() == id);
332+
assert!(has(string::StringDictScheme.id()));
333+
assert!(!has(binary::BinaryDictScheme.id()));
334+
}
324335
}

0 commit comments

Comments
 (0)