Repository navigation
Commit 0767966
feat(btrblocks): allow string dictionaries in the CUDA compression preset (#10917)
## Summary
- Tracking Issue: #10914
- Depends on #10915 and #10916; this branch contains both. The diff of
its own is one line plus a test.
`CompressionMode::Cuda` excludes `StringDictScheme`, but
`execute_dict_varbinview` already decodes string dictionaries on the
GPU. Keeping the scheme makes GPU-mode files 16.0% smaller across
compress-bench's 17-dataset GPU suite (3,351.3 → 2,813.9 MB).
Per-dataset gains: CMSprovider 45.7%, Food 52.9%, HashTags 33.2%, both
TPC-H l_comment variants 12.2%, Euro2016 10.7%. The issue has the full
table.
## Changes
- `vortex-btrblocks/src/builder.rs`: drops `StringDictScheme` from the
Cuda exclusions and updates the comment. A new test checks that the
preset keeps string dictionaries but not binary ones. This is the only
change to CPU-compressor code.
Evidence with this branch (A40, CUDA 13.0):
- `--gpu-verify` passes on all 17 datasets. develop plus only this line
fails 5: taxi, CMSprovider, Euro2016, Food, HashTags.
- The CPU-fallback census is identical to develop's: 59 constant-array
fallbacks, none from dictionaries.
- The dictionary gather takes at most 6.1% of decode time and reaches
62-85% of the measured 563 GB/s.
- `vortex-cuda` lib suite on this branch: 723 passed, 0 failed.
- `compress_gpu` fuzz (CI flags, ASAN): 100,000 runs in 256 s, 0
crashes.
## Benchmarks
Setup:
- `compress-bench --gpu-decompress --iterations 5`, a fresh process per
run, `taskset -c 2-7`, 3 ABBA rounds.
- Speedup = develop time / branch time.
- The same-design A/A floor is 0.14 in |log|, so a "faster" claim needs
a 95% CI above 1 and a speedup above 1.32×.
| Dataset | File MB, develop → branch | Speedup [95% CI] | |
|---|---|---:|---|
| Food | 84.7 → 39.9 | 1.55× [1.41, 1.71] | faster |
| CMSprovider | 771.0 → 418.5 | 1.37× [0.66, 2.86] | within noise |
| Euro2016 | 163.3 → 145.9 | 1.46× [0.92, 2.32] | within noise |
| TPC-H l_comment chunked | 169.6 → 148.8 | 1.36× [0.55, 3.34] | within
noise |
| taxi | 52.0 → 51.5 | 1.28× [0.32, 5.11] | within noise |
| HashTags | 235.2 → 157.0 | 1.22× [0.89, 1.69] | within noise |
| TPC-H l_comment canonical | 169.6 → 148.8 | 1.18× [0.71, 1.96] |
within noise |
| Arade | 144.2 → 141.7 | 1.05× [0.42, 2.68] | within noise |
The nine datasets whose files don't change got one AB/BA round each.
Eight came out at 1.01-1.18×, and one (wide table cols=1000 chunks=50)
at 0.91×, inside the A/A floor. The measurements are from an A40
(GA102); the repo's GPU benchmarks run on A10G.
## API changes
`only_cuda_compatible()`, and therefore `cuda_write_strategy`, can now
emit string dictionaries.
- An `ArrowDeviceArrayStream` with the default
`DictionaryExport::Preserve` rejects a batch whose Arrow schema differs
from the first.
- With this change, 6 of CMSprovider's 14 string columns mix dictionary
and constant chunks.
- Today's Cuda files already do this in numeric columns: 7 of taxi's 18
numeric columns and 5 of CMSprovider's 12.
- Streams with `DictionaryExport::Decode`, which the FFI scan uses, are
unaffected.
Overlap: #10425 adds a line next to the removed one, so whichever merges
second needs a one-line rebase. #9762 changes dictionary selection, so
the sizes are as of a59294e.
## AI assistance
Developed with Claude Code (Claude Opus 5.5), which wrote most of the
code, tests and this description under my direction. I reviewed the diff
and the measurements.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Tanish Gudise <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Alexander Droste <[email protected]>1 parent 30446ea commit 0767966
1 file changed
Lines changed: 16 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
| |||
76 | 78 | | |
77 | 79 | | |
78 | 80 | | |
79 | | - | |
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
| |||
321 | 322 | | |
322 | 323 | | |
323 | 324 | | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
324 | 335 | | |
0 commit comments