Repository navigation
Commit 583a985
authored
Blocked BitPacked: decode blocked bitwidths (#10202)
Tracking Issue: #10167
## Summary
Adds support for decoding `BitPacked` arrays with blocked bitwidths.
Now each public function that decodes bitpacked arrays with global
bitwidths has a version for blocked bitwidths.
| Global width | Blocked |
|---|---|
| `unpack_array(array, ctx)` | `unpack_array_blocked(array, offsets,
ctx)` |
| `unpack_primitive_array` | `unpack_primitive_array_blocked` |
| `unpack_into_primitive_builder` |
`unpack_into_primitive_builder_blocked` |
| `unpack_single(array, index)` | `unpack_single_blocked(array, offsets,
index, ctx)` |
Callers choose between them by matching on `bit_widths()`.
There is no blocked `unpack_map_into_builder`: its only caller, the
widening `CastKernel`, declines blocked arrays, so the mapped path would
only run in tests.
## Design
#10004 only checks block boundary values in debug builds, so the decoder
checks them itself:
- Bulk decode casts the offsets child to `u64` and executes it once,
checks that it holds one boundary per block plus a trailing one, and
validates every boundary before writing to the builder. A single offset
type keeps one copy of the block loop per value type, for 8 bytes per
1024 values. Each block must be a whole number of 128-byte rows, no
wider than the array's type, and the boundaries must span the packed
buffer.
- `scalar_at` reads only the selected block's boundaries, directly for
host primitive offsets and through `execute_scalar` otherwise, and
validates just that block.
Both use `block_range` in `bitpacking/array/mod.rs`, which
`validate_primitive_offsets` now also uses. Boundaries are relative to
the first one, which may be nonzero.
Full blocks unpack straight into the output, like the global decoder's
full chunks. A partial first or last block unpacks into one 1024-value
scratch block.
---------
Signed-off-by: Matt Katz <[email protected]>1 parent 9d5a844 commit 583a985
9 files changed
Lines changed: 799 additions & 34 deletions
File tree
- encodings/fastlanes
- benches
- src/bitpacking
- array
- bitpack_decompress
- vtable
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
58 | 62 | | |
59 | 63 | | |
60 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
0 commit comments