Repository navigation

Commit e277ccd

authored

Blocked BitPacked: add fastlanes.bitpacked.v2 serialization (#10244)

Tracking Issue: #10167 ## Summary Adds a wire format for `BitPacked` arrays whose blocks each have their own bit width, and makes it the in-memory ID, following `fastlanes.for.v2` (#10135) and `DecimalByteParts`. | ID | Role | |---|---| | `fastlanes.bitpacked` | Frozen wire format for global-width arrays. Unchanged. | | `fastlanes.bitpacked.v2` | In-memory ID, and the wire format for per-block bit widths. | `BitPackedPlugin` declares both serialized IDs and picks one per array from `bit_widths()`: - **Global width** writes `fastlanes.bitpacked` exactly as before. Existing files read as before, and default writes stay byte-identical. - **Blocked** writes `fastlanes.bitpacked.v2`. The plugin moves into `bitpacking/plugin/`, laid out like FoR's: `mod.rs` holds the IDs and the dispatch, `v1.rs` the v1 metadata and serde, `v2.rs` the new format, and `tests.rs` the existing tests plus new ones. The v1 code is moved unchanged, except that it writes `bitpacked_v1_id()` rather than the plugin's ID. ## `fastlanes.bitpacked.v2` layout - **Buffers**: the packed buffer, as in v1. - **Children**: v1's children (patch indices, values and chunk offsets, then validity), followed by the block offsets. The block offsets are serialized like any other child, so the compressor can compress them. - **Metadata**: a prost message holding `offset`, the block offsets' unsigned `PType`, and the patches metadata. There is no bit width, and the block offsets' length, `ceil((offset + len) / 1024) + 1`, is derived. Deserialization checks for one buffer and the expected number of children, and validates the offset before deriving the block offsets' length. Construction then checks the block offsets' dtype and length. The decoder (#10202) bounds-checks the boundary values themselves. ## `BitPackedPatchedPlugin` The experimental-patches plugin declares both IDs as well. It moves a v2 array's interior patches into a `Patched` wrapper just as it does for v1, rebuilding the inner array with its block offsets instead of panicking on a blocked array. ## Editions and compressor `fastlanes.bitpacked.v2` is in no edition. Writers that enforce editions reject it, and readers that predate it report an unknown encoding. Nothing produces blocked arrays yet, so default writes don't change. `BitPackingScheme::produced_encodings` now declares `bitpacked_v1_id()`, since `BitPacked.id()` is the in-memory ID. --------- Signed-off-by: Matt Katz <[email protected]>

1 parent 644fbcd commit e277ccdCopy full SHA for e277ccd

33 files changed

encodings/fastlanes/src/bitpacking
- mod.rs
- plugin.rs
- plugin
- vtable
  - mod.rs
vortex-btrblocks
- src/schemes/integer
  - bitpacking.rs
- tests/snapshots
vortex-file/src
- tests.rs

`‎encodings/fastlanes/src/bitpacking/mod.rs‎`

-Original file line number
+Diff line change
 pub(crate) use plugin::BitPackedPatchedPlugin;
 pub use plugin::BitPackedPlugin;
 +pub use plugin::bitpacked_v1_id;
 +pub use plugin::bitpacked_v2_id;
 pub use vtable::BitPacked;
 pub use vtable::BitPackedArray;

Comments

(0)