feat: add RLE v2 run length widths#7376
Conversation
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
westonpace
left a comment
There was a problem hiding this comment.
Why are we doing this as a feature flag and not part of the 2.3 file version?
Is the goal to somehow get something out faster than a 2.3 release?
Or maybe we want to move towards a more incremental release strategy?
I don't think it's wrong, but it is different, and I am curious why the change.
My first thought is that the RLE v2 change is not significant enough to be worth including in v2.3. However, I agree it is better to follow the same pattern. I have updated the code. |
…value count (#7708) #7376 made the RLE miniblock decoder reject chunks whose run lengths sum past the declared value count. The legacy (pre run-length-width) miniblock encoder rolled back to a power-of-2 checkpoint after a run had already crossed it, so existing files contain chunks that declare 2048 values but hold runs summing slightly more; the excess values are re-encoded at the start of the next chunk. The legacy decoder truncated the excess, so those files were valid when written — after #7376 they fail to read with: ``` Invalid user input: RLE decoding overflowed expected value count: produced at least 2080, expected 2048 ``` This restores the truncating behavior for miniblock decoding: clamp the final crossing run at the declared count and ignore trailing legacy runs, while still rejecting underflow and zero run lengths. Block payloads have no chunk boundaries, so an overflow there can only be corruption and remains a hard error. Current encoders emit exact chunks, so this only changes how legacy files are read. Includes regression tests for the legacy chunk-boundary overflow and the strict block path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved run-length (RLE) decoding to cap results to the expected number of values, avoiding failures when runs exceed the requested range. * Enhanced compatibility with legacy data by clamping miniblock overflow at chunk boundaries and ensuring decoded output matches the expected length and values. * Updated miniblock test coverage to reflect the new overflow semantics, and added a regression test for boundary-crossing truncation. * Preserved strict overflow handling for full block payloads, which still reports an error. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary - fail RLE miniblock encoding when chunk selection cannot advance instead of returning partial buffers with the original value count - assert that raw encoded run lengths match each declared chunk count across U8, U16, and U32 run-length widths - retain the exact-prefix implementation introduced in #7376; this does not change the file format ## Context `LANCE_MINIBLOCK_MAX_VALUES=1` is currently accepted, but a multi-value RLE page cannot represent a non-final one-value chunk because `log_num_values == 0` identifies the final chunk. The encoder previously broke out of the loop and returned incomplete data while retaining the original `num_values`. ## Test plan - [x] `cargo test -p lance-encoding` (430 passed, 5 ignored) - [x] `cargo fmt --all -- --check` - [x] `cargo clippy --all --tests --benches -- -D warnings` Made with [Cursor](https://cursor.com) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Prevented incomplete or invalid compressed data from being returned when encoding cannot make progress. * Improved error reporting with contextual details for encoding failures. * Fixed run-length handling across the 2,048-value boundary. * **Tests** * Added coverage for multiple run-length widths and zero-progress encoding scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Cursor <[email protected]>
Summary
Adds RLE v2 run-length widths so newly created datasets can write RLE pages with
u16oru32run lengths instead of splitting every run at 255 values. The capability is recorded as a reader feature flag and is only enabled when a new dataset is created withWriteParams::enable_rle_v2; existing unflagged datasets reject attempts to turn it on mid-stream.Closes #7327.
Benchmark
Ran on
xuanwo-lance-lazy-metadata-benchwith a #6941-style sorted low-cardinalityasset_idworkload.150M rows / 5k assets / random5 value167.36 MiB164.57 MiB1.67%150M rows / 5k assets / by-asset5 value7.62 MiB2.03 MiB73.34%The first row keeps the random low-cardinality value column from the issue-like workload, which dominates total size. The second row isolates the long-run case RLE2 targets.
Validation
Validated with focused RLE2 tests and full Rust clippy before publishing.