You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use the shared cascading compression framework for sparse structural position and count arrays instead of growing a separate set of format-specific encoding choices.
Context
#7754 represents position sets as empty, all, range, or explicit and count sets as empty, constant, or explicit. Explicit positions are converted to deltas before the resulting u64 array enters CompressionStrategy; explicit counts enter the same strategy directly. The metadata-only variants avoid structural buffers and let readers retain O(1) normalized forms.
The review in #7754 (comment) noted that these choices overlap with cascading compression. General delta and range encodings could simplify the sparse wire contract while also benefiting non-sparse integer and offset paths.
This follow-up does not block #7754. If it changes the Lance 2.3 SparseLayout wire contract, the change must land before 2.3 is declared stable; compatibility with draft 2.3 files is not required.
Requirements
Add composable delta and range encodings to the block compression path.
Let cascading selection consider constant, delta, range, RLE, bitpacking, dictionary, general compression, and flat encoding where applicable.
Define whether sparse positions enter the cascade as absolute positions or deltas, including the cost model used to choose.
Preserve metadata-only or otherwise zero-payload representations for empty, all, range, and constant sequences.
Preserve O(1) normalized sparse plans for those representations instead of materializing large u64 arrays.
Keep malformed metadata and decoded ordering, bounds, cardinality, and overflow checks returning format errors without panics.
Goal
Use the shared cascading compression framework for sparse structural position and count arrays instead of growing a separate set of format-specific encoding choices.
Context
#7754 represents position sets as empty, all, range, or explicit and count sets as empty, constant, or explicit. Explicit positions are converted to deltas before the resulting u64 array enters CompressionStrategy; explicit counts enter the same strategy directly. The metadata-only variants avoid structural buffers and let readers retain O(1) normalized forms.
The review in #7754 (comment) noted that these choices overlap with cascading compression. General delta and range encodings could simplify the sparse wire contract while also benefiting non-sparse integer and offset paths.
This follow-up does not block #7754. If it changes the Lance 2.3 SparseLayout wire contract, the change must land before 2.3 is declared stable; compatibility with draft 2.3 files is not required.
Requirements
Part of #7750.