Skip to content

Support cascading compression for sparse structural arrays #7886

Description

@Xuanwo

Goal

Use the shared cascading compression framework for sparse structural position and count arrays instead of growing a separate set of format-specific encoding choices.

Context

#7754 represents position sets as empty, all, range, or explicit and count sets as empty, constant, or explicit. Explicit positions are converted to deltas before the resulting u64 array enters CompressionStrategy; explicit counts enter the same strategy directly. The metadata-only variants avoid structural buffers and let readers retain O(1) normalized forms.

The review in #7754 (comment) noted that these choices overlap with cascading compression. General delta and range encodings could simplify the sparse wire contract while also benefiting non-sparse integer and offset paths.

This follow-up does not block #7754. If it changes the Lance 2.3 SparseLayout wire contract, the change must land before 2.3 is declared stable; compatibility with draft 2.3 files is not required.

Requirements

  • Add composable delta and range encodings to the block compression path.
  • Let cascading selection consider constant, delta, range, RLE, bitpacking, dictionary, general compression, and flat encoding where applicable.
  • Define whether sparse positions enter the cascade as absolute positions or deltas, including the cost model used to choose.
  • Preserve metadata-only or otherwise zero-payload representations for empty, all, range, and constant sequences.
  • Preserve O(1) normalized sparse plans for those representations instead of materializing large u64 arrays.
  • Keep malformed metadata and decoded ordering, bounds, cardinality, and overflow checks returning format errors without panics.
  • Compare wire complexity, encoded size, planning memory, and scan / point-read performance with the representation introduced by feat(format): read and write sparse structural pages #7754.
  • Exercise the generic encodings on representative non-sparse integer or offset paths so this is not a sparse-only codec subsystem.

Part of #7750.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-encodingEncoding, IO, file reader/writerA-formatOn-disk format: protos and format spec docsenhancementNew feature or requestformat-changeA change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo).

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions