Skip to content

[C++][Parquet] Fall back to PLAIN when ALP is larger, and simplify ALP preset sampling #55579

Open

Description

@prtkgaur

Describe the enhancement requested

Pages of values that have no decimal form, such as full-precision measurements, can come out
larger under ALP than under PLAIN. In that case the writer should write the page as PLAIN
instead, and record each page's encoding in its page header and the column chunk metadata.
Readers already pick a decoder per page, so no reader change is needed.

Alongside that, I'd like to simplify how the ALP writer samples a page to build its preset.
Sampling has two jobs:

  • pick spots spread evenly through the page,
  • and take enough values at each spot to be representative.

Today the first is done by cutting the page into fixed-size chunks and taking every few chunks
on a stride derived from a fixed interval, and the second by sampling again inside each chosen
chunk, reading only its start, with a separate rule for short chunks.

Both can be simpler: pick whole vectors at even spacing across the page, and take a small number
of values spread across each vector, the way the ALP paper does. On the columns I measured this
gave equal or smaller output and made the preset cheaper to build.

Another simplification that can be done is that the per-vector search can also drop its early exit,
since with a five-candidate preset it never triggers.

Please note that neither change touches the file format or the decoder.

Component(s)

C++

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions