Skip to content

Support the Parquet ALP encoding in the parquet writer options (format.encoding / datafusion.execution.parquet.encoding) #26936

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

Parquet 2.14.0 added the ALP encoding for FLOAT / DOUBLE columns (see the ALP blog post), and the parquet crate reads and writes it since arrow-rs 60.0.0 (apache/arrow-rs#9372). DataFusion main is on arrow/parquet 60, so it can already read ALP encoded files, but there is no way to write them from SQL or the config:

I want to write to a file like this

COPY (SELECT round(68 + 15 * sin(x / 5000.0) + random(), 2) AS temp_f FROM generate_series(1, 1000000) AS t(x))
TO 'readings_alp.parquet' 
OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' 'false');

Note the "alp"

This results in an error:

Error: Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: alp.
Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array,
delta_byte_array, rle_dictionary, and byte_stream_split.

The same happens with SET datafusion.execution.parquet.encoding = 'alp'.

The cause is that DFParquetEncoding in datafusion/common/src/parquet_config.rs has no Alp variant, even though parquet::basic::Encoding::ALP exists and "alp".parse::<Encoding>() works.

Describe the solution you'd like

The command should work and produce ALP encoded files

Describe alternatives you've considered

  1. Add an Alp variant to DFParquetEncoding with the FromStr / Display / From<DFParquetEncoding> for parquet::basic::Encoding plumbing, so 'alp' is accepted by datafusion.execution.parquet.encoding and the per-column format.encoding options.
  2. Mention it in the config docs (docs/source/user-guide/configs.md) and the parquet writer option docs.
  3. Add a sqllogictest that writes a DOUBLE column with 'format.encoding' 'alp' and checks parquet_metadata(...) reports ALP in encodings.

Additional context

Activity

  1. self-assigned this
    on Oct 30, 2026
  2. Surajsuthar commented on Nov 2, 2026

    @Surajsuthar
    Contributor

    take

  3. alamb commented on Nov 2, 2026

    @alamb
    ContributorAuthor

    Thank you @Surajsuthar -- I am particularly interested in this feature, so please tag me on the PR when you are done. It should be fairly straightforward

  4. added a commit that references this issue on Nov 3, 2026
    2f4d087
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions