You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support the Parquet ALP encoding in the parquet writer options (format.encoding / datafusion.execution.parquet.encoding) #26936
Is your feature request related to a problem or challenge?
Parquet 2.14.0 added the ALP encoding for FLOAT / DOUBLE columns (see the ALP blog post), and the parquet crate reads and writes it since arrow-rs 60.0.0 (apache/arrow-rs#9372). DataFusion main is on arrow/parquet 60, so it can already read ALP encoded files, but there is no way to write them from SQL or the config:
I want to write to a file like this
COPY (SELECT round(68+15* sin(x /5000.0) + random(), 2) AS temp_f FROM generate_series(1, 1000000) AS t(x))
TO 'readings_alp.parquet'
OPTIONS ('format.encoding''alp', 'format.dictionary_enabled''false');
Note the "alp"
This results in an error:
Error: Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: alp.
Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array,
delta_byte_array, rle_dictionary, and byte_stream_split.
The same happens with SET datafusion.execution.parquet.encoding = 'alp'.
The cause is that DFParquetEncoding in datafusion/common/src/parquet_config.rs has no Alp variant, even though parquet::basic::Encoding::ALP exists and "alp".parse::<Encoding>() works.
Describe the solution you'd like
The command should work and produce ALP encoded files
Describe alternatives you've considered
Add an Alp variant to DFParquetEncoding with the FromStr / Display / From<DFParquetEncoding> for parquet::basic::Encoding plumbing, so 'alp' is accepted by datafusion.execution.parquet.encoding and the per-column format.encoding options.
Mention it in the config docs (docs/source/user-guide/configs.md) and the parquet writer option docs.
Add a sqllogictest that writes a DOUBLE column with 'format.encoding' 'alp' and checks parquet_metadata(...) reports ALP in encodings.
Thank you @Surajsuthar -- I am particularly interested in this feature, so please tag me on the PR when you are done. It should be fairly straightforward
Activity
Surajsuthar commentedon Nov 2, 2026
take
alamb commentedon Nov 2, 2026
Thank you @Surajsuthar -- I am particularly interested in this feature, so please tag me on the PR when you are done. It should be fairly straightforward
feat: support ALP parquet encoding in writer options (apache#27036)
[branch-56] feat: support ALP parquet encoding in writer options (bac…