Skip to content

Commit 2f4d087

Browse files
authored
feat: support ALP parquet encoding in writer options (apache#27036)
Add ALP to DataFusion's parquet encoding configuration, conversion, documentation, protobuf round-trip tests, and SQL writer coverage. > [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#26936 ## Rationale for this change DataFusion could read ALP-encoded Parquet files but did not allow users to select ALP when writing files. ## What changes are included in this PR? - Add `alp` to `DFParquetEncoding` - Support ALP through `SET datafusion.execution.parquet.encoding` - Support ALP through `format.encoding` - Update configuration and SQL documentation - Add parser, conversion, protobuf, and SQLLogicTest coverage - Verify writing and reading ALP-encoded `DOUBLE` columns ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 7a3cab8 commit 2f4d087

8 files changed

Lines changed: 48 additions & 9 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1512,7 +1512,7 @@ config_namespace! {
15121512
/// (writing) Sets default encoding for any column.
15131513
/// Valid values are: plain, plain_dictionary, rle,
15141514
/// bit_packed, delta_binary_packed, delta_length_byte_array,
1515-
/// delta_byte_array, rle_dictionary, and byte_stream_split.
1515+
/// delta_byte_array, rle_dictionary, byte_stream_split, and alp.
15161516
/// These values are not case sensitive. If NULL, uses
15171517
/// default parquet writer setting
15181518
pub encoding: Option<DFParquetEncoding>, default = None
@@ -3509,7 +3509,7 @@ config_namespace_with_hashmap! {
35093509
/// Sets encoding for the column path.
35103510
/// Valid values are: plain, plain_dictionary, rle,
35113511
/// bit_packed, delta_binary_packed, delta_length_byte_array,
3512-
/// delta_byte_array, rle_dictionary, and byte_stream_split.
3512+
/// delta_byte_array, rle_dictionary, byte_stream_split, and alp.
35133513
/// These values are not case-sensitive. If NULL, uses
35143514
/// default parquet options
35153515
pub encoding: Option<String>, default = None

‎datafusion/common/src/parquet_config.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,8 @@ pub enum DFParquetEncoding {
285285
RleDictionary,
286286
/// Byte stream split encoding
287287
ByteStreamSplit,
288+
/// Adaptive Lossless Floating-Point Encoding
289+
Alp,
288290
}
289291

290292
impl FromStr for DFParquetEncoding {
@@ -301,11 +303,12 @@ impl FromStr for DFParquetEncoding {
301303
"delta_byte_array" => Ok(Self::DeltaByteArray),
302304
"rle_dictionary" => Ok(Self::RleDictionary),
303305
"byte_stream_split" => Ok(Self::ByteStreamSplit),
306+
"alp" => Ok(Self::Alp),
304307
_ => Err(DataFusionError::Configuration(format!(
305308
"Unknown or unsupported parquet encoding: \
306309
{s}. Valid values are: plain, plain_dictionary, rle, \
307310
bit_packed, delta_binary_packed, delta_length_byte_array, \
308-
delta_byte_array, rle_dictionary, and byte_stream_split."
311+
delta_byte_array, rle_dictionary, byte_stream_split, and alp."
309312
))),
310313
}
311314
}
@@ -323,6 +326,7 @@ impl Display for DFParquetEncoding {
323326
Self::DeltaByteArray => "delta_byte_array",
324327
Self::RleDictionary => "rle_dictionary",
325328
Self::ByteStreamSplit => "byte_stream_split",
329+
Self::Alp => "alp",
326330
})
327331
}
328332
}
@@ -374,6 +378,7 @@ impl From<DFParquetEncoding> for parquet::basic::Encoding {
374378
DFParquetEncoding::DeltaByteArray => Self::DELTA_BYTE_ARRAY,
375379
DFParquetEncoding::RleDictionary => Self::RLE_DICTIONARY,
376380
DFParquetEncoding::ByteStreamSplit => Self::BYTE_STREAM_SPLIT,
381+
DFParquetEncoding::Alp => Self::ALP,
377382
}
378383
}
379384
}
@@ -737,6 +742,7 @@ mod tests {
737742
DFParquetEncoding::ByteStreamSplit,
738743
Encoding::BYTE_STREAM_SPLIT,
739744
),
745+
("alp", DFParquetEncoding::Alp, Encoding::ALP),
740746
] {
741747
let parsed: DFParquetEncoding = input.parse().unwrap();
742748
assert_eq!(parsed, expected, "input: {input}");

‎datafusion/proto-common/src/from_proto/mod.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1723,7 +1723,11 @@ mod tests {
17231723

17241724
#[test]
17251725
fn test_parquet_encoding_round_trip() {
1726-
for encoding in [None, Some(DFParquetEncoding::ByteStreamSplit)] {
1726+
for encoding in [
1727+
None,
1728+
Some(DFParquetEncoding::ByteStreamSplit),
1729+
Some(DFParquetEncoding::Alp),
1730+
] {
17271731
let opts = ParquetOptions {
17281732
encoding,
17291733
..ParquetOptions::default()

‎datafusion/sqllogictest/test_files/copy.slt‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -387,6 +387,27 @@ OPTIONS (
387387
----
388388
2
389389

390+
# ALP encoding is accepted for floating-point columns.
391+
query I
392+
COPY (
393+
VALUES (CAST(1.25 AS DOUBLE)), (CAST(2.5 AS DOUBLE)), (CAST(3.75 AS DOUBLE))
394+
)
395+
TO 'test_files/scratch/copy/alp/' STORED AS PARQUET
396+
OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' false);
397+
----
398+
3
399+
400+
statement ok
401+
CREATE EXTERNAL TABLE validate_alp
402+
STORED AS PARQUET LOCATION 'test_files/scratch/copy/alp/';
403+
404+
query R rowsort
405+
SELECT column1 FROM validate_alp;
406+
----
407+
1.25
408+
2.5
409+
3.75
410+
390411
# valid vs invalid metadata
391412

392413
# accepts map with a single entry

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -410,7 +410,7 @@ datafusion.execution.parquet.dictionary_enabled true (writing) Sets if dictionar
410410
datafusion.execution.parquet.dictionary_page_size_limit 1048576 (writing) Sets best effort maximum dictionary page size, in bytes
411411
datafusion.execution.parquet.enable_page_index true (reading) If true, reads the Parquet data page level metadata (the Page Index), if present, to reduce the I/O and number of rows decoded.
412412
datafusion.execution.parquet.enable_rle_to_dictionary false (reading) If true, top-level string and binary Parquet columns with dictionary pages are inferred and scanned as `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>` instead of their plain value type. This applies only when DataFusion infers the table schema. Tables with a user-supplied schema are not promoted because the Parquet footer is not read at DDL time, so dictionary pages cannot be detected per column. See <https://github.com/apache/datafusion/issues/24112>
413-
datafusion.execution.parquet.encoding NULL (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split. These values are not case sensitive. If NULL, uses default parquet writer setting
413+
datafusion.execution.parquet.encoding NULL (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp. These values are not case sensitive. If NULL, uses default parquet writer setting
414414
datafusion.execution.parquet.force_filter_selections false (reading) Force the use of RowSelections for filter results, when pushdown_filters is enabled. If false, the reader will automatically choose between a RowSelection and a Bitmap based on the number and pattern of selected rows.
415415
datafusion.execution.parquet.max_in_list_size 20 Maximum number of input values in an `IN (...)` list eligible for min/max pruning. Lists above this cap, or a cap of 0, skip this rewrite; other predicates and Bloom-filter pruning remain available. Within the cap, nonempty lists of at most 20 values use the existing per-value rewrite. Larger literal lists use a compact representation when the column type is string, variable-length binary, integer, decimal, date, time, timestamp, or duration. This applies to both `IN` and `NOT IN`, including lists with NULL members. `NOT IN` with NULL and all-NULL `IN` lists cannot match any rows. Compact lists containing NULL do not use the fully-matched-row-group optimization. Floating-point and other lists retain the existing per-value rewrite, so raising the cap can make those predicates expensive to build and evaluate. Defaults to 20.
416416
datafusion.execution.parquet.max_predicate_cache_size NULL (reading) The maximum predicate cache size, in bytes. When `pushdown_filters` is enabled, sets the maximum memory used to cache the results of predicate evaluation between filter evaluation and output generation. Decreasing this value will reduce memory usage, but may increase IO and CPU usage. None means use the default parquet reader setting. 0 means no caching.

‎datafusion/sqllogictest/test_files/set_variable.slt‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -754,18 +754,26 @@ SHOW datafusion.execution.parquet.encoding
754754
----
755755
datafusion.execution.parquet.encoding plain
756756

757+
statement ok
758+
SET datafusion.execution.parquet.encoding = 'ALP'
759+
760+
query TT
761+
SHOW datafusion.execution.parquet.encoding
762+
----
763+
datafusion.execution.parquet.encoding alp
764+
757765
statement error
758766
SET datafusion.execution.parquet.encoding = 'invalid'
759767
----
760768
DataFusion error: Error setting config datafusion.execution.parquet.encoding
761769
caused by
762-
Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: invalid. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split.
770+
Invalid or Unsupported Configuration: Unknown or unsupported parquet encoding: invalid. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp.
763771

764772

765773
query TT
766774
SHOW datafusion.execution.parquet.encoding
767775
----
768-
datafusion.execution.parquet.encoding plain
776+
datafusion.execution.parquet.encoding alp
769777

770778
statement ok
771779
RESET datafusion.execution.parquet.encoding

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ The following configuration settings are available:
111111
| datafusion.execution.parquet.column_index_truncate_length | 64 | (writing) Sets column index truncate length |
112112
| datafusion.execution.parquet.statistics_truncate_length | 64 | (writing) Sets statistics truncate length. If NULL, uses default parquet writer setting |
113113
| datafusion.execution.parquet.data_page_row_count_limit | 20000 | (writing) Sets best effort maximum number of rows in data page |
114-
| datafusion.execution.parquet.encoding | NULL | (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, and byte_stream_split. These values are not case sensitive. If NULL, uses default parquet writer setting |
114+
| datafusion.execution.parquet.encoding | NULL | (writing) Sets default encoding for any column. Valid values are: plain, plain_dictionary, rle, bit_packed, delta_binary_packed, delta_length_byte_array, delta_byte_array, rle_dictionary, byte_stream_split, and alp. These values are not case sensitive. If NULL, uses default parquet writer setting |
115115
| datafusion.execution.parquet.bloom_filter_on_write | false | (writing) Write bloom filters for all columns when creating parquet files |
116116
| datafusion.execution.parquet.bloom_filter_fpp | NULL | (writing) Sets bloom filter false positive probability. If NULL, uses default parquet writer setting |
117117
| datafusion.execution.parquet.bloom_filter_ndv | NULL | (writing) Sets bloom filter number of distinct values. If NULL, uses default parquet writer setting |

0 commit comments

Comments
 (0)