Skip to content

Commit fde7e46

Browse files
Rich-T-kidOmega359
authored andcommitted
introduce optional rle reads from parquet (apache#24227)
## a large portion of this PR is test! ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#24112. - part of apache#24111 ## Rationale for this change When DataFusion reads a parquet file with dictionary-encoded string or binary columns, it currently decodes the dictionary and returns plain Utf8/Binary arrays, discarding the encoding. For low-cardinality columns (status, country, category, etc.) this doesn't take full advantage of the compacted format parquet gives the engine Preserving the dictionary encoding reduces memory usage and can improve aggregation performance on these columns. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> ## What changes are included in this PR? Adds `datafusion.execution.parquet.enable_rle_to_dictionary` with a default of `false`. When enabled for inferred-schema Parquet tables, DataFusion inspects Parquet footer metadata and promotes top-level string/binary columns with dictionary pages to Arrow dictionary types. Mixed dictionary/plain files are normalized before schema merge when the value types are compatible. At scan time, the parquet opener passes the promoted schema to arrow-rs so those columns can be read as dictionary arrays directly. *Tables with a user-supplied schema are not promoted because DataFusion does not use footer metadata to infer their schema.* <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? yes. - `datafusion/sqllogictest/test_files/parquet_rle_to_dictionary.slt` - `datafusion/datasource-parquet/src/schema_coercion.rs` - `uniform_dict_schemas_respects_value_type_families` checks that mixed file schemas are normalized only across compatible string/binary families. - `rle_schema_coercion_respects_dictionary_value_type` checks scan-time coercion into dictionary types, including incompatible cases and the flag-off path. - `datafusion/datasource-parquet/src/opener/mod.rs` - `test_rle_binary_column_promotion` verifies the opener passes a promoted `Dictionary(Int32, Binary)` schema to arrow-rs so binary columns can be read as dictionary arrays directly. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? New session config option: `SET datafusion.execution.parquet.enable_rle_to_dictionary = true`. Default is false so existing behavior is unchanged. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent 7382445 commit fde7e46

19 files changed

Lines changed: 1238 additions & 23 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1426,6 +1426,17 @@ config_namespace! {
14261426
/// Defaults to 20.
14271427
pub max_in_list_size: usize, default = 20
14281428

1429+
/// (reading) If true, top-level string and binary Parquet columns with
1430+
/// dictionary pages are inferred and scanned as
1431+
/// `Dictionary<Int32, Utf8>` / `Dictionary<Int32, Binary>` instead of
1432+
/// their plain value type.
1433+
///
1434+
/// This applies only when DataFusion infers the table schema. Tables with
1435+
/// a user-supplied schema are not promoted because the Parquet footer is
1436+
/// not read at DDL time, so dictionary pages cannot be detected per column.
1437+
/// See <https://github.com/apache/datafusion/issues/24112>
1438+
pub enable_rle_to_dictionary: bool, default = false
1439+
14291440
/// (reading) Which byte range of a split file reads each row group.
14301441
/// `start_offset` picks the range containing the row group's start.
14311442
/// `midpoint` picks the range containing its midpoint, as Spark does,

‎datafusion/common/src/file_options/parquet_writer.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,7 @@ impl ParquetOptions {
249249
skip_arrow_metadata: _,
250250
max_predicate_cache_size: _,
251251
max_in_list_size: _,
252+
enable_rle_to_dictionary: _,
252253
row_group_range_assignment: _,
253254
} = self;
254255

@@ -429,6 +430,7 @@ mod tests {
429430
coerce_int96_tz: None,
430431
max_predicate_cache_size: defaults.max_predicate_cache_size,
431432
content_defined_chunking: defaults.content_defined_chunking.clone(),
433+
enable_rle_to_dictionary: defaults.enable_rle_to_dictionary,
432434
}
433435
}
434436

@@ -558,6 +560,8 @@ mod tests {
558560
coerce_int96: None,
559561
coerce_int96_tz: None,
560562
content_defined_chunking: props.content_defined_chunking().into(),
563+
enable_rle_to_dictionary: global_options_defaults
564+
.enable_rle_to_dictionary,
561565
},
562566
column_specific_options,
563567
key_value_metadata,

‎datafusion/datasource-parquet/src/file_format.rs‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -356,14 +356,16 @@ impl FileFormat for ParquetFormat {
356356
&object.location,
357357
)
358358
.await?;
359-
let result = DFParquetMetadata::new(store.as_ref(), object)
359+
let meta = DFParquetMetadata::new(store.as_ref(), object)
360360
.with_metadata_size_hint(self.metadata_size_hint())
361361
.with_decryption_properties(file_decryption_properties)
362362
.with_file_metadata_cache(Some(Arc::clone(&file_metadata_cache)))
363363
.with_coerce_int96(coerce_int96)
364364
.with_coerce_int96_tz(coerce_int96_tz.clone())
365-
.fetch_schema_with_location()
366-
.await?;
365+
.with_enable_rle_to_dictionary(
366+
self.options.global.enable_rle_to_dictionary,
367+
);
368+
let result = meta.fetch_schema_with_location().await?;
367369
Ok::<_, DataFusionError>(result)
368370
})
369371
.boxed() // Workaround https://github.com/rust-lang/rust/issues/64552
@@ -398,7 +400,12 @@ impl FileFormat for ParquetFormat {
398400
}
399401
drop(seen);
400402

401-
let schemas = schemas.into_iter().map(|(_, schema)| schema);
403+
// Normalize dict-promoted schemas before merging so mixed dict/plain files merge cleanly.
404+
let mut schemas: Vec<Schema> =
405+
schemas.into_iter().map(|(_, schema)| schema).collect();
406+
if self.options.global.enable_rle_to_dictionary {
407+
schemas = crate::schema_coercion::uniform_dict_schemas(schemas);
408+
}
402409

403410
let schema = if self.skip_metadata() {
404411
Schema::try_merge(clear_metadata(schemas))
@@ -520,7 +527,7 @@ impl FileFormat for ParquetFormat {
520527
source = source.with_parquet_file_reader_factory(cached_parquet_read_factory);
521528

522529
if let Some(metadata_size_hint) = metadata_size_hint {
523-
source = source.with_metadata_size_hint(metadata_size_hint)
530+
source = source.with_metadata_size_hint(metadata_size_hint);
524531
}
525532

526533
source = self.set_source_encryption_factory(source, state)?;
@@ -672,6 +679,7 @@ impl From<&ParquetFormatFactory> for protobuf::TableParquetOptions {
672679
compression_opt: global_options.global.compression.map(|compression| {
673680
parquet_options::CompressionOpt::Compression(compression.to_string())
674681
}),
682+
enable_rle_to_dictionary: global_options.global.enable_rle_to_dictionary,
675683
dictionary_enabled_opt: global_options.global.dictionary_enabled.map(|enabled| {
676684
parquet_options::DictionaryEnabledOpt::DictionaryEnabled(enabled)
677685
}),

‎datafusion/datasource-parquet/src/metadata.rs‎

Lines changed: 73 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,11 @@ use crate::{Int96Coercer, apply_file_schema_type_coercions};
2323
use arrow::array::{Array, ArrayRef, BooleanArray};
2424
use arrow::compute::kernels::cmp::eq;
2525
use arrow::compute::{and, sum};
26-
use arrow::datatypes::{DataType, Schema, SchemaRef, TimeUnit};
26+
use arrow::datatypes::{DataType, Field, Schema, SchemaRef, TimeUnit};
2727
use datafusion_common::encryption::FileDecryptionProperties;
2828
use datafusion_common::stats::Precision;
2929
use datafusion_common::{
30-
ColumnStatistics, DataFusionError, HashMap, Result, ScalarValue, Statistics,
30+
ColumnStatistics, DataFusionError, HashMap, HashSet, Result, ScalarValue, Statistics,
3131
internal_datafusion_err,
3232
};
3333
use datafusion_execution::cache::cache_manager::{
@@ -152,6 +152,8 @@ pub struct DFParquetMetadata<'a> {
152152
pub coerce_int96: Option<TimeUnit>,
153153
/// Optional timezone applied to INT96-coerced timestamps.
154154
pub coerce_int96_tz: Option<Arc<str>>,
155+
/// If true, promote string/binary columns with dictionary pages to `Dictionary(Int32, ...)`.
156+
enable_rle_to_dictionary: bool,
155157
}
156158

157159
impl<'a> DFParquetMetadata<'a> {
@@ -169,9 +171,16 @@ impl<'a> DFParquetMetadata<'a> {
169171
page_index_policy: None,
170172
coerce_int96: None,
171173
coerce_int96_tz: None,
174+
enable_rle_to_dictionary: false,
172175
}
173176
}
174177

178+
/// Promote string/binary columns with dictionary pages to `Dictionary(Int32, ...)`.
179+
pub fn with_enable_rle_to_dictionary(mut self, enable: bool) -> Self {
180+
self.enable_rle_to_dictionary = enable;
181+
self
182+
}
183+
175184
/// Set a hint for the number of trailing bytes to prefetch from the end
176185
/// of the file, equivalent to
177186
/// [`ParquetMetaDataReader::with_prefetch_hint`].
@@ -449,6 +458,68 @@ impl<'a> DFParquetMetadata<'a> {
449458
.coerce()
450459
})
451460
.unwrap_or(schema);
461+
462+
let schema = if self.enable_rle_to_dictionary {
463+
let schema_descr = file_metadata.schema_descr();
464+
// Top-level columns that have a dictionary page in at least one row group.
465+
let dict_cols: HashSet<String> = metadata
466+
.row_groups()
467+
.iter()
468+
.flat_map(|rg| {
469+
rg.columns()
470+
.iter()
471+
.enumerate()
472+
.filter_map(|(col_idx, col)| {
473+
col.dictionary_page_offset()?;
474+
let col_desc = schema_descr.column(col_idx);
475+
let parts = col_desc.path().parts();
476+
// Skip nested columns: their leaf name doesn't match the
477+
// Arrow top-level field name.
478+
(parts.len() == 1).then(|| parts[0].clone())
479+
})
480+
})
481+
.collect();
482+
if dict_cols.is_empty() {
483+
schema
484+
} else {
485+
let promoted: Vec<_> = schema
486+
.fields()
487+
.iter()
488+
.map(|field| {
489+
if !dict_cols.contains(field.name()) {
490+
return Arc::clone(field);
491+
}
492+
let dict_value_type = match field.data_type() {
493+
DataType::Utf8 => Some(DataType::Utf8),
494+
DataType::LargeUtf8 => Some(DataType::LargeUtf8),
495+
DataType::Binary => Some(DataType::Binary),
496+
DataType::LargeBinary => Some(DataType::LargeBinary),
497+
_ => None,
498+
};
499+
dict_value_type.map_or_else(
500+
|| Arc::clone(field),
501+
|value_type| {
502+
Arc::new(
503+
Field::new(
504+
field.name(),
505+
DataType::Dictionary(
506+
Box::new(DataType::Int32),
507+
Box::new(value_type),
508+
),
509+
field.is_nullable(),
510+
)
511+
.with_metadata(field.metadata().clone()),
512+
)
513+
},
514+
)
515+
})
516+
.collect();
517+
Schema::new_with_metadata(promoted, schema.metadata().clone())
518+
}
519+
} else {
520+
schema
521+
};
522+
452523
Ok(schema)
453524
}
454525

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 70 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ use crate::row_group_filter::{RowGroupAccessPlanFilter, row_group_in_range};
3636
use crate::{
3737
BloomFilterStatistics, Int96Coercer, ParquetAccessPlan, ParquetFileMetrics,
3838
ParquetFileReaderFactory, ParquetRowSelection, ParquetVirtualColumn, RowGroupAccess,
39-
apply_file_schema_type_coercions,
39+
schema_coercion::apply_file_schema_type_coercions_with_rle,
4040
};
4141
use arrow::array::RecordBatch;
4242
use arrow::datatypes::DataType;
@@ -300,6 +300,8 @@ pub(super) struct ParquetMorselizer {
300300
/// lists skip container-level pruning. Sourced from
301301
/// `datafusion.execution.parquet.max_in_list_size`.
302302
pub max_in_list_size: usize,
303+
/// Whether to ask arrow-rs to read promoted dictionary columns directly.
304+
pub enable_rle_to_dictionary: bool,
303305
/// How row groups are assigned to the byte ranges of a split file. Sourced
304306
/// from `datafusion.execution.parquet.row_group_range_assignment`.
305307
pub row_group_range_assignment: RowGroupRangeAssignment,
@@ -498,6 +500,7 @@ struct PreparedParquetOpen {
498500
predicate_creation_errors: Count,
499501
max_predicate_cache_size: Option<usize>,
500502
max_in_list_size: usize,
503+
enable_rle_to_dictionary: bool,
501504
row_group_range_assignment: RowGroupRangeAssignment,
502505
reverse_row_groups: bool,
503506
sort_order_for_reorder: Option<LexOrdering>,
@@ -1000,6 +1003,7 @@ impl ParquetMorselizer {
10001003
predicate_creation_errors,
10011004
max_predicate_cache_size: self.max_predicate_cache_size,
10021005
max_in_list_size: self.max_in_list_size,
1006+
enable_rle_to_dictionary: self.enable_rle_to_dictionary,
10031007
row_group_range_assignment: self.row_group_range_assignment,
10041008
reverse_row_groups: self.reverse_row_groups,
10051009
sort_order_for_reorder: self.sort_order_for_reorder.clone(),
@@ -1116,9 +1120,10 @@ impl MetadataLoadedParquetOpen {
11161120
// desired schema (for example if we want to instruct the parquet
11171121
// reader to read strings using Utf8View instead). Update if necessary
11181122
let mut metadata_dirty = false;
1119-
if let Some(merged) = apply_file_schema_type_coercions(
1123+
if let Some(merged) = apply_file_schema_type_coercions_with_rle(
11201124
&prepared.logical_file_schema,
11211125
&physical_file_schema,
1126+
prepared.enable_rle_to_dictionary,
11221127
) {
11231128
physical_file_schema = Arc::new(merged);
11241129
options = options.with_schema(Arc::clone(&physical_file_schema));
@@ -2224,6 +2229,7 @@ mod test {
22242229
coerce_int96: Option<TimeUnit>,
22252230
max_predicate_cache_size: Option<usize>,
22262231
max_in_list_size: usize,
2232+
enable_rle_to_dictionary: bool,
22272233
row_group_range_assignment: RowGroupRangeAssignment,
22282234
reverse_row_groups: bool,
22292235
preserve_order: bool,
@@ -2443,6 +2449,7 @@ mod test {
24432449
coerce_int96: None,
24442450
max_predicate_cache_size: None,
24452451
max_in_list_size: MAX_IN_LIST_SIZE,
2452+
enable_rle_to_dictionary: false,
24462453
row_group_range_assignment: RowGroupRangeAssignment::default(),
24472454
reverse_row_groups: false,
24482455
preserve_order: false,
@@ -2519,6 +2526,11 @@ mod test {
25192526
self
25202527
}
25212528

2529+
fn with_enable_rle_to_dictionary(mut self, enable: bool) -> Self {
2530+
self.enable_rle_to_dictionary = enable;
2531+
self
2532+
}
2533+
25222534
fn with_enable_bloom_filter(mut self, enable: bool) -> Self {
25232535
self.enable_bloom_filter = enable;
25242536
self
@@ -2643,6 +2655,7 @@ mod test {
26432655
encryption_factory: None,
26442656
max_predicate_cache_size: self.max_predicate_cache_size,
26452657
max_in_list_size: self.max_in_list_size,
2658+
enable_rle_to_dictionary: self.enable_rle_to_dictionary,
26462659
row_group_range_assignment: self.row_group_range_assignment,
26472660
reverse_row_groups: self.reverse_row_groups,
26482661
sort_order_for_reorder: None,
@@ -5955,4 +5968,59 @@ mod test {
59555968
assert_eq!(rows, 5);
59565969
}
59575970
}
5971+
5972+
async fn collect_batches(
5973+
morselizer: &ParquetMorselizer,
5974+
file: PartitionedFile,
5975+
) -> Vec<RecordBatch> {
5976+
let mut stream = open_file(morselizer, file).await.unwrap();
5977+
let mut batches = Vec::new();
5978+
while let Some(batch) = stream.next().await {
5979+
batches.push(batch.unwrap());
5980+
}
5981+
batches
5982+
}
5983+
5984+
// Proves the opener passes a promoted binary Dictionary schema to arrow-rs.
5985+
#[tokio::test]
5986+
async fn test_rle_binary_column_promotion() {
5987+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
5988+
let bin_schema = Arc::new(Schema::new(vec![Field::new(
5989+
"payload",
5990+
DataType::Binary,
5991+
true,
5992+
)]));
5993+
let values =
5994+
Arc::new(arrow::array::BinaryArray::from_vec(vec![b"a", b"b", b"a"]));
5995+
let batch = RecordBatch::try_new(Arc::clone(&bin_schema), vec![values]).unwrap();
5996+
let props = WriterProperties::builder()
5997+
.set_dictionary_enabled(true)
5998+
.build();
5999+
let bin_size = write_parquet_batches(
6000+
Arc::clone(&store),
6001+
"binary.parquet",
6002+
vec![batch],
6003+
Some(props),
6004+
)
6005+
.await;
6006+
let dict_bin_schema = Arc::new(Schema::new(vec![Field::new(
6007+
"payload",
6008+
DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Binary)),
6009+
true,
6010+
)]));
6011+
let morselizer = ParquetMorselizerBuilder::new()
6012+
.with_store(Arc::clone(&store))
6013+
.with_schema(Arc::clone(&dict_bin_schema))
6014+
.with_enable_rle_to_dictionary(true)
6015+
.build();
6016+
let batches = collect_batches(
6017+
&morselizer,
6018+
PartitionedFile::new("binary.parquet".to_string(), bin_size as u64),
6019+
)
6020+
.await;
6021+
assert_eq!(
6022+
batches[0].schema().field(0).data_type(),
6023+
&DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Binary))
6024+
);
6025+
}
59586026
}

0 commit comments

Comments
 (0)