Skip to content

Commit 0ef1b4b

Browse files
authored
fix: keep extension types when skip_metadata clears Parquet field metadata (apache#26854)
## Which issue does this PR close? - Closes apache#26851. ## Rationale for this change With `datafusion.execution.parquet.skip_metadata = true` (the default), a Parquet column with an Arrow extension type (`arrow.uuid`, `arrow.json`, `geoarrow.*`, `arrow.parquet.variant`, ...) is read as its plain storage type. Extension-aware UDFs and type planners then see a plain `FixedSizeBinary` / `Binary` / `Struct`. This happens for any number of files. See apache#26851 for a reproducer. The extension type is part of the type of a column, not a writer annotation, so `skip_metadata` should not remove it. This is the fix proposed in the Variant epic apache#26517. ## What changes are included in this PR? `clear_metadata` in `datafusion/datasource-parquet/src/file_format.rs` keeps the `ARROW:extension:name` and `ARROW:extension:metadata` keys of each field and clears the rest of the field metadata and the schema metadata, as before. Nested fields are not affected: their metadata was never cleared. ## What is the testing strategy for this PR? The new test `schema_merge_keeps_extension_types_by_default` in `datafusion/core/tests/parquet/schema.rs` reads three files with default options: - two files have the same extension type, but the rest of their field metadata conflicts; - one file has no extension type. The test checks that the merged schema keeps only the extension type keys. It fails without the change. The test checks the inferred schema, not `arrow_metadata(col)` in SQL. At execution time, the Parquet scan produces columns without their field metadata, so `arrow_metadata(col)` over a Parquet table returns `{}` even with this fix. That is a separate bug, also present with `skip_metadata = false` on 55.0.0, filed as apache#26855. ## Are there any user-facing changes? Yes, Parquet files with extension-typed columns are now read with their extension type by default: - Plans and type coercion over such columns may change. - If files disagree on `ARROW:extension:metadata` for the same column, schema inference now fails with `Fail to merge field ... due to conflicting metadata`. Before, it dropped the extension type. If a column has an extension type in some files only, the whole column gets it. See the discussion in apache#26851. There are no public API changes.
1 parent 9411463 commit 0ef1b4b

5 files changed

Lines changed: 64 additions & 7 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1325,7 +1325,9 @@ config_namespace! {
13251325

13261326
/// (reading) If true, the parquet reader skip the optional embedded metadata that may be in
13271327
/// the file Schema. This setting can help avoid schema conflicts when querying
1328-
/// multiple parquet files with schemas containing compatible types but different metadata
1328+
/// multiple parquet files with schemas containing compatible types but different metadata.
1329+
/// Arrow extension types (the `ARROW:extension:name` and `ARROW:extension:metadata`
1330+
/// field metadata) are kept, because they are part of the type of a column
13291331
pub skip_metadata: bool, default = true
13301332

13311333
/// (reading) If specified, the parquet reader will try and fetch the last `size_hint`

‎datafusion/datasource-parquet/src/file_format.rs‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ pub use crate::schema_coercion::{
2121
pub use crate::sink::ParquetSink;
2222

2323
use arrow::datatypes::{DataType, FieldRef, Fields, Metadata, Schema, SchemaRef};
24+
use arrow_schema::extension::{EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY};
2425
use datafusion_datasource::TableSchema;
2526
use datafusion_datasource::file_compression_type::FileCompressionType;
2627
use datafusion_datasource::file_sink_config::FileSinkConfig;
@@ -244,7 +245,8 @@ impl ParquetFormat {
244245
}
245246

246247
/// Clears all metadata (Schema level and field level) on an iterator
247-
/// of Schemas
248+
/// of Schemas, except for the extension type of the fields, which is part
249+
/// of the type of a field.
248250
fn clear_metadata(
249251
schemas: impl IntoIterator<Item = Schema>,
250252
) -> impl Iterator<Item = Schema> {
@@ -253,7 +255,16 @@ fn clear_metadata(
253255
.fields()
254256
.iter()
255257
.map(|field| {
256-
field.as_ref().clone().with_metadata(Metadata::new()) // clear meta
258+
let metadata = field
259+
.metadata()
260+
.iter()
261+
.filter(|(key, _)| {
262+
key.as_str() == EXTENSION_TYPE_NAME_KEY
263+
|| key.as_str() == EXTENSION_TYPE_METADATA_KEY
264+
})
265+
.map(|(key, value)| (key.clone(), value.clone()))
266+
.collect::<Metadata>();
267+
field.as_ref().clone().with_metadata(metadata)
257268
})
258269
.collect::<Fields>();
259270
Schema::new(fields)

‎datafusion/sqllogictest/test_files/cast_extension_type_metadata.slt‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,8 @@ FROM (
106106
00010203040506070809000102030507 arrow.uuid 00010203040506070809000102030506 arrow.uuid
107107

108108
# Extension types of Parquet columns are visible to expressions evaluated in the scan,
109-
# also when the scan replaces a constant column with a literal from the statistics
109+
# also when the scan replaces a constant column with a literal from the statistics.
110+
# skip_metadata = true clears field metadata but keeps extension types.
110111
statement ok
111112
COPY (
112113
SELECT CAST(arrow_cast(X'00010203040506070809000102030506', 'FixedSizeBinary(16)') AS UUID) AS uuid_val
@@ -115,7 +116,7 @@ TO 'test_files/scratch/cast_extension_type_metadata/uuid.parquet'
115116
STORED AS PARQUET;
116117

117118
statement ok
118-
SET datafusion.execution.parquet.skip_metadata = false;
119+
SET datafusion.execution.parquet.skip_metadata = true;
119120

120121
statement ok
121122
CREATE EXTERNAL TABLE uuid_parquet
@@ -132,3 +133,46 @@ DROP TABLE uuid_parquet;
132133

133134
statement ok
134135
RESET datafusion.execution.parquet.skip_metadata;
136+
137+
# Schema inference over several files with skip_metadata = true (the default) keeps the
138+
# extension type and clears the other field metadata, even if it conflicts between files.
139+
# A file without the extension type does not remove it.
140+
statement ok
141+
COPY (
142+
SELECT with_metadata(
143+
with_metadata(with_metadata(1, 'ARROW:extension:name', 'example.ext'), 'ARROW:extension:metadata', '{"width":5}'),
144+
'other', 'bar'
145+
) AS id
146+
)
147+
TO 'test_files/scratch/cast_extension_type_metadata/merge/1.parquet'
148+
STORED AS PARQUET;
149+
150+
statement ok
151+
COPY (
152+
SELECT with_metadata(
153+
with_metadata(with_metadata(2, 'ARROW:extension:name', 'example.ext'), 'ARROW:extension:metadata', '{"width":5}'),
154+
'other', 'baz'
155+
) AS id
156+
)
157+
TO 'test_files/scratch/cast_extension_type_metadata/merge/2.parquet'
158+
STORED AS PARQUET;
159+
160+
statement ok
161+
COPY (SELECT 3 AS id)
162+
TO 'test_files/scratch/cast_extension_type_metadata/merge/3.parquet'
163+
STORED AS PARQUET;
164+
165+
statement ok
166+
CREATE EXTERNAL TABLE merged
167+
STORED AS PARQUET
168+
LOCATION 'test_files/scratch/cast_extension_type_metadata/merge/';
169+
170+
query I?
171+
SELECT id, arrow_metadata(id) FROM merged ORDER BY id;
172+
----
173+
1 {ARROW:extension:metadata: {"width":5}, ARROW:extension:name: example.ext}
174+
2 {ARROW:extension:metadata: {"width":5}, ARROW:extension:name: example.ext}
175+
3 {ARROW:extension:metadata: {"width":5}, ARROW:extension:name: example.ext}
176+
177+
statement ok
178+
DROP TABLE merged;

‎datafusion/sqllogictest/test_files/information_schema.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -425,7 +425,7 @@ datafusion.execution.parquet.reorder_filters false (reading) If true, filter exp
425425
datafusion.execution.parquet.row_group_range_assignment start_offset (reading) Which byte range of a split file reads each row group. `start_offset` picks the range containing the row group's start. `midpoint` picks the range containing its midpoint, as Spark does, which spreads large row groups more evenly across ranges.
426426
datafusion.execution.parquet.schema_force_view_types true (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`, and `Binary/BinaryLarge` with `BinaryView`. The parquet reader is optimized for reading `Utf8View` and `BinaryView`, so such queries are significantly faster than reading `Utf8`/`Binary` and then casting to the view types.
427427
datafusion.execution.parquet.skip_arrow_metadata false (writing) Skip encoding the embedded arrow metadata in the KV_meta This is analogous to the `ArrowWriterOptions::with_skip_arrow_metadata`. Refer to <https://docs.rs/parquet/53.3.0/parquet/arrow/arrow_writer/struct.ArrowWriterOptions.html#method.with_skip_arrow_metadata>
428-
datafusion.execution.parquet.skip_metadata true (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata
428+
datafusion.execution.parquet.skip_metadata true (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata. Arrow extension types (the `ARROW:extension:name` and `ARROW:extension:metadata` field metadata) are kept, because they are part of the type of a column
429429
datafusion.execution.parquet.statistics_enabled page (writing) Sets if statistics are enabled for any column Valid values are: "none", "chunk", and "page" These values are not case sensitive. If NULL, uses default parquet writer setting
430430
datafusion.execution.parquet.statistics_truncate_length 64 (writing) Sets statistics truncate length. If NULL, uses default parquet writer setting
431431
datafusion.execution.parquet.write_batch_size 1024 (writing) Sets write_batch_size in rows

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ The following configuration settings are available:
8383
| datafusion.execution.time_zone | NULL | The default time zone Some functions, e.g. `now`, return timestamps in this time zone. This is also used to interpret timezone-naive timestamps in comparisons and subtraction with timezone-aware timestamps. |
8484
| datafusion.execution.parquet.enable_page_index | true | (reading) If true, reads the Parquet data page level metadata (the Page Index), if present, to reduce the I/O and number of rows decoded. |
8585
| datafusion.execution.parquet.pruning | true | (reading) If true, the parquet reader attempts to skip entire row groups based on the predicate in the query and the metadata (min/max values) stored in the parquet file |
86-
| datafusion.execution.parquet.skip_metadata | true | (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata |
86+
| datafusion.execution.parquet.skip_metadata | true | (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata. Arrow extension types (the `ARROW:extension:name` and `ARROW:extension:metadata` field metadata) are kept, because they are part of the type of a column |
8787
| datafusion.execution.parquet.metadata_size_hint | 524288 | (reading) If specified, the parquet reader will try and fetch the last `size_hint` bytes of the parquet file optimistically. If not specified, two reads are required: One read to fetch the 8-byte parquet footer and another to fetch the metadata length encoded in the footer Default setting to 512 KiB, which should be sufficient for most parquet files, it can reduce one I/O operation per parquet file. If the metadata is larger than the hint, two reads will still be performed. |
8888
| datafusion.execution.parquet.pushdown_filters | false | (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization". |
8989
| datafusion.execution.parquet.reorder_filters | false | (reading) If true, filter expressions evaluated during the parquet decoding operation will be reordered heuristically to minimize the cost of evaluation. If false, the filters are applied in the same order as written in the query |

0 commit comments

Comments
 (0)