You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0ef1b4b
Browse filesBrowse the repository at this point in the historyBrowse files
fix: keep extension types when skip_metadata clears Parquet field metadata (apache#26854)
## Which issue does this PR close?
- Closesapache#26851.
## Rationale for this change
With `datafusion.execution.parquet.skip_metadata = true` (the default),
a Parquet column with an Arrow extension type (`arrow.uuid`,
`arrow.json`, `geoarrow.*`, `arrow.parquet.variant`, ...) is read as its
plain storage type. Extension-aware UDFs and type planners then see a
plain `FixedSizeBinary` / `Binary` / `Struct`. This happens for any
number of files. See apache#26851 for a reproducer.
The extension type is part of the type of a column, not a writer
annotation, so `skip_metadata` should not remove it. This is the fix
proposed in the Variant epic apache#26517.
## What changes are included in this PR?
`clear_metadata` in `datafusion/datasource-parquet/src/file_format.rs`
keeps the `ARROW:extension:name` and `ARROW:extension:metadata` keys of
each field and clears the rest of the field metadata and the schema
metadata, as before. Nested fields are not affected: their metadata was
never cleared.
## What is the testing strategy for this PR?
The new test `schema_merge_keeps_extension_types_by_default` in
`datafusion/core/tests/parquet/schema.rs` reads three files with default
options:
- two files have the same extension type, but the rest of their field
metadata conflicts;
- one file has no extension type.
The test checks that the merged schema keeps only the extension type
keys. It fails without the change.
The test checks the inferred schema, not `arrow_metadata(col)` in SQL.
At execution time, the Parquet scan produces columns without their field
metadata, so `arrow_metadata(col)` over a Parquet table returns `{}`
even with this fix. That is a separate bug, also present with
`skip_metadata = false` on 55.0.0, filed as apache#26855.
## Are there any user-facing changes?
Yes, Parquet files with extension-typed columns are now read with their
extension type by default:
- Plans and type coercion over such columns may change.
- If files disagree on `ARROW:extension:metadata` for the same column,
schema inference now fails with `Fail to merge field ... due to
conflicting metadata`. Before, it dropped the extension type. If a
column has an extension type in some files only, the whole column gets
it. See the discussion in apache#26851.
There are no public API changes.
datafusion.execution.parquet.row_group_range_assignment start_offset (reading) Which byte range of a split file reads each row group. `start_offset` picks the range containing the row group's start. `midpoint` picks the range containing its midpoint, as Spark does, which spreads large row groups more evenly across ranges.
426
426
datafusion.execution.parquet.schema_force_view_types true (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`, and `Binary/BinaryLarge` with `BinaryView`. The parquet reader is optimized for reading `Utf8View` and `BinaryView`, so such queries are significantly faster than reading `Utf8`/`Binary` and then casting to the view types.
427
427
datafusion.execution.parquet.skip_arrow_metadata false (writing) Skip encoding the embedded arrow metadata in the KV_meta This is analogous to the `ArrowWriterOptions::with_skip_arrow_metadata`. Refer to <https://docs.rs/parquet/53.3.0/parquet/arrow/arrow_writer/struct.ArrowWriterOptions.html#method.with_skip_arrow_metadata>
428
-
datafusion.execution.parquet.skip_metadata true (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata
428
+
datafusion.execution.parquet.skip_metadata true (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata. Arrow extension types (the `ARROW:extension:name` and `ARROW:extension:metadata` field metadata) are kept, because they are part of the type of a column
429
429
datafusion.execution.parquet.statistics_enabled page (writing) Sets if statistics are enabled for any column Valid values are: "none", "chunk", and "page" These values are not case sensitive. If NULL, uses default parquet writer setting
Copy file name to clipboardExpand all lines: docs/source/user-guide/configs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,7 +83,7 @@ The following configuration settings are available:
83
83
| datafusion.execution.time_zone | NULL | The default time zone Some functions, e.g. `now`, return timestamps in this time zone. This is also used to interpret timezone-naive timestamps in comparisons and subtraction with timezone-aware timestamps. |
84
84
| datafusion.execution.parquet.enable_page_index | true | (reading) If true, reads the Parquet data page level metadata (the Page Index), if present, to reduce the I/O and number of rows decoded. |
85
85
| datafusion.execution.parquet.pruning | true | (reading) If true, the parquet reader attempts to skip entire row groups based on the predicate in the query and the metadata (min/max values) stored in the parquet file |
86
-
| datafusion.execution.parquet.skip_metadata | true | (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata |
86
+
| datafusion.execution.parquet.skip_metadata | true | (reading) If true, the parquet reader skip the optional embedded metadata that may be in the file Schema. This setting can help avoid schema conflicts when querying multiple parquet files with schemas containing compatible types but different metadata. Arrow extension types (the `ARROW:extension:name` and `ARROW:extension:metadata` field metadata) are kept, because they are part of the type of a column |
87
87
| datafusion.execution.parquet.metadata_size_hint | 524288 | (reading) If specified, the parquet reader will try and fetch the last `size_hint` bytes of the parquet file optimistically. If not specified, two reads are required: One read to fetch the 8-byte parquet footer and another to fetch the metadata length encoded in the footer Default setting to 512 KiB, which should be sufficient for most parquet files, it can reduce one I/O operation per parquet file. If the metadata is larger than the hint, two reads will still be performed. |
88
88
| datafusion.execution.parquet.pushdown_filters | false | (reading) If true, filter expressions are be applied during the parquet decoding operation to reduce the number of rows decoded. This optimization is sometimes called "late materialization". |
89
89
| datafusion.execution.parquet.reorder_filters | false | (reading) If true, filter expressions evaluated during the parquet decoding operation will be reordered heuristically to minimize the cost of evaluation. If false, the filters are applied in the same order as written in the query |
0 commit comments