You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5fabdb1
Browse filesBrowse the repository at this point in the historyBrowse files
fix: keep field metadata when the Parquet scan replaces columns with literals (apache#26856)
## Which issue does this PR close?
- Closesapache#26855.
## Rationale for this change
Expressions pushed into a Parquet scan, such as `arrow_metadata(col)` or
extension-aware UDFs, can see a column without its field metadata:
`SELECT arrow_metadata(u) FROM t` returns `{}` although the table schema
has `ARROW:extension:name` on `u`. See apache#26855 for a reproducer.
The Parquet opener replaces column references with literals for
partition columns and for columns that the file statistics prove
constant (`min == max`). The literals had no field metadata, so
expressions evaluated on them lost the extension type. This happens for
example for every column of a one-row file.
## What changes are included in this PR?
- New function `replace_columns_with_literals_keeping_metadata` in
`datafusion-physical-expr-adapter`, next to
`replace_columns_with_literals`. It takes the metadata of each literal
from the field with the same name in a given schema.
`DefaultPhysicalExprAdapter` already creates its literals for missing
columns the same way.
- The Parquet opener uses it with the table schema, for the projection
and for the predicate.
`replace_columns_with_literals` is unchanged.
## What is the testing strategy for this PR?
- `pushed_down_projection_sees_field_metadata` in
`datafusion/core/tests/parquet/schema.rs`: `arrow_metadata(id,
'ARROW:extension:name')` over a one-row Parquet file.
- A new case in `cast_extension_type_metadata.slt`: a `UUID` column
written with `COPY ... TO` and read back. It sets `skip_metadata =
false` because until apache#26854 is merged, schema inference drops extension
types by default (apache#26851).
- A unit test for `replace_columns_with_literals_keeping_metadata`.
Both integration tests fail without the change in the opener.
## Are there any user-facing changes?
Expressions evaluated in a Parquet scan now see the field metadata of
the table schema for partition columns and for columns that the
statistics prove constant.
There is one new public function,
`replace_columns_with_literals_keeping_metadata`, and no breaking API
changes.
0 commit comments