You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ec74746
Browse filesBrowse the repository at this point in the historyBrowse files
[branch-55] Preserve field metadata when coercing INT96 timestamps (#24790) (#25895)
## Which issue does this PR close?
- Backport of #24790 to `branch-55`, for the 55.2.0 patch release
(#25758).
- Closes#24786 on `branch-55`.
## Rationale for this change
With `datafusion.execution.parquet.coerce_int96` set, a file with an
INT96 column comes back with the metadata of every struct, list and map
field emptied, while leaf fields keep theirs. Readers that match fields
by Parquet field id lose the id of any container whose id sits only on
the container. Apache DataFusion Comet always enables the coercion, so
with Spark's field id reads on it null fills such a container where
Spark reads it (apache/datafusion-comet#6131).
## What changes are included in this PR?
A clean cherry-pick of #24790 (with `-x`): the three container
constructors in `schema_coercion.rs` keep the original field's metadata,
plus that PR's tests.
## What is the testing strategy for this PR?
The tests from #24790 come with the cherry-pick. The
`datafusion-datasource-parquet` crate's tests, `cargo fmt` and `cargo
clippy` pass on this branch.
## Are there any user-facing changes?
Struct, list and map fields read from files with INT96 columns keep
their field metadata when `coerce_int96` is set. No API changes.
Signed-off-by: Leonid Ryzhyk <[email protected]>
Co-authored-by: Leonid Ryzhyk <[email protected]>
0 commit comments