You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields #25192
ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields.
For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different.
The issue is visible in integration for example with delta-rs.
Delta-rs - when schema_force_view_types is true, applies this flag recursively on nested schemas.
The recursively-modified schema is passed through the layers to datafusion, until apply_file_schema_type_coercions
But, apply_file_schema_type_coercions does not take into account modifications on the nested fields, onlt on the top-level fields.
So, the Parquet is opened without having view types on nested fields
Then, when we try for example to apply predicates - which ARE using view types, because that is what delta-rs knows it has - datafusion needs to cast the data to have view types, resulting in longer predicate evaluation.
Activity
[HSTACK] fix: fix: make Parquet file schema type coercion work on nes…
fix: make Parquet file schema type coercion work on nested schemas (a…
Match Parquet map key/value by position in nested schema coercion (ap…
[HSTACK] fix: make Parquet file schema type coercion work on nested s…
fix: make Parquet file schema type coercion work on nested schemas (a…
Match Parquet map key/value by position in nested schema coercion (ap…
[HSTACK] fix: fix: make Parquet file schema type coercion work on nes…
[HSTACK] fix: make Parquet file schema type coercion work on nested s…
[HSTACK] fix: make Parquet file schema type coercion work on nested s…
[HSTACK] fix: make Parquet file schema type coercion work on nested s…
[HSTACK] fix: make Parquet file schema type coercion work on nested s…