Repository navigation
Commit c882119
fix: preserve explicit Struct casts and evolved field semantics (apache#24680)
## Which issue does this PR close?
Closes apache#24679. Follow-up to apache#24125 and apache#24530.
## Rationale for this change
When Parquet files have different schemas, DataFusion may need to
convert a file's Struct column to the table's logical Struct type. If a
query only reads `s.x`, struct-cast narrowing can save work by
converting just `x` instead of every field in `s`.
That optimization must preserve the meaning of the original expression.
The earlier narrowing rule could hide an explicit cast's error, change
the result's nullability, or make an entirely null input fail during
decimal conversion.
### Selecting one field must not hide an explicit cast error
For `s = {x: 1, y: 'bad'}`, consider this physical expression:
```sql
get_field(CAST(s AS STRUCT<x INT, y INT>), 'x')
```
The explicit cast asks to convert both fields. It must fail because
`'bad'` cannot become an integer, even though the caller only uses `x`.
Narrowing it to `CAST(get_field(s, 'x') AS INT)` instead returns `1`:
the conversion of `y`, and its error, have disappeared.
This failure is reproduced directly at the physical-expression adapter
boundary. The distinction is whether a cast was inserted to reconcile
file and table schemas or was already part of the query; the two can
have the same shape but different obligations.
### An entirely null Struct should not acquire a decimal-conversion
error
Consider a Parquet batch with this schema evolution:
```text
File schema: s: Struct<x: Utf8>
Table schema: s: Struct<x: Decimal128(10, -1)>
Batch values: s is NULL for every row
```
DataFusion's whole-Struct conversion returns nulls without converting
its children. If the rewrite extracts `x` first, however, it invokes
string-to-decimal conversion on an array of null strings. Arrow rejects
the negative scale while setting up that conversion, before looking at
any values. A query that should return nulls now fails despite there
being no non-null value to convert.
The same problem occurs inside a selected container. For example,
evolving `s.x` from `List<Utf8>` to `List<Decimal128(10, -1)>` can fail
even when every parent Struct is null. Checking only whether `x` itself
is a decimal misses the conversion inside the List.
This case also reproduces through a Parquet scan with filter pushdown:
`WHERE get_field(s, 'x') IS NULL` should select every row in the batch,
not fail while preparing a decimal conversion.
There is a related schema-contract problem. A required child `x` inside
a nullable logical parent `s` still gives a nullable result for `s.x`.
Rebuilding an expression from the child's Field alone can lose that
inherited nullability, including through nested parents.
## What changes are included in this PR?
The adapter now distinguishes casts it introduces for schema adaptation
from casts already present in the query. It can continue narrowing
generated conversions where that is safe while preserving explicit
Struct casts and their errors. Narrowing also preserves the original
logical result Field, including metadata and nullability; matching
scalar types alone do not establish an equivalent result. Cast tracking
also avoids quadratic lookup work for large expressions.
For the covered decimal changes, the conversion stays inside its Struct
ancestors so DataFusion's existing casting code retains control over the
all-null shortcut. The decision looks through container value types too,
so selecting a List, Map, or Dictionary does not hide a decimal
conversion from the check. The cast target keeps only the selected field
path, excluding conversions for unselected siblings. Matching types
still narrow normally, and existing container-to-Struct conversion paths
are preserved. This extends the existing fix without duplicating Arrow's
conversion-validation rules or changing the underlying casting
implementation. Unwrapping a Dictionary whose values are Structs retains
its existing all-null decimal limitation; the built-in Parquet reader
does not produce that source shape.
The Parquet reader must then be able to evaluate the retained expression
after schema adaptation. This matters when the query plan has already
delegated a predicate to the reader: retaining a cast must not prevent
that filter from executing. The runtime allowance is specifically for
field access through a Struct-to-Struct cast of a column.
The reader preserves the cast and reuses projection's existing read
clipping to decode only the leaves named by the cast target, with a
full-root fallback when clipping is unsafe. The target, rather than the
final field selection, determines which conversions must run: an
explicit cast that names `y` still reads and converts `y` even if the
query selects only `x`. Filter cost estimates use the same selected
leaves. Planning remains conservative about explicit casts, keeping a
residual filter for them.
## Are these changes tested?
The Rust regressions cover explicit cast errors, nullability inherited
from parents, decimal conversions on entirely null Structs, unselected
sibling conversion errors, and execution through Parquet filters.
Container cases include the List families, Maps, Dictionaries, and
deeper nesting, with checks that matching types and existing
container-to-Struct conversions keep their previous behavior.
The existing `schema_evolution_nested.slt` now adds SQL coverage using
generated Parquet files: an explicit Struct cast whose unselected
sibling must still fail, all-null scalar and List decimal projections,
and null predicates with filter pushdown both enabled and disabled.
Statistics and pruning shortcuts are disabled for the filter cases.
These SQL tests supplement the Rust tests, which also check adapter
provenance, logical Field metadata/nullability, expression shape, and
Arrow types that are difficult to express in SQL. The explicit-cast SQL
case is an integration control; the direct adapter test exercises the
cast-provenance regression.
The retained-cast read-plan regression now uses three physical siblings.
It verifies that an unused sibling is pruned, a sibling named by the
cast still raises its conversion error, the projected physical schema is
correct, and the estimated compressed bytes count only the selected
leaves. A target containing just the selected field also decodes and
evaluates successfully.
Local validation for
[ec4b3f2](apache@ec4b3f2)
used the tracked `Cargo.lock` unchanged, with `--locked` for the Cargo
test and all-feature Clippy commands:
- The required extended workspace suite passed **10,795 Rust tests**
(eight existing ignores) and **all 505 SQL logic files**.
- All **107 CLI tests** passed.
- The adapter and Parquet datasource crates also passed separately:
**291 unit tests and eight doctests**, with five existing doctest
ignores.
- `cargo fmt --all`, full-workspace Clippy across all targets and
features with warnings denied, and the complete `./dev/rust_lint.sh`
suite passed.
The strengthened read-plan regression was also checked with the old
full-root collection behavior restored: it failed because it selected
leaves `[1, 2, 3]` instead of `[1, 2]`. Restoring the fix made it pass
again.
<details>
<summary>Existing benchmark comparison</summary>
The existing benchmarks were compared against `main` at
`ee59f628b44eb80e8a4f126288632ef51fda5dc2`, using `release-nonlto`,
freshly built workspace artifacts for each revision, and the checked-in
Criterion settings. Variants ran sequentially after builds and tests had
stopped.
| Existing benchmark case | Result |
| --- | --- |
| `parquet_struct_filter_pushdown`: `select_id/with_pushdown` | No
change detected (p = 0.83; 100 samples per variant). |
| `parquet_nested_schema_pruning`:
`top_level_struct/select_struct_narrow_schema` | PR/main mean-time
changes were +4.15%, -2.96%, and -2.84% across three repeats, including
reversed execution order (10 samples per variant). The direction
changed, so no stable timing conclusion is claimed. |
Scan-byte counts were identical in every projection run. These cases
cover ordinary field-filter pushdown and the existing
projection-clipping path; neither directly measures retained-cast
row-filter speedup. The `main` snapshot also contains unrelated commits,
so this comparison does not isolate the follow-up patch.
</details>
## Are there any user-facing changes?
The intended changes preserve explicit-cast errors, logical field
metadata and nullability, and the covered all-null decimal behavior.
Ordinary field pruning remains enabled. Retained-cast Parquet filters
can also avoid reading sibling leaves outside the cast target; required
conversions are preserved, with a full-root fallback where necessary.
There are no public API or dependency changes. General changes to
container-to-Struct conversion semantics, generic `get_field` behavior
under null parents, and masking of encoded arrays remain outside this
PR's scope.
AI assistance: Codex assisted with the implementation, regression tests,
PR text, and the stated local checks and source reviews. The row-filter
clipping follow-up incorporates Adrian Garcia Badaracco's [recommended
change](pydantic@9746f3f),
with additional regression coverage.
---------
Co-authored-by: Adrian Garcia Badaracco <[email protected]>1 parent e4c72c5 commit c882119
5 files changed
Lines changed: 1251 additions & 72 deletions
File tree
- datafusion
- core/tests/parquet
- datasource-parquet/src
- physical-expr-adapter/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
789 | 789 | | |
790 | 790 | | |
791 | 791 | | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
792 | 998 | | |
793 | 999 | | |
794 | 1000 | | |
| |||
0 commit comments