Repository navigation
Commit de0e5d9
fix: apply struct field filters when the file schema needs adaptation
When the declared table schema differs from the physical file schema for a
struct column, the expression adapter wraps the whole struct column in a
cast, so `s['x']` becomes `get_field(cast(s AS Struct<..>), 'x')`.
That hides the column from consumers that pattern match on
`get_field(column, 'f')`. The Parquet scan is one such consumer: it decides
at planning time (against the table schema) that a struct-field predicate
can be evaluated as a row filter and reports it as fully handled, so
`FilterExec` is removed from the plan. At runtime the row filter builder no
longer recognizes the adapted expression, silently drops the predicate, and
the query returns unfiltered rows.
Narrow the cast to the field that is actually read:
`get_field(cast(s AS Struct<..>), 'x')` becomes
`cast(get_field(s, 'x') AS <type of x>)`. This keeps the column visible
under the `get_field`, and also avoids materializing a whole cast struct
just to read one field. Fields that are missing from the file collapse to a
typed null literal, matching what the struct cast would have produced.
`get_field` on a Map column is a runtime key lookup rather than a
schema-level field access, so those keep the whole-column cast.
Closes apache#24109.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MebN5PsVnYvXUeVKju5K7P1 parent e6b4221 commit de0e5d9
3 files changed
Lines changed: 437 additions & 0 deletions
File tree
- datafusion
- datasource-parquet/src/opener
- physical-expr-adapter/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3775 | 3775 | | |
3776 | 3776 | | |
3777 | 3777 | | |
| 3778 | + | |
| 3779 | + | |
| 3780 | + | |
| 3781 | + | |
| 3782 | + | |
| 3783 | + | |
| 3784 | + | |
| 3785 | + | |
| 3786 | + | |
| 3787 | + | |
| 3788 | + | |
| 3789 | + | |
| 3790 | + | |
| 3791 | + | |
| 3792 | + | |
| 3793 | + | |
| 3794 | + | |
| 3795 | + | |
| 3796 | + | |
| 3797 | + | |
| 3798 | + | |
| 3799 | + | |
| 3800 | + | |
| 3801 | + | |
| 3802 | + | |
| 3803 | + | |
| 3804 | + | |
| 3805 | + | |
| 3806 | + | |
| 3807 | + | |
| 3808 | + | |
| 3809 | + | |
| 3810 | + | |
| 3811 | + | |
| 3812 | + | |
| 3813 | + | |
| 3814 | + | |
| 3815 | + | |
| 3816 | + | |
| 3817 | + | |
| 3818 | + | |
| 3819 | + | |
| 3820 | + | |
| 3821 | + | |
| 3822 | + | |
| 3823 | + | |
| 3824 | + | |
| 3825 | + | |
| 3826 | + | |
| 3827 | + | |
| 3828 | + | |
| 3829 | + | |
| 3830 | + | |
| 3831 | + | |
| 3832 | + | |
| 3833 | + | |
| 3834 | + | |
| 3835 | + | |
| 3836 | + | |
| 3837 | + | |
| 3838 | + | |
| 3839 | + | |
| 3840 | + | |
| 3841 | + | |
| 3842 | + | |
| 3843 | + | |
| 3844 | + | |
| 3845 | + | |
| 3846 | + | |
| 3847 | + | |
| 3848 | + | |
| 3849 | + | |
| 3850 | + | |
| 3851 | + | |
| 3852 | + | |
| 3853 | + | |
| 3854 | + | |
| 3855 | + | |
| 3856 | + | |
| 3857 | + | |
| 3858 | + | |
| 3859 | + | |
| 3860 | + | |
| 3861 | + | |
| 3862 | + | |
| 3863 | + | |
| 3864 | + | |
| 3865 | + | |
| 3866 | + | |
| 3867 | + | |
| 3868 | + | |
| 3869 | + | |
| 3870 | + | |
| 3871 | + | |
| 3872 | + | |
| 3873 | + | |
| 3874 | + | |
| 3875 | + | |
| 3876 | + | |
| 3877 | + | |
| 3878 | + | |
| 3879 | + | |
| 3880 | + | |
| 3881 | + | |
| 3882 | + | |
3778 | 3883 | | |
0 commit comments