Repository navigation
Commit 045b590
## Which issue does this PR close?
- Part of #24462
- Backport of #24125 to `branch-55` (for 55.1.0, tracked in #24462).
- Fixes #24109
## Rationale for this change
With `datafusion.execution.parquet.pushdown_filters = true`, a predicate
on a struct field was reported as fully handled by the scan whenever the
file needed schema adaptation, so `FilterExec` was removed from the plan
and the predicate was silently dropped — returning every row instead of
the filtered set. This is a correctness bug (wrong results), not
specific to 55.0.0, so it fits the backport criteria.
## What changes are included in this PR?
Cherry-pick of #24125 (commit 40c208e).
Git's recursive merge auto-resolved surrounding context differences in
`datafusion/physical-expr-adapter/src/schema_rewriter.rs` and
`datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt`;
no manual conflict resolution or adaptation of the fix itself was
required.
One follow-up commit adapts a test expectation: `branch-55` doesn't have
#24130/#24315, which taught nested schema pruning to union the leaves
needed by mixed whole-column + field-access reads (e.g. `select s,
s['y'] from narrow`). Without that optimization, the mixed-access case
falls back to reading every physical leaf, so `bytes_scanned` is `219`
here instead of the `146` the original PR's test expects on `main`. This
is a pre-existing difference in pruning capability, not a correctness
regression from this fix.
## Are these changes tested?
Yes. Carries the original regression coverage, all tests pass.
## Are there any user-facing changes?
`WHERE s['field'] = ...` predicates on struct columns now filter
correctly when Parquet filter pushdown requires schema adaptation. No
API changes.
---------
Co-authored-by: Adrian Garcia Badaracco <[email protected]>
Co-authored-by: Claude <[email protected]>
1 parent a323585 commit 045b590
3 files changed
Lines changed: 654 additions & 12 deletions
File tree
- datafusion
- physical-expr-adapter/src
- sqllogictest/test_files
0 commit comments