fix: filter pushdown when merge filter by haohuaijin · Pull Request #20110 · apache/datafusion

haohuaijin · 2026-02-02T12:32:06Z

Which issue does this PR close?

Closes FilterPushdown do not generate correct column index when merge FilterExec #20109

Rationale for this change

see issue #20109

What changes are included in this PR?

Remap parent filter expressions: When a FilterExec has a projection, remap unsupported parent filter expressions from output schema coordinates to input schema coordinates using reassign_expr_columns() before combining them with the current filter's predicates.
Preserve projection: When creating the merged FilterExec, preserve the original projection instead of discarding it .

Are these changes tested?

yes, add some test case

Are there any user-facing changes?

adriangb · 2026-02-02T14:02:19Z

@haohuaijin please ping me when this is ready for review

haohuaijin · 2026-02-03T02:50:42Z

cc @adriangb, ready for reviews

adriangb

Thanks for the fix! It would be great to add an SLT test but not a blocker for merging this if it's hard to hit this case from SQL.

datafusion/core/tests/physical_optimizer/filter_pushdown.rs

adriangb · 2026-02-03T13:52:52Z

datafusion/physical-plan/src/filter.rs

-        let unsupported_parent_filters =
-            child_pushdown_result.parent_filters.iter().filter_map(|f| {
-                matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
-            });
+        let mut unsupported_parent_filters: Vec<Arc<dyn PhysicalExpr>> =
+            child_pushdown_result
+                .parent_filters
+                .iter()
+                .filter_map(|f| {
+                    matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
+                })
+                .collect();


Note: this is the same code, just added the mut which caused a reformat.

adriangb · 2026-02-03T13:53:11Z

datafusion/physical-plan/src/filter.rs

+        // If this FilterExec has a projection, the unsupported parent filters
+        // are in the output schema (after projection) coordinates. We need to
+        // remap them to the input schema coordinates before combining with self filters.
+        if self.projection.is_some() {


👍🏻 note that this is because filters get evaluated before the projection

adriangb · 2026-02-03T13:54:07Z

datafusion/core/tests/physical_optimizer/filter_pushdown.rs

+
+/// related to https://github.com/apache/datafusion/issues/20109
+#[tokio::test]
+async fn test_filter_with_projection_pushdown() {


Any chance you could add an SLT test that reproduces this? It could go in datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt

Co-authored-by: Adrian Garcia Badaracco <[email protected]>

haohuaijin · 2026-02-03T14:36:14Z

Thanks for you reviews @adriangb, i try add a SLT test, but use sql is hard to do.

adriangb · 2026-02-03T15:24:37Z

No worries. Let's merge this once CI passes.

alamb · 2026-02-03T21:26:53Z

🚀

hengfeiyang · 2026-02-04T02:49:55Z

Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.

adriangb · 2026-02-04T13:05:34Z

@hengfeiyang we can certainly put this on the docket for the next minor release, but we don’t tend to do more than 1 minor release per major version, so this may have to wait until the next major version if there aren’t many other fixes. Would you be able to point your system at a git commit or fork with this commit cherry picked? That’s what we do

hengfeiyang · 2026-02-04T13:18:39Z

Yes. definitely we can do that, actually we did that in the past if we can't get a release soon. just we don't like to use fork version if we can have official release.

Thank you for you reply. we will use fork version before next release.

alamb · 2026-02-04T21:51:50Z

Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.

I am happy to help support another release (run the voting, etc) if someone else is willing do make the tickets, backport and changelog PRs

alamb · 2026-02-04T21:52:48Z

(BTW I think I may also have a bug in 52 that I'll file tomorrow which might be good to fix too). I'll post here if I do

## Which issue does this PR close? - Closes apache#20109 ## Rationale for this change see issue apache#20109 ## What changes are included in this PR? 1. Remap parent filter expressions: When a FilterExec has a projection, remap unsupported parent filter expressions from output schema coordinates to input schema coordinates using `reassign_expr_columns()` before combining them with the current filter's predicates. 2. Preserve projection: When creating the merged FilterExec, preserve the original projection instead of discarding it . ## Are these changes tested? yes, add some test case ## Are there any user-facing changes? --------- Co-authored-by: Adrian Garcia Badaracco <[email protected]>

alamb · 2026-02-11T11:46:21Z

Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.

I am happy to help support another release (run the voting, etc) if someone else is willing do make the tickets, backport and changelog PRs

Update: Filed a ticket to track a next patch release

Release DataFusion 52.2.0 (minor/) Release (Feb 2026) #20287

## Which issue does this PR close? - Closes apache#20109 ## Rationale for this change see issue apache#20109 ## What changes are included in this PR? 1. Remap parent filter expressions: When a FilterExec has a projection, remap unsupported parent filter expressions from output schema coordinates to input schema coordinates using `reassign_expr_columns()` before combining them with the current filter's predicates. 2. Preserve projection: When creating the merged FilterExec, preserve the original projection instead of discarding it . ## Are these changes tested? yes, add some test case ## Are there any user-facing changes? --------- Co-authored-by: Adrian Garcia Badaracco <[email protected]>

## Which issue does this PR close? - related to #20287 ## Rationale for this change see issue #20109 ## What changes are included in this PR? 1. Remap parent filter expressions: When a FilterExec has a projection, remap unsupported parent filter expressions from output schema coordinates to input schema coordinates using `reassign_expr_columns()` before combining them with the current filter's predicates. 2. Preserve projection: When creating the merged FilterExec, preserve the original projection instead of discarding it . ## Are these changes tested? yes, add some test case ## Are there any user-facing changes? --------- ## Which issue does this PR close?  - Closes #. ## Rationale for this change  ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Adrian Garcia Badaracco <[email protected]>

fix: filter pushdown when merge filter

84b8d20

github-actions bot added the physical-plan Changes to the physical-plan crate label Feb 2, 2026

add test case

26567f3

github-actions bot added the core Core DataFusion crate label Feb 2, 2026

adriangb self-requested a review February 2, 2026 14:02

haohuaijin added 2 commits February 3, 2026 10:40

remove unused code

e57c7cf

update

687d5d1

haohuaijin marked this pull request as ready for review February 3, 2026 02:50

Merge branch 'main' into fix-filterpushdown-issue

f191e9e

chore

ab0bb28

adriangb approved these changes Feb 3, 2026

View reviewed changes

Update datafusion/core/tests/physical_optimizer/filter_pushdown.rs

c95db6c

Co-authored-by: Adrian Garcia Badaracco <[email protected]>

Merge branch 'main' into fix-filterpushdown-issue

1e21b1c

alamb added this pull request to the merge queue Feb 3, 2026

Merged via the queue into apache:main with commit b80bf2c Feb 3, 2026
33 checks passed

haohuaijin deleted the fix-filterpushdown-issue branch February 4, 2026 01:13

alamb mentioned this pull request Feb 11, 2026

[branch-52] fix: filter pushdown when merge filter (#20110) #20289

Merged

Conversation

haohuaijin commented Feb 2, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

adriangb commented Feb 2, 2026

Uh oh!

haohuaijin commented Feb 3, 2026

Uh oh!

adriangb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adriangb Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

haohuaijin commented Feb 3, 2026

Uh oh!

adriangb commented Feb 3, 2026

Uh oh!

alamb commented Feb 3, 2026

Uh oh!

Uh oh!

hengfeiyang commented Feb 4, 2026

Uh oh!

adriangb commented Feb 4, 2026

Uh oh!

hengfeiyang commented Feb 4, 2026

Uh oh!

alamb commented Feb 4, 2026

Uh oh!

alamb commented Feb 4, 2026

Uh oh!

alamb commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haohuaijin commented Feb 2, 2026 •

edited by alamb

Loading

adriangb left a comment •

edited

Loading

alamb commented Feb 11, 2026 •

edited

Loading