Repository navigation
Closed
Description
Is your feature request related to a problem or challenge?
- Part of [EPIC] DataFusion Releases #19783
- Follow on to Release DataFusion 52.1.0 (minor/patch) Release (Jan 2026) #19784
We have found a few more issues while upgrading to 52 across the ecosystem, so let' contemplate another patch release for the DataFUsion 52 line
Steps:
- Update version and changelog: [branch-52] Update version and prepare changelog #20560
- Test with DataFusion Python
- Test with DataFusion Comet
- Test with delta.rs:
- Test vortex:
- Test with iceberg-rust:
- Test with LakeSail:
- Test with parquet viewer
- Test with datafusion-materialized-views
- Voting Thread: https://lists.apache.org/thread/8g5ozm5vxfnzf1ows028vfrbxl1sz21z
- Publish to crates.io
Potential Content:
Requested Bug Fixes. Issue / requester / Backport PR
- [Bug] [v52 regression] FilterExec parent dynamic filters have incorrect column indices #20285 by @jackkleeman / [branch-52] FilterExec should remap indices of parent dynamic filters (#20286) #20347
- FilterPushdown do not generate correct column index when merge FilterExec #20109 by @haohuaijin / [branch-52] fix: filter pushdown when merge filter (#20110) #20289
- Incorrect answers due to an incorrectly removed Sort before AggregateExec #20244 by @alamb / [branch-52] Fix incorrect
SortExecremoval beforeAggregateExec(#20247) #20507 - REMOVED per discussion here Dynamic filter applied to the wrong table when using subqueries #20213 by @adriangb / @jackkleeman / NA
- Invalid inter-file ordering in eq_properties() #20508 by @adriangb / [branch-52] fix: validate inter-file ordering in eq_properties() (#20329) #20509
- [Bug] [v52 regression] Panic in
GroupOrderingPartial::remove_groupswhen Partial aggregate with PartiallySorted hits memory pressure #20445 by @jackkleeman / NEEDS BACKPORT - Panic in HashJoin with dictionary-encoded column in multi-column join key #20437 by @erratic-pattern / [branch-52] fix: HashJoin panic with dictionary-encoded columns in multi-key joins (#20441) #20512
- Improve substrait NameTracker so it doesn't require uuids #17508 by @hareshkh / Fix name tracker (#19856) #20539
- [branch-52] Update aws-smithy, bytes and time for security audits #20546
Reacted by Hengfei Yang and Huaijin
Metadata
Metadata
Assignees
Labels
No labels
Activity
adriangb commentedon Feb 11, 2026
I think it's worth including #20213
[branch-52] fix: filter pushdown when merge filter (#20110) (#20289)
alamb commentedon Feb 13, 2026
I am out next week so I can't realistically drive this release (though I can probably do the mechanics of tarball and voting)
Here is a backport of #20109
After that, I think we are still waiting on
Incorrect answers due to an incorrectly removed Sort before AggregateExec #20244
REview / merge fix: Fix incorrect
SortExecremoval beforeAggregateExec(option 2) #20247Backport: TBD
Dynamic filter applied to the wrong table when using subqueries #20213
Fix: TBD
Backport: TBD
adriangb commentedon Feb 14, 2026
Another one to include: #20329 (review)
alamb commentedon Feb 14, 2026
Added to list
jackkleeman commentedon Feb 20, 2026
#20445 I think this is a candidate of something we need to fix in 52.2, as it is a runtime panic that you can trigger reliably with certain queries
alamb commentedon Feb 22, 2026
@erratic-pattern also found
alamb commentedon Feb 22, 2026
I added
I plan to review this ticket and the related backports tomorrow and hopefully make a release candidate Monday or Tuesday
[branch-52] FilterExec should remap indices of parent dynamic filters (…
hareshkh commentedon Feb 24, 2026
@alamb @gabotechs : Can we get #19856 onto 52.2.0? I can create a PR to branch-52.
cc: @xanderbailey
milenkovicm commentedon Feb 24, 2026
would this change be backward incompatible change for a minor release?
[branch-52] fix: validate inter-file ordering in eq_properties() (#20329
xanderbailey commentedon Feb 24, 2026
I don't think so, this should fix substrait plans that previously failed. The only "change" is that conflicting literal columns are no longer aliased to a UUID which I don't think is a break. The output schema is unchanged in this case.
alamb commentedon Feb 24, 2026
Added to the list above. Please do create a branch-52 PR @hareshkh
hareshkh commentedon Feb 25, 2026
@alamb Here is the backport for branch-52 : #20539
alamb commentedon Feb 25, 2026
I am looking for some committers to approve backports
SortExecremoval beforeAggregateExec(#20247) #20507[branch-52] fix: HashJoin panic with dictionary-encoded columns in mu…
alamb commentedon Feb 25, 2026
Looks like we will not backport this fix (per discussion here with @nuno-faria and @adriangb )
[branch-52] Fix incorrect `SortExec` removal before `AggregateExec` (#…
alamb commentedon Feb 25, 2026
Ok, I think we have the content finalized. We just need to merge a few more PRs and I can make the Release Candidate:
Thank you to @adriangb and @comphead who have helped approve others so far
[branch-52] Update aws-smithy, bytes and time for security audits (#2…
[branch-53] Clamp early aggregation emit to the sort boundary when us…
[branch-52] Update version and prepare changelog (#20560)
[branch-52] Update version and prepare changelog (apache#20560)
alamb commentedon Feb 26, 2026
I made a release candidate and started a voting thread:
alamb commentedon Mar 1, 2026
The release has been approved and submitted to crates.io
https://dist.apache.org/repos/dist/release/datafusion/datafusion-52.2.0/
https://crates.io/crates/datafusion/52.2.0
https://lists.apache.org/thread/k7t4wn61cf7pro74cd6gxr1bcbtjbyhg
[main] Update version to 52.2.0 (#20573)
Merge datafusion upstream 52.4.0 into branch-52 (#98)