Repository navigation
Conversation
Move from DataFusion 52 to the unreleased branch-56 (git dependencies until 56.0.0 is published), arrow 60.1, object_store 0.14, MSRV 1.95. API adaptations: - Drop `as_any` from TableProvider / ExecutionPlan / PhysicalExpr / ScalarUDFImpl impls and downcast via trait upcasting - ExecutionPlan: `properties()` returns `&Arc<PlanProperties>`, implement `apply_expressions`, `replace_children`, and `statistics_from_inputs` + `child_stats_requests` instead of the deprecated `partition_statistics` - OneOfExec declares `LimitBehavior::CopyToChildren` instead of the deprecated `supports_limit_pushdown` - ExtensionPlanner / PhysicalPlanner take `&dyn Session` and a `PhysicalPlanningContext` - `TableProvider::scan` projection is `Option<&[usize]>` - Replace deprecated `TableFunctionImpl::call`, `scan_with_filters`, `TableScan::try_new` with `call_with_args` / `TableScanBuilder` - `TableReference` / `ResolvedTableReference` now come from datafusion_common - `ListingOptions` lost `collect_stat` / `target_partitions` Test updates (DataFusion behavior changes, not branch-56 regressions): - struct unnest output columns no longer carry the `__unnest_placeholder` prefix (DF 54, apache/datafusion#21725) - `concat_ws` coerces its separator literal to Utf8View (DF 55, apache/datafusion#22244) Add a regression test asserting limit pushdown still copies the fetch into every OneOfExec candidate under the rewritten DF 56 LimitPushdown rule. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Member
Author
|
cc @zhuqi-lucas |
40 of 61 tasks
zhuqi-lucas
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades from DataFusion 52 to the (not yet released) branch-56, plus arrow 60.1, object_store 0.14, and MSRV 1.95.
API adaptations
as_anyfromTableProvider/ExecutionPlan/PhysicalExpr/ScalarUDFImplimpls; downcast via trait upcasting (plan.downcast_ref::<T>()).UserDefinedLogicalNodestill hasas_any, so theOneOfdowncasts are unchanged.ExecutionPlan:properties()returns&Arc<PlanProperties>; implement the now-requiredapply_expressions; implementreplace_childrenand delegate the deprecatedwith_new_childrento it; replace deprecatedpartition_statisticswithstatistics_from_inputs+child_stats_requests(OneOfExeconly requests stats for the best candidate).OneOfExecdeclaresLimitBehavior::CopyToChildreninstead of the deprecatedsupports_limit_pushdown(the same behavior the old default mapped to).ExtensionPlanner::plan_extension/PhysicalPlannertake&dyn Sessionand aPhysicalPlanningContext;refresh_candidate_readinessnow takes&dyn Session.TableProvider::scanprojection isOption<&[usize]>.TableFunctionImpl::call→call_with_args,LogicalPlanBuilder::scan_with_filters/TableScan::try_new→TableScanBuilder;OptimizerRule::supports_rewritewas removed.TableReference/ResolvedTableReferenceare no longer re-exported from thedatafusion_sqlroot; import fromdatafusion_common.ListingOptionsno longer hascollect_stat/target_partitions(DF 55);object_store::headnow needsObjectStoreExt.Regression check
No branch-56-specific regressions found. The only test expectation changes come from behavior changes that landed in DF 54 / 55:
__unnest_placeholder(...)prefix (DF 54, feat: remove__unnest_placeholderfrom struct unnest projection apache/datafusion#21725) — the "window & unnest" test SQL referenced the old names and now usesdate.yearetc.concat_wscoerces its separator literal toUtf8View(DF 55, feat: support mixed binary and string types for concat UDFs apache/datafusion#22244) — explain text only, results unchanged.DF 56 rewrote
LimitPushdownon top of the newlimit_behaviorAPI, so I verified limit pushdown throughOneOfExecexplicitly: the fetch is still copied into every candidate with no extra limit node, matching DF 52. That check is kept as a new test (tests_limit_pushdown).