Skip to content

Upgrade to DataFusion branch-56 - #124

Open
xudong963 wants to merge 1 commit into
datafusion-contrib:mainfrom
xudong963:xudong963/data-fusion-branch-56-upgrade
Open

xudong963 wants to merge 1 commit into
datafusion-contrib:mainfrom
xudong963:xudong963/data-fusion-branch-56-upgrade

Conversation

@xudong963

@xudong963 xudong963 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Upgrades from DataFusion 52 to the (not yet released) branch-56, plus arrow 60.1, object_store 0.14, and MSRV 1.95.

Note: 56.0.0 is not on crates.io yet (latest is 55.2.0), so the datafusion* dependencies temporarily point at git = "https://github.com/apache/datafusion.git", branch = "branch-56". This blocks cargo publish; they should be switched back to "56" once 56.0.0 is released.

API adaptations

  • Drop as_any from TableProvider / ExecutionPlan / PhysicalExpr / ScalarUDFImpl impls; downcast via trait upcasting (plan.downcast_ref::<T>()). UserDefinedLogicalNode still has as_any, so the OneOf downcasts are unchanged.
  • ExecutionPlan: properties() returns &Arc<PlanProperties>; implement the now-required apply_expressions; implement replace_children and delegate the deprecated with_new_children to it; replace deprecated partition_statistics with statistics_from_inputs + child_stats_requests (OneOfExec only requests stats for the best candidate).
  • OneOfExec declares LimitBehavior::CopyToChildren instead of the deprecated supports_limit_pushdown (the same behavior the old default mapped to).
  • ExtensionPlanner::plan_extension / PhysicalPlanner take &dyn Session and a PhysicalPlanningContext; refresh_candidate_readiness now takes &dyn Session.
  • TableProvider::scan projection is Option<&[usize]>.
  • Replace deprecated TableFunctionImpl::call → call_with_args, LogicalPlanBuilder::scan_with_filters / TableScan::try_new → TableScanBuilder; OptimizerRule::supports_rewrite was removed.
  • TableReference / ResolvedTableReference are no longer re-exported from the datafusion_sql root; import from datafusion_common.
  • ListingOptions no longer has collect_stat / target_partitions (DF 55); object_store::head now needs ObjectStoreExt.

Regression check

No branch-56-specific regressions found. The only test expectation changes come from behavior changes that landed in DF 54 / 55:

DF 56 rewrote LimitPushdown on top of the new limit_behavior API, so I verified limit pushdown through OneOfExec explicitly: the fetch is still copied into every candidate with no extra limit node, matching DF 52. That check is kept as a new test (tests_limit_pushdown).

Move from DataFusion 52 to the unreleased branch-56 (git dependencies until
56.0.0 is published), arrow 60.1, object_store 0.14, MSRV 1.95.

API adaptations:
- Drop `as_any` from TableProvider / ExecutionPlan / PhysicalExpr /
  ScalarUDFImpl impls and downcast via trait upcasting
- ExecutionPlan: `properties()` returns `&Arc<PlanProperties>`, implement
  `apply_expressions`, `replace_children`, and `statistics_from_inputs` +
  `child_stats_requests` instead of the deprecated `partition_statistics`
- OneOfExec declares `LimitBehavior::CopyToChildren` instead of the
  deprecated `supports_limit_pushdown`
- ExtensionPlanner / PhysicalPlanner take `&dyn Session` and a
  `PhysicalPlanningContext`
- `TableProvider::scan` projection is `Option<&[usize]>`
- Replace deprecated `TableFunctionImpl::call`, `scan_with_filters`,
  `TableScan::try_new` with `call_with_args` / `TableScanBuilder`
- `TableReference` / `ResolvedTableReference` now come from datafusion_common
- `ListingOptions` lost `collect_stat` / `target_partitions`

Test updates (DataFusion behavior changes, not branch-56 regressions):
- struct unnest output columns no longer carry the `__unnest_placeholder`
  prefix (DF 54, apache/datafusion#21725)
- `concat_ws` coerces its separator literal to Utf8View (DF 55,
  apache/datafusion#22244)

Add a regression test asserting limit pushdown still copies the fetch into
every OneOfExec candidate under the rewritten DF 56 LimitPushdown rule.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@xudong963

Copy link
Copy Markdown
Member Author

cc @zhuqi-lucas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants