Skip to content

feat(datafusion): implement try_pushdown_sort on DeltaScanExec - #5037

Open
Shahzeb-DataEngineer wants to merge 1 commit into
delta-io:mainfrom
Shahzeb-DataEngineer:feat/try-pushdown-sort
Open

Shahzeb-DataEngineer wants to merge 1 commit into
delta-io:mainfrom
Shahzeb-DataEngineer:feat/try-pushdown-sort

Conversation

@Shahzeb-DataEngineer

Copy link
Copy Markdown
Contributor

Description

Implements ExecutionPlan::try_pushdown_sort on DeltaScanExec so DataFusion's PushdownSort physical optimizer rule can push sort-order hints through the scan to the underlying Parquet DataSourceExec. This enables row-group reordering and TopK dynamic-filter pruning for ORDER BY ... LIMIT queries, which today read every row (see the report in #4969).

The pushdown is refused (Unsupported) when the scan has deletion vectors, retains the row index, or uses column mapping, and when the sort references a Delta-materialized column (partition value, file id, row index) that lives above the Parquet child. The result is always downgraded to Inexact so the Sort operator is kept: the files are not sorted and can overlap, so no exact ordering is promised.

The approach follows the reference implementation in truefoundry#20 (linked from the issue), adapted to current main and verified with the regression test below.

Related Issue

closes #4969

Tests

New regression test test_try_pushdown_sort_delegates_data_columns_and_rejects_unsupported_cases in crates/core/src/delta_datafusion/table_provider/next/scan/exec.rs:

  • a physical data column pushes down as Inexact and configures the Parquet reorder hints (reverse_row_groups=true, sort_order_for_reorder=);
  • an end-to-end SELECT value ... ORDER BY value DESC LIMIT 1 plan goes through the PushdownSort rule and returns 1 row;
  • partition column (year), synthetic file_id, retained row index, deletion-vector, and column-mapped scans all return Unsupported.

cargo test -p deltalake-core --features datafusion --lib test_try_pushdown_sort: 1 passed. cargo fmt -p deltalake-core clean; cargo clippy -p deltalake-core --features datafusion --lib --all-targets: no warnings in the touched file (4 pre-existing warnings in scan/mod.rs).

Note: the dat git submodule is not checked out in my environment, so the test_scan_* tests that read ../../dat/... fixtures fail there both with and without this change (verified on pristine main); the new test uses only crates/test/tests/data fixtures.

Push sort-order hints through DeltaScanExec to the underlying Parquet
DataSourceExec so DataFusion's PushdownSort rule can enable row-group
reordering and TopK dynamic-filter pruning for ORDER BY ... LIMIT queries.

The pushdown is refused (Unsupported) when the scan has deletion vectors,
retains the row index, or uses column mapping, and when the sort references
a Delta-materialized column (partition value, file id, row index). The
result is always downgraded to Inexact so the Sort operator is kept: the
files are not sorted and can overlap, so no exact ordering is promised.

Adds a regression test covering the delegated data-column case, the
end-to-end SQL plan, and each rejection case.

Closes delta-io#4969

Signed-off-by: Shahzeb Ali <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 7, 2026 05:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot added the binding/rust Issues for the Rust crate label Nov 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/rust Issues for the Rust crate

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

feat(datafusion): implement try_pushdown_sort on DeltaScanExec

2 participants