Repository navigation
feat(datafusion): implement try_pushdown_sort on DeltaScanExec - #5037
Open
Shahzeb-DataEngineer wants to merge 1 commit into
Open
Shahzeb-DataEngineer wants to merge 1 commit into
Shahzeb-DataEngineer wants to merge 1 commit into
Conversation
Push sort-order hints through DeltaScanExec to the underlying Parquet DataSourceExec so DataFusion's PushdownSort rule can enable row-group reordering and TopK dynamic-filter pruning for ORDER BY ... LIMIT queries. The pushdown is refused (Unsupported) when the scan has deletion vectors, retains the row index, or uses column mapping, and when the sort references a Delta-materialized column (partition value, file id, row index). The result is always downgraded to Inexact so the Sort operator is kept: the files are not sorted and can overlap, so no exact ordering is promised. Adds a regression test covering the delegated data-column case, the end-to-end SQL plan, and each rejection case. Closes delta-io#4969 Signed-off-by: Shahzeb Ali <[email protected]>
Shahzeb-DataEngineer
requested review from
ethan-tyler,
ion-elgreco,
roeap and
rtyler
as code owners
November 7, 2026 05:13
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements
ExecutionPlan::try_pushdown_sortonDeltaScanExecso DataFusion'sPushdownSortphysical optimizer rule can push sort-order hints through the scan to the underlying ParquetDataSourceExec. This enables row-group reordering and TopK dynamic-filter pruning forORDER BY ... LIMITqueries, which today read every row (see the report in #4969).The pushdown is refused (
Unsupported) when the scan has deletion vectors, retains the row index, or uses column mapping, and when the sort references a Delta-materialized column (partition value, file id, row index) that lives above the Parquet child. The result is always downgraded toInexactso theSortoperator is kept: the files are not sorted and can overlap, so no exact ordering is promised.The approach follows the reference implementation in truefoundry#20 (linked from the issue), adapted to current
mainand verified with the regression test below.Related Issue
closes #4969
Tests
New regression test
test_try_pushdown_sort_delegates_data_columns_and_rejects_unsupported_casesincrates/core/src/delta_datafusion/table_provider/next/scan/exec.rs:Inexactand configures the Parquet reorder hints (reverse_row_groups=true,sort_order_for_reorder=);SELECT value ... ORDER BY value DESC LIMIT 1plan goes through thePushdownSortrule and returns 1 row;year), syntheticfile_id, retained row index, deletion-vector, and column-mapped scans all returnUnsupported.cargo test -p deltalake-core --features datafusion --lib test_try_pushdown_sort: 1 passed.cargo fmt -p deltalake-coreclean;cargo clippy -p deltalake-core --features datafusion --lib --all-targets: no warnings in the touched file (4 pre-existing warnings inscan/mod.rs).Note: the
datgit submodule is not checked out in my environment, so thetest_scan_*tests that read../../dat/...fixtures fail there both with and without this change (verified on pristinemain); the new test uses onlycrates/test/tests/datafixtures.