Repository navigation
Commit 4130ce4
feat: sort file groups by statistics during sort pushdown
Sort files within each file group by min/max statistics during sort
pushdown to better align with the requested ordering. When files are
non-overlapping and within-file ordering is guaranteed (e.g. Parquet
with sorting_columns metadata), the SortExec is completely eliminated.
Key changes:
- ParquetSource::try_pushdown_sort returns Exact when natural ordering
satisfies the request, enabling sort elimination
- FileScanConfig sorts files within groups by statistics and verifies
non-overlapping property to determine Exact vs Inexact
- Multi-group files are redistributed consecutively to preserve both
sort elimination and I/O parallelism across partitions
- Statistics-based file reordering as fallback when FileSource returns
Unsupported (benefits TopK via better dynamic filter pruning)
- New sort_pushdown benchmark for measuring sort elimination speedup
Closes apache#17348
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>1 parent c17c87c commit 4130ce4
4 files changed
Lines changed: 974 additions & 111 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- datasource-parquet/src
- datasource/src
- physical-optimizer/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
258 | | - | |
259 | | - | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
260 | 261 | | |
261 | 262 | | |
262 | 263 | | |
| |||
265 | 266 | | |
266 | 267 | | |
267 | 268 | | |
268 | | - | |
| 269 | + | |
269 | 270 | | |
270 | 271 | | |
271 | 272 | | |
| |||
278 | 279 | | |
279 | 280 | | |
280 | 281 | | |
281 | | - | |
282 | | - | |
| 282 | + | |
283 | 283 | | |
284 | 284 | | |
285 | 285 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
743 | 743 | | |
744 | 744 | | |
745 | 745 | | |
746 | | - | |
747 | | - | |
748 | | - | |
749 | | - | |
750 | | - | |
751 | | - | |
752 | | - | |
753 | | - | |
754 | | - | |
755 | | - | |
756 | | - | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
757 | 754 | | |
758 | 755 | | |
| 756 | + | |
759 | 757 | | |
760 | 758 | | |
761 | 759 | | |
| |||
767 | 765 | | |
768 | 766 | | |
769 | 767 | | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
770 | 778 | | |
771 | 779 | | |
772 | 780 | | |
| |||
811 | 819 | | |
812 | 820 | | |
813 | 821 | | |
814 | | - | |
815 | | - | |
816 | | - | |
817 | | - | |
818 | | - | |
819 | 822 | | |
820 | 823 | | |
821 | 824 | | |
| |||
0 commit comments