Repository navigation
Commit ddb4f73
refactor: use row-group-local selections for Parquet scans (apache#25608)
## Which issue does this PR close?
- Related to apache#24358
- Related to apache#23883
- Related to
apache#24186 (comment)
## Rationale for this change
Parquet access plans already describe selections per row group, but the
scan path flattens them into one selection and reconstructs group
boundaries when reversing the scan. Keeping selections attached to their
row groups preserves local coordinates and bitmap representations while
simplifying ordering and decoder rebuilds.
## What changes are included in this PR?
- Pass row-group-local selections to the Parquet push decoder and keep
each selection together with its static match status.
- Validate selection lengths and remove empty selections during
preparation; reverse and reorder whole row-group entries.
- Remove global-selection conversion from the scan path and simplify
reversal. Deprecate `ParquetAccessPlan::into_overall_row_selection`
while retaining its existing behavior.
- Reuse the decoder's remaining selections when only the row filter
changes, avoiding redundant selector copies.
- Preserve the current policy that disables statistics reordering and
runtime pruning for scans with selections.
## What is the testing strategy for this PR?
The new integration test covers mixed bitmap and selector selections
across row groups. Unit tests cover selection validation, empty
selections, match-status alignment, and reversal of non-contiguous
groups with unequal sizes.
Passed targeted tests:
- `cargo test -p datafusion-datasource-parquet --lib access_plan::test
--offline`
- `cargo test -p datafusion-datasource-parquet --lib sort::tests
--offline`
- `cargo test -p datafusion-datasource-parquet --lib push_decoder::tests
--offline`
- `cargo test -p datafusion-datasource-parquet --lib
statistics_order_tests --offline`
- `cargo test -p datafusion --test parquet_integration
external_access_plan --offline`
- `cargo test -p datafusion --test parquet_integration
dynamic_row_group_pruning --offline`
Pre-submission checks: `cargo fmt --all`, `cargo clippy --all-targets
--all-features -- -D warnings`, and `uv run ./dev/rust_lint.sh`.
## Are there any user-facing changes?
`ParquetAccessPlan::into_overall_row_selection` is deprecated starting
in 56.0.0 but remains available with its existing behavior. The upgrade
guide describes migration for direct callers. Users who attach a
`ParquetAccessPlan` to a `PartitionedFile` need no changes.1 parent e3812d0 commit ddb4f73
8 files changed
Lines changed: 522 additions & 1338 deletions
File tree
- datafusion
- core/tests/parquet
- datasource-parquet/src
- opener
- docs/source/library-user-guide/upgrading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
310 | 310 | | |
311 | 311 | | |
312 | 312 | | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
313 | 343 | | |
314 | 344 | | |
315 | 345 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1665 | 1665 | | |
1666 | 1666 | | |
1667 | 1667 | | |
1668 | | - | |
1669 | | - | |
| 1668 | + | |
| 1669 | + | |
1670 | 1670 | | |
1671 | 1671 | | |
1672 | 1672 | | |
| |||
1685 | 1685 | | |
1686 | 1686 | | |
1687 | 1687 | | |
1688 | | - | |
| 1688 | + | |
1689 | 1689 | | |
1690 | 1690 | | |
1691 | 1691 | | |
| |||
1720 | 1720 | | |
1721 | 1721 | | |
1722 | 1722 | | |
1723 | | - | |
1724 | | - | |
1725 | | - | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
1726 | 1726 | | |
1727 | 1727 | | |
1728 | 1728 | | |
| |||
1734 | 1734 | | |
1735 | 1735 | | |
1736 | 1736 | | |
1737 | | - | |
1738 | | - | |
1739 | | - | |
1740 | | - | |
1741 | | - | |
1742 | | - | |
1743 | | - | |
1744 | | - | |
1745 | | - | |
1746 | | - | |
1747 | | - | |
1748 | | - | |
1749 | | - | |
1750 | | - | |
1751 | | - | |
1752 | | - | |
1753 | | - | |
1754 | | - | |
1755 | | - | |
1756 | | - | |
1757 | | - | |
1758 | | - | |
1759 | | - | |
1760 | | - | |
1761 | | - | |
1762 | | - | |
1763 | | - | |
1764 | | - | |
1765 | | - | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
| 1749 | + | |
| 1750 | + | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
1766 | 1754 | | |
1767 | 1755 | | |
1768 | 1756 | | |
| |||
1778 | 1766 | | |
1779 | 1767 | | |
1780 | 1768 | | |
1781 | | - | |
1782 | | - | |
| 1769 | + | |
1783 | 1770 | | |
1784 | 1771 | | |
1785 | 1772 | | |
| |||
1857 | 1844 | | |
1858 | 1845 | | |
1859 | 1846 | | |
1860 | | - | |
1861 | | - | |
1862 | | - | |
1863 | | - | |
1864 | | - | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
1865 | 1850 | | |
1866 | 1851 | | |
1867 | 1852 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
| |||
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
63 | | - | |
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
| |||
83 | 84 | | |
84 | 85 | | |
85 | 86 | | |
86 | | - | |
| 87 | + | |
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
92 | 93 | | |
93 | | - | |
| 94 | + | |
94 | 95 | | |
95 | 96 | | |
96 | 97 | | |
| |||
100 | 101 | | |
101 | 102 | | |
102 | 103 | | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
| 104 | + | |
107 | 105 | | |
108 | 106 | | |
109 | 107 | | |
| |||
301 | 299 | | |
302 | 300 | | |
303 | 301 | | |
304 | | - | |
| 302 | + | |
305 | 303 | | |
306 | 304 | | |
307 | 305 | | |
| |||
565 | 563 | | |
566 | 564 | | |
567 | 565 | | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
568 | 569 | | |
569 | 570 | | |
570 | 571 | | |
| |||
669 | 670 | | |
670 | 671 | | |
671 | 672 | | |
672 | | - | |
673 | 673 | | |
674 | | - | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
675 | 685 | | |
676 | 686 | | |
677 | 687 | | |
| |||
0 commit comments