Repository navigation
Commit fd101e9
authored
perf: prune ListingTable files before grouping (apache#26441)
## Which issue does this PR close?
Closes apache#26130.
## Rationale for this change
When files contain disjoint ranges of a filtered column, ListingTable
groups all files before file-statistics pruning. The matching files can
therefore land in a single scan partition, and scan statistics still
include files that cannot match.
In the eight-file example from the issue, `d >= 20 AND d < 40` currently
places both matching files in the second of four groups and estimates 80
rows. Pruning before grouping retains only `f2` and `f3`, puts them in
separate groups, and aggregates 20 survivor rows.
## What changes are included in this PR?
- Pass non-partition filters through a private listing path and reuse
`PruningPredicateBuilder` with already collected file statistics before
splitting regular scan file groups.
- Keep files with missing/inexact statistics or evaluation errors,
retain normal row filtering, and avoid evaluating volatile filters
during planning.
- Bypass the optimization for custom expression adapters and declared
output partitioning, preserving partition identities and empty trailing
groups.
- Recompute aggregate statistics from survivors without additional
object-store requests.
- Add regression coverage and update physical-plan/runtime-metric
expectations for files eliminated during planning. No public API change.
## What is the testing strategy for this PR?
On the investigated baseline (`a4b8198cf`), local validation passed
12,579 distinct Rust tests and 533 SQL logic files. This includes 23
listing unit tests, 10 focused pruning integration tests, all 246
Parquet integration tests, and 126 extended fuzz tests. The no-extra-I/O
test compares object-store requests during planning.
Coverage includes the eight-file grouping example, partially matching
files, missing and inexact bounds, NULL semantics, Hash/Range partition
layouts, Hive/data predicates, LIMIT/OFFSET, custom adapters, volatile
UDF invocation counts, survivor statistics, and conservative
filtered-statistics precision. Nine SQL snapshot files change
physical-plan expectations only.
On current upstream main (`5f2158269`), all 1,459 core integration tests
(including all 10 pruning regressions and the no-extra-I/O test), all
247 Parquet tests, and all 535 SQL logic files pass. The full extended
workspace run completed successfully: 12,742 Rust tests passed,
including all 126 fuzz tests, with eight tests ignored and no failures.
All 535 SQL logic files also passed. Formatting, diff whitespace, TOML
formatting, and license headers on all 2,331 tracked project source
files pass.
The earlier full workspace Clippy run passed with explicit allowances
for pre-existing upstream/toolchain diagnostics. The mandatory strict
Clippy invocation also fails on the updated main branch with Rust 1.99,
on three pre-existing `clippy::mutable_key_type` diagnostics in
untouched `datafusion/optimizer/src/push_down_filter.rs`. The CI-style
Clippy invocation (`--workspace --all-targets --features
avro,integration-tests,extended_tests -- -D warnings`) passes. The full
lint wrapper stops at its license walk because locally exported
third-party SQLite fixture snapshots lack project headers; the separate
tracked-project-source license check passes. Subsequent wrapper steps
have not been run. No lint suppression or unrelated optimizer change is
included in this PR. No performance speedup is claimed: the prior
Hive-only planning benchmark is not representative of data-column
pruning and ran under differing concurrent load.
## Are there any user-facing changes?
With statistics collection enabled, regular listing scans can contain
fewer files and report tighter statistics in EXPLAIN. Query results and
row filtering are unchanged. Unsupported or inconclusive statistics keep
files, and declared partition layouts are preserved.
This contribution was developed with AI assistance. The design and
safeguards are described above; performance under distributed execution
remains unmeasured.1 parent 6e0e0da commit fd101e9
19 files changed
Lines changed: 970 additions & 57 deletions
File tree
- datafusion
- catalog-listing
- src
- core/tests
- catalog_listing
- datasource
- parquet
- sql
- sqllogictest/test_files
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| 45 | + | |
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
| |||
662 | 664 | | |
663 | 665 | | |
664 | 666 | | |
665 | | - | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
666 | 673 | | |
667 | 674 | | |
668 | 675 | | |
| |||
915 | 922 | | |
916 | 923 | | |
917 | 924 | | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
918 | 937 | | |
919 | 938 | | |
920 | 939 | | |
921 | 940 | | |
922 | | - | |
| 941 | + | |
923 | 942 | | |
924 | 943 | | |
925 | 944 | | |
926 | | - | |
| 945 | + | |
| 946 | + | |
927 | 947 | | |
928 | 948 | | |
929 | 949 | | |
| |||
979 | 999 | | |
980 | 1000 | | |
981 | 1001 | | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
982 | 1042 | | |
983 | 1043 | | |
984 | 1044 | | |
985 | 1045 | | |
| 1046 | + | |
986 | 1047 | | |
987 | 1048 | | |
988 | 1049 | | |
| |||
1005 | 1066 | | |
1006 | 1067 | | |
1007 | 1068 | | |
| 1069 | + | |
1008 | 1070 | | |
1009 | 1071 | | |
1010 | 1072 | | |
| |||
1018 | 1080 | | |
1019 | 1081 | | |
1020 | 1082 | | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
1021 | 1094 | | |
1022 | 1095 | | |
1023 | 1096 | | |
1024 | 1097 | | |
1025 | | - | |
1026 | | - | |
1027 | | - | |
1028 | | - | |
1029 | | - | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
1030 | 1103 | | |
1031 | 1104 | | |
| 1105 | + | |
| 1106 | + | |
1032 | 1107 | | |
1033 | 1108 | | |
1034 | 1109 | | |
| |||
1260 | 1335 | | |
1261 | 1336 | | |
1262 | 1337 | | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
1263 | 1371 | | |
1264 | 1372 | | |
1265 | 1373 | | |
| |||
1421 | 1529 | | |
1422 | 1530 | | |
1423 | 1531 | | |
| 1532 | + | |
| 1533 | + | |
| 1534 | + | |
| 1535 | + | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
| 1550 | + | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
| 1571 | + | |
| 1572 | + | |
| 1573 | + | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
| 1580 | + | |
| 1581 | + | |
| 1582 | + | |
| 1583 | + | |
| 1584 | + | |
| 1585 | + | |
| 1586 | + | |
| 1587 | + | |
| 1588 | + | |
| 1589 | + | |
| 1590 | + | |
| 1591 | + | |
| 1592 | + | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
| 1619 | + | |
| 1620 | + | |
| 1621 | + | |
| 1622 | + | |
| 1623 | + | |
| 1624 | + | |
| 1625 | + | |
| 1626 | + | |
| 1627 | + | |
| 1628 | + | |
| 1629 | + | |
| 1630 | + | |
| 1631 | + | |
| 1632 | + | |
| 1633 | + | |
| 1634 | + | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
1424 | 1665 | | |
0 commit comments