Repository navigation
Commit 9082d6b
fix: discard parquet bounds when row group statistics are missing (#25228)
## Which issue does this PR close?
Closes #25226.
## Rationale for this change
MIN/MAX queries can return incorrect results when a Parquet file
contains a mix of row groups with and without bounds. A missing bound is
converted to a NULL in the per-row-group statistics array. Aggregating
that array skips the NULL, allowing another group's endpoint to be
reported as an exact bound for the entire file. The optimizer can then
answer the aggregate from incomplete statistics.
For example, a file contains `["a", "b"]` with statistics and two
8192-character strings starting with `z` without statistics. `SELECT
LENGTH(MIN(a)), LENGTH(MAX(a))` previously returned `1, 1`; the correct
result is `1, 8192`.
## What changes are included in this PR?
- Invalidate a file-level bound when any row group has no converted
bound and may contain non-NULL values. Return `Precision::Absent` so
that incomplete statistics cannot establish the file's extrema.
- Handle minimum and maximum independently, preserving a usable bound
when only the other endpoint is missing.
- Preserve bounds from other groups when the group with no bound is
empty or is proven all NULL. The all-NULL check requires both the column
value count and the explicitly recorded NULL count to equal the row
count.
- Express the active-accumulator precondition with `expect`: callers
already check for `Some`, and the path that clears the accumulator
returns immediately. This removes an unreachable fallback return.
## What is the testing strategy for this PR?
`test_min_max_require_complete_row_group_bounds` covers absent
statistics, independently missing minimum/maximum values, unknown and
partial NULL counts, empty groups, all-NULL groups, and complete bounds.
Every case runs in both row-group orders.
`parquet_missing_bounds.slt` verifies both the stored string lengths and
the aggregate results. Its fixture is generated at runtime using public
Rust Parquet writer APIs, enabling chunk statistics for the first row
group and disabling them for the second. Disabling statistics is
deliberate: parquet-rs truncates long extrema rather than omitting them.
No generated Parquet file is committed and no footer is manually edited.
Local validation:
- All 10 tests in `metadata::tests::statistics_tests` passed.
- `parquet_missing_bounds.slt` passed with the fix. Running the same
fixture and SQL test against unmodified main reproduced the incorrect
`1, 1` result.
- LLVM coverage confirmed that the removed fallback return had zero
executions and that the replacement precondition assertion was exercised
36 times by the statistics tests. The updated Codecov patch percentage
remains subject to the next CI run.
## Are there any user-facing changes?
MIN/MAX queries return correct results for Parquet files with incomplete
row-group bounds. Valid statistics remain usable for each independently
complete endpoint, including when other groups are empty or all NULL.
There are no public API changes.
---------
Co-authored-by: Jay Zhan <[email protected]>1 parent b5345a8 commit 9082d6b
3 files changed
Lines changed: 248 additions & 6 deletions
File tree
- datafusion
- datasource-parquet/src
- sqllogictest
- src
- test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
739 | 739 | | |
740 | 740 | | |
741 | 741 | | |
742 | | - | |
| 742 | + | |
743 | 743 | | |
744 | | - | |
| 744 | + | |
745 | 745 | | |
746 | 746 | | |
747 | 747 | | |
| |||
750 | 750 | | |
751 | 751 | | |
752 | 752 | | |
753 | | - | |
| 753 | + | |
754 | 754 | | |
755 | | - | |
| 755 | + | |
756 | 756 | | |
757 | 757 | | |
758 | 758 | | |
| |||
785 | 785 | | |
786 | 786 | | |
787 | 787 | | |
788 | | - | |
| 788 | + | |
789 | 789 | | |
790 | 790 | | |
791 | 791 | | |
792 | 792 | | |
793 | 793 | | |
794 | 794 | | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
795 | 823 | | |
796 | 824 | | |
797 | 825 | | |
| |||
1281 | 1309 | | |
1282 | 1310 | | |
1283 | 1311 | | |
1284 | | - | |
| 1312 | + | |
1285 | 1313 | | |
1286 | 1314 | | |
1287 | 1315 | | |
| |||
1678 | 1706 | | |
1679 | 1707 | | |
1680 | 1708 | | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
| 1736 | + | |
| 1737 | + | |
| 1738 | + | |
| 1739 | + | |
| 1740 | + | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
| 1746 | + | |
| 1747 | + | |
| 1748 | + | |
| 1749 | + | |
| 1750 | + | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
| 1761 | + | |
| 1762 | + | |
| 1763 | + | |
| 1764 | + | |
| 1765 | + | |
| 1766 | + | |
| 1767 | + | |
| 1768 | + | |
| 1769 | + | |
| 1770 | + | |
| 1771 | + | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
1681 | 1811 | | |
1682 | 1812 | | |
1683 | 1813 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
151 | 154 | | |
152 | 155 | | |
153 | 156 | | |
| |||
415 | 418 | | |
416 | 419 | | |
417 | 420 | | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
418 | 492 | | |
419 | 493 | | |
420 | 494 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
0 commit comments