Repository navigation
Commit 2a4191c
authored
[improvement](be) Use zero NaN counts for Parquet floating-point pruning (#69655)
### What problem does this PR solve?
Problem Summary:
Parquet V2 treats every FLOAT/DOUBLE column as having an unknown NaN
count. Finite min/max bounds omit NaNs, so the existing comparison and
NOT IN guards correctly retain groups that might contain a matching NaN.
This also prevents pruning when a writer explicitly supplies
`nan_count=0`.
For example, a column chunk with `min=max=1` and a trustworthy zero NaN
count cannot match `x > 1`, `x >= 2`, `x <> 1`, or `x NOT IN (1)`.
Previously the reader decoded those rows anyway. This change makes that
zero proof available to the existing ZoneMap evaluator.
### Optimization approach
- Add standard optional Statistics field 9, `nan_count`, matching the
[Apache Parquet format
definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift).
Compared the touched definition on master and branch-3.1/4.0/4.1/4.2:
existing field IDs retain their meanings and field 9 is unused on those
Doris branches. Older readers skip the optional field, and existing
files retain the conservative fallback.
- Retain a zero-count proof in converted FLOAT/DOUBLE statistics only
when the count is explicitly zero, the value/null counts are consistent,
and the decoded bounds are usable (or the chunk is all NULL). Missing,
positive, negative, or excessive counts do not clear the NaN-unknown
flag. Invalid/NaN/unordered bounds remain unusable.
- Pass the proof alongside the row-group ZoneMap. For PageIndex
evaluation, reuse only the proof from the exact owning row group and
physical leaf column. It covers every page of that chunk. Both
single-slot and compound OR/AND page evaluation use this scoped proof.
- Keep the existing comparator and NOT IN semantics. Integer/string
columns and legacy floating files avoid the additional footer decode in
the page proof path. Writer support and additional column orders are
outside this reader change.
### Optimization effect
| Per reader query | Absent count (previous fallback) | Explicit zero
count |
| --- | ---: | ---: |
| RawRowsRead | 524,288 | 4,096 (-99.22%) |
| RowGroupsFilteredByMinMax | 0 | 127 |
| Actual local read bytes | 6,358,496 | 98,444 (-98.45%) |
| Median wall time | 6.908 ms | 0.690 ms (~10.0x) |
| Median process CPU time | 6.868 ms | 0.687 ms (~10.0x) |
Measured using Clang 20 RELEASE, warm local filesystem cache, one pinned
CPU. The absent-count samples range from 6.787 to 7.252 ms and
zero-count samples from 0.673 to 0.730 ms. All returned IDs and
per-query work counters match their expectations across all samples.
The opt-in `ParquetScanTest.ZeroNanCountReaderMicrobench` uses the
actual V2 reader and corresponds to:
```sql
SELECT id FROM parquet_source WHERE d > 1;
```
Fixture: 128 row groups x 4,096 rows; the first 127 groups contain
`d=1`, the last contains `d=2`. FLOAT, DOUBLE, and an integer row
identifier are stored using uncompressed PLAIN DataPageV2 with
dictionary and PageIndex disabled. Two files have identical data pages
and differ only in the optional zero-count fields. The absent-count
variant exercises the same conservative behavior as the pre-change
reader. Each query checks every returned ID and the pruning counters.
Seven samples of eight reader queries per variant follow warmup,
alternating variant order. The benchmark records wall time, process CPU
time, actual local read bytes, RawRowsRead, and
RowGroupsFilteredByMinMax. Results describe a local reader
microbenchmark; full SQL timings depend on storage, parallelism, and the
rest of the query.
### Validation and reproduction
- Related BE suites: 372 tests passed; the opt-in benchmark was skipped
in that run and then passed separately.
- The three initial reader regressions first failed on unmodified master
because zero counts did not reduce RawRowsRead or increase pruning.
Their result rows were already correct; the missing behavior was
pruning.
- Generated C++ Thrift code compiled successfully with the optional
field. Old files without the field remain covered by the complete reader
suite.
- clang-format 16 and BE build-hygiene checks passed. Changed-line
clang-tidy passed on all four affected C++ files.
Coverage includes FLOAT/DOUBLE; zero/positive/absent/negative/excessive
counts; GT/GE/NE/NOT IN; invalid null counts and bounds; all NULL/all
NaN/mixed NULL-finite-NaN rows; dictionary on/off; DataPageV1/V2;
PageIndex on/off; column/row-group proof scope; and compound OR page
pruning when one branch lacks a zero proof. Mixed NaN cases are also
checked against a statistics-disabled file.
Correctness suites:
```bash
BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \
--filter='*Parquet*:*ColumnMapper*:*TableReader*'
```
Opt-in benchmark:
```bash
DORIS_PARQUET_NAN_COUNT_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \
./run-be-ut.sh -j8 --run --filter=ParquetScanTest.ZeroNanCountReaderMicrobench
```
### Release note
Improve Parquet V2 floating-point Min-Max pruning when trustworthy zero
NaN counts are available.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (reader microbenchmark)
- Behavior changed:
- [x] Yes. Proven NaN-free floating chunks/pages can be skipped by
existing comparison and NOT IN predicates.
- Does this need documentation?
- [x] No. Internal reader optimization using existing file metadata.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent fc1b822 commit 2a4191c
5 files changed
Lines changed: 494 additions & 36 deletions
File tree
- be
- src/format_v2/parquet
- test/format_v2/parquet
- gensrc/thrift
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
942 | 942 | | |
943 | 943 | | |
944 | 944 | | |
945 | | - | |
| 945 | + | |
946 | 946 | | |
947 | 947 | | |
948 | 948 | | |
| 949 | + | |
| 950 | + | |
949 | 951 | | |
950 | 952 | | |
951 | 953 | | |
| |||
1015 | 1017 | | |
1016 | 1018 | | |
1017 | 1019 | | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
1018 | 1029 | | |
1019 | 1030 | | |
1020 | 1031 | | |
| |||
1035 | 1046 | | |
1036 | 1047 | | |
1037 | 1048 | | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
1038 | 1092 | | |
1039 | 1093 | | |
1040 | 1094 | | |
| |||
1092 | 1146 | | |
1093 | 1147 | | |
1094 | 1148 | | |
1095 | | - | |
1096 | | - | |
1097 | | - | |
1098 | | - | |
1099 | | - | |
1100 | | - | |
1101 | | - | |
1102 | | - | |
1103 | | - | |
1104 | | - | |
1105 | | - | |
1106 | | - | |
1107 | | - | |
1108 | | - | |
1109 | | - | |
1110 | | - | |
1111 | | - | |
1112 | | - | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
1113 | 1154 | | |
1114 | 1155 | | |
1115 | 1156 | | |
| |||
1881 | 1922 | | |
1882 | 1923 | | |
1883 | 1924 | | |
1884 | | - | |
| 1925 | + | |
1885 | 1926 | | |
1886 | 1927 | | |
1887 | 1928 | | |
1888 | 1929 | | |
1889 | 1930 | | |
1890 | 1931 | | |
| 1932 | + | |
1891 | 1933 | | |
1892 | 1934 | | |
1893 | 1935 | | |
| |||
1912 | 1954 | | |
1913 | 1955 | | |
1914 | 1956 | | |
| 1957 | + | |
1915 | 1958 | | |
1916 | 1959 | | |
1917 | 1960 | | |
| |||
1964 | 2007 | | |
1965 | 2008 | | |
1966 | 2009 | | |
1967 | | - | |
| 2010 | + | |
| 2011 | + | |
1968 | 2012 | | |
1969 | 2013 | | |
1970 | 2014 | | |
| |||
1999 | 2043 | | |
2000 | 2044 | | |
2001 | 2045 | | |
| 2046 | + | |
| 2047 | + | |
| 2048 | + | |
| 2049 | + | |
2002 | 2050 | | |
2003 | 2051 | | |
2004 | 2052 | | |
| |||
2020 | 2068 | | |
2021 | 2069 | | |
2022 | 2070 | | |
| 2071 | + | |
2023 | 2072 | | |
2024 | 2073 | | |
2025 | 2074 | | |
| |||
2088 | 2137 | | |
2089 | 2138 | | |
2090 | 2139 | | |
| 2140 | + | |
| 2141 | + | |
2091 | 2142 | | |
2092 | 2143 | | |
2093 | 2144 | | |
| |||
2103 | 2154 | | |
2104 | 2155 | | |
2105 | 2156 | | |
2106 | | - | |
| 2157 | + | |
2107 | 2158 | | |
2108 | 2159 | | |
2109 | 2160 | | |
| |||
2123 | 2174 | | |
2124 | 2175 | | |
2125 | 2176 | | |
2126 | | - | |
2127 | | - | |
| 2177 | + | |
| 2178 | + | |
2128 | 2179 | | |
2129 | 2180 | | |
2130 | 2181 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
104 | 105 | | |
105 | 106 | | |
106 | 107 | | |
| |||
0 commit comments