Repository navigation
Commit 257417b
authored
[feature](be) Support expression zonemap pruning (apache#63389)
Problem Summary: Support expression-based ZoneMap pruning for internal
and Parquet scan paths. Supported expressions include comparisons,
IN/NOT IN, IS NULL/IS NOT NULL, and starts_with. The implementation adds
segment/page/row-group pruning, conservative fallback semantics, and
profile counters, with BE unit and SQL regression coverage. CHAR string
range statistics are supported through the trimmed logical bounds
produced by ZoneMap deserialization, so CHAR predicates can participate
in expression ZoneMap pruning.
This follow-up also removes duplicated capability checks between
`can_evaluate_zonemap_filter` and `evaluate_zonemap_filter`. Capability
checks now focus on expression shape and NULL-literal cases, while
storage/runtime datatype invariants are asserted with `DORIS_CHECK` in
the evaluation path.
The latest follow-up centralizes HybridSet-to-Field materialization for
scalar and direct IN predicates, removes identity-cast unwrapping from
expression ZoneMap matching, and lets null-aware runtime filters use
ZoneMap pruning when the target zone has no NULL values while
conservatively keeping zones that may contain NULL.
The latest crash-fix follow-up makes comparison expression ZoneMap
capability checks conservative when the scan slot type and literal type
are not directly compatible, such as external table predicates comparing
`Nullable(DateTimeV2(6))` with a `DateTimeV2(0)` literal. In that case
Doris now skips expression ZoneMap pruning instead of triggering a BE
`DORIS_CHECK`, leaving the normal expression evaluation path to filter
rows.
### Release note
Support expression-based ZoneMap pruning for scan predicates to skip
data using ZoneMap statistics.
### Check List (For Author)
- Test: Unit Test / Regression test / Format check
- DORIS_HOME=<repo> ninja -C be/ut_build_ASAN Exprs Storage Exec Format
doris_be_test
- DORIS_HOME=<repo> ninja -C be/ut_build_ASAN Exprs doris_be_test
- ./run-be-ut.sh --run --filter=SegmentFilterHelpersTest.*
- ./run-be-ut.sh --run
--filter=ScanNormalizePredicateTest.*:RuntimeFilterConsumerHelperTest.*:VRuntimeFilterWrapperSamplingTest.*:ScannerLateArrivalRFTest.*
- ./run-be-ut.sh --run --filter=ParquetExprTest.test_expr_zonemap_*
- ./run-be-ut.sh --run --filter=ExprZonemapFilterTest.*
- ./run-be-ut.sh --run
--filter='ExprZonemapFilterTest.FunctionStringStartsWithZonemapUsesPrefixRange:function_string_test.function_starts_with_test'
- ./run-be-ut.sh --run
--filter='ExprZonemapFilterTest.*:ParquetExprTest.test_expr_zonemap_*'
- ninja -C be/ut_build_ASAN
src/storage/CMakeFiles/Storage.dir/segment/segment.cpp.o
src/storage/CMakeFiles/Storage.dir/segment/segment_iterator.cpp.o
- ./run-be-ut.sh --run
--filter=ParquetExprTest.test_expr_zonemap_page_filter_keeps_unsupported_results_and_counts_stats
- doris-local-regression.sh all -d inverted_index_p0 -s
test_index_range_in_select
- ./run-regression-test.sh --conf <local-regression-conf> --run -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- ./run-regression-test.sh --conf <local-regression-conf> --run -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- doris-local-regression --network 10.26.20.3/24 all -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- doris-local-regression --network 10.26.20.3/24 all -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- doris-local-regression --network 10.26.20.3/24 all -d mtmv_p0 -s
test_iceberg_mtmv -conf enableIcebergTest=true -conf
iceberg_rest_uri_port=18283 -conf iceberg_minio_port=19001 -conf
externalEnvIp=127.0.0.1
- build-support/clang-format.sh --files
be/src/exprs/function/functions_comparison.h
be/test/exprs/expr_zonemap_filter_test.cpp
- build-support/check-format.sh --files
be/src/exprs/function/functions_comparison.h
be/test/exprs/expr_zonemap_filter_test.cpp
- build-support/clang-format.sh
- build-support/check-format.sh
- doris-local-regression --network 10.26.20.3/24 run -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- doris-local-regression --network 10.26.20.3/24 run -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- git diff --check
- Behavior changed: Yes. Supported scan predicates, including CHAR
string predicates, can now prune data with expression ZoneMap
evaluation.
- Does this need documentation: No1 parent f9105c3 commit 257417b
45 files changed
Lines changed: 3481 additions & 27 deletions
File tree
- be
- src
- exec
- operator
- runtime_filter
- scan
- exprs
- function
- format
- orc
- parquet
- storage
- index/zone_map
- segment
- test
- exec/runtime_filter
- exprs
- format/parquet
- storage/segment
- fe/fe-core/src/main/java/org/apache/doris/qe
- gensrc/thrift
- regression-test/suites
- query_p0/expr_zonemap
- query_p1/expr_zonemap
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
232 | 242 | | |
233 | 243 | | |
234 | 244 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
167 | 178 | | |
168 | 179 | | |
169 | 180 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
103 | | - | |
| 103 | + | |
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
935 | 935 | | |
936 | 936 | | |
937 | 937 | | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
938 | 946 | | |
939 | 947 | | |
940 | 948 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
0 commit comments