Repository navigation
Commit 349cd15
authored
[Enhancement](zonemap) Support column-vs-column comparisons in expression zone map pruning (apache#67774)
Zone map pruning only fires when a predicate compares a column against a
constant. A predicate over two columns of the same table, `WHERE a != b`
or `WHERE a < b`, never participates, so every segment and every Parquet
row group is read even when the two columns' min/max ranges make the
predicate unsatisfiable.
Everything on the path except the leaf already supports this.
`ZoneMapEvalContext` is keyed per slot and the segment-level and Parquet
row-group-level callers already populate an entry for every slot a
compound expression references, which is why cross-column AND / OR
compounds prune today. What was missing is the comparison operator's
capability gate: it accepts one slot plus one literal and nothing else,
so a two-slot comparison is dropped before evaluation.
This PR adds the slot-vs-slot shape next to the existing slot-vs-literal
one and gives the six always-false tests a single implementation shared
by both. With the left operand's non-null values in `[lmin, lmax]` and
the right operand's in `[rmin, rmax]`, the comparison cannot be TRUE for
any row when:
| operator | condition |
| --- | --- |
| `=` | `lmin > rmax \|\| rmin > lmax` |
| `!=` | `lmin == lmax && rmin == rmax && lmin == rmin` |
| `<` | `lmin >= rmax` |
| `<=` | `lmin > rmax` |
| `>` | `rmin >= lmax` |
| `>=` | `rmin > lmax` |
Substituting the degenerate range `[literal, literal]` for the right
operand reduces each row to the slot-vs-literal test already in the
tree, so the single-column path keeps its exact current behaviour. The
existing tests passing unchanged is the mechanical check on that.
`kNoMatch` means "no row in this zone can make the conjunct TRUE", not
"every row is FALSE", so NULL does not weaken these rules: a WHERE
conjunct discards FALSE and NULL alike, and the bounds summarize exactly
the non-null values. A column with no non-null value at all makes the
comparison NULL on every row and returns `kNoMatch` unconditionally.
One trap for reviewers: `can_evaluate` must not be widened. It is shared
by dictionary filtering and by `can_evaluate_equality`, and both
dereference `extract_slot_and_literal` behind a `DORIS_CHECK`, so
accepting a slot-vs-slot expression there would abort. This PR adds a
separate `can_evaluate_slot_slot` and ORs it into
`can_evaluate_zonemap_filter` only; a unit test pins that down.
Three deliberate limitations: a cast around either operand is rejected,
because the bounds are raw stored values and the optimizer inserts a
cast whenever the two column types differ; on Parquet a floating column
pair does not prune at all, because Parquet bounds omit NaN without
reporting how many were skipped, which would flip the `!=` rule; and
VARIANT, JSONB, ARRAY/MAP/STRUCT and aggregate-state columns are not
admitted, because `Field` compares any two of their values as equal, and
a segment types a VARIANT path slot by its inferred leaf type, which the
expression's VARIANT type would not match.
The v1 Parquet reader is excluded from slot-vs-slot pruning entirely,
because that reader is being removed. `_process_expr_zonemap_filter`
restricts itself to single-slot conjuncts (the v1 page path already
required a single column), so column-vs-column pruning runs on native
segments and the v2 Parquet reader only. This also moots the v1-specific
slot-vs-slot review threads here (malformed fixed-width statistics,
VARBINARY mapping, and the adjusted-UTC timestamp rollback tracked by
the now-closed apache#68118). The v1 single-column path keeps the
decimal-domain and NaN fixes described next.
A limitation on the v1 single-column path came out of review.
`ParquetPredicate::parse_min_max_value` decodes v1 bounds in the file's
own logical type, while `_type_matches` compares only primitive types,
so a file DECIMAL at scale 3 and a table DECIMAL at scale 2 both passed
and their unscaled payloads were then compared as if they shared a
domain. The v1 reader now leaves the zone map out unless the file and
table types agree, which is the check `format_v2` already applies. That
costs pruning in three narrow places, all of them correctness-driven: a
DATETIMEV2 or TIMESTAMPTZ column whose declared scale differs from the
Parquet unit loses expression zone map pruning entirely, which is the
widest practical case and matches the policy already written down for
the slot-vs-literal path; a decimal pair differing only in precision is
domain-identical yet also skipped; and IS NULL / IS NOT NULL pruning
goes with it, because a zone map carrying only the null flags leaves
min/max at `TYPE_NULL` and any range predicate would then fail
`range_stats_usable_for_zonemap`'s type check.
This also fixes a bug it uncovered. The flag that carries "these bounds
came from Parquet, so a hidden NaN is possible" was only ever set in the
format_v2 reader; the v1 reader assigned the slot's data type and left
the flag at its default, so the NaN guard never fired on that path. Two
DOUBLE columns whose Parquet bounds both read `[1.0, 1.0]` were then
pruned by the `!=` rule even when one of them hid a NaN, dropping a row
that should have come back. The rule now lives in one setter on
`SlotZoneMap` that assigns the type and the flag together, and all three
Parquet call sites go through it.
The hidden version and TSO columns need no special handling here: since
apache#65684, a single-version segment reads them through a
`ConstantColumnReader` whose zone map holds the substituted value, so
pruning sees the same value the rows come back with.1 parent 7a7e86c commit 349cd15
9 files changed
Lines changed: 1342 additions & 81 deletions
File tree
- be
- src
- exprs
- function
- format_v2/parquet
- format/parquet
- storage/index/zone_map
- test/exprs
- regression-test
- data/query_p0/expr_zonemap
- suites/query_p0/expr_zonemap
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
250 | 250 | | |
251 | 251 | | |
252 | 252 | | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
253 | 286 | | |
254 | 287 | | |
255 | 288 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
100 | 115 | | |
101 | 116 | | |
102 | 117 | | |
| |||
120 | 135 | | |
121 | 136 | | |
122 | 137 | | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
123 | 148 | | |
124 | 149 | | |
125 | 150 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
293 | 294 | | |
294 | 295 | | |
295 | 296 | | |
296 | | - | |
297 | | - | |
298 | | - | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
299 | 324 | | |
300 | | - | |
301 | | - | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
302 | 330 | | |
303 | 331 | | |
304 | 332 | | |
305 | | - | |
| 333 | + | |
306 | 334 | | |
307 | 335 | | |
308 | 336 | | |
| |||
314 | 342 | | |
315 | 343 | | |
316 | 344 | | |
317 | | - | |
318 | | - | |
| 345 | + | |
| 346 | + | |
319 | 347 | | |
320 | 348 | | |
321 | 349 | | |
322 | 350 | | |
323 | 351 | | |
324 | | - | |
| 352 | + | |
325 | 353 | | |
326 | 354 | | |
327 | 355 | | |
328 | | - | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
342 | | - | |
343 | | - | |
344 | | - | |
345 | | - | |
346 | | - | |
347 | | - | |
348 | | - | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
349 | 371 | | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
350 | 422 | | |
351 | | - | |
352 | | - | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
353 | 434 | | |
354 | 435 | | |
355 | 436 | | |
| |||
377 | 458 | | |
378 | 459 | | |
379 | 460 | | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
380 | 495 | | |
381 | 496 | | |
382 | 497 | | |
| |||
736 | 851 | | |
737 | 852 | | |
738 | 853 | | |
739 | | - | |
740 | | - | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
741 | 859 | | |
742 | 860 | | |
743 | 861 | | |
| |||
0 commit comments