Repository navigation
Commit 30a44d6
authored
[improvement](be) Optimize filtered Parquet scans and condition cache (#65602)
[improvement](be) Optimize filtered Parquet scans and condition cache
Problem Summary: Parquet v2 repeatedly advanced lazy non-predicate readers for every fully filtered batch, and external condition cache could not safely reuse entries when runtime filters were present. Accumulate skipped rows for lazy readers and flush once before the next surviving batch, discard the lag at row-group boundaries, expose Arrow skip time, and key scanner-driven condition cache entries with the exact predicate and runtime-filter snapshot. Runtime filters without a reliable digest and standalone callers without an explicit split digest remain uncached.
Condition cache + runtime filter behavior
Before this change, the presence of any runtime filter disabled condition cache, even when the filter was ready and had a stable payload. Runtime-filter pushdown still worked, but repeated scans could neither look up nor publish a granule survivor bitmap.
After this change, scanner-driven splits rebuild the condition-cache digest from the exact conjunct snapshot used by that split. Ready IN, Bloom, and Min/Max runtime filters encode their probe expression, filter semantics, payload, and null behavior in the digest.
For example, for static predicate P: price > 100:
P AND customer_id IN {7, 9} uses a payload-specific key digest(P, RF{7,9}). After a cache miss reaches reader EOF, its survivor bitmap is published under that key.
P AND customer_id IN {9, 7} has the same IN-set semantics and therefore reuses the same key.
P AND customer_id IN {8, 10} produces a different key and cannot reuse the {7,9} bitmap.
If split 0 starts with only P and RF {7,9} arrives before split 1, split 0 uses digest(P) while split 1 rebuilds and uses digest(P, RF{7,9}); the late RF is never stored under the stale P key.
If any current conjunct returns digest 0, condition cache is disabled for that split while runtime-filter pushdown remains enabled. This is the correctness fallback for expressions that cannot represent their complete semantics in a stable digest—for example, an RF probe expression containing a digest-unsupported lambda, or a future/custom stateful RF whose payload is not digestible. Standalone TableReader callers with an RF also remain conservative unless they provide an explicit digest for the exact split snapshot.
Problem Summary: Parquet v2 repeatedly advanced lazy non-predicate
readers for every fully filtered batch, and external condition cache
could not safely reuse entries when runtime filters were present.
Accumulate skipped rows for lazy readers and flush once before the next
surviving batch, discard the lag at row-group boundaries, expose Arrow
skip time, and key scanner-driven condition cache entries with the exact
predicate and runtime-filter snapshot. Runtime filters without a
reliable digest and standalone callers without an explicit split digest
remain uncached.
#### Condition cache + runtime filter behavior
Before this change, the presence of any runtime filter disabled
condition cache, even when the filter was ready and had a stable
payload. Runtime-filter pushdown still worked, but repeated scans could
neither look up nor publish a granule survivor bitmap.
After this change, scanner-driven splits rebuild the condition-cache
digest from the exact conjunct snapshot used by that split. Ready IN,
Bloom, and Min/Max runtime filters encode their probe expression, filter
semantics, payload, and null behavior in the digest.
For example, for static predicate `P: price > 100`:
- `P AND customer_id IN {7, 9}` uses a payload-specific key `digest(P,
RF{7,9})`. After a cache miss reaches reader EOF, its survivor bitmap is
published under that key.
- `P AND customer_id IN {9, 7}` has the same IN-set semantics and
therefore reuses the same key.
- `P AND customer_id IN {8, 10}` produces a different key and cannot
reuse the `{7,9}` bitmap.
- If split 0 starts with only `P` and RF `{7,9}` arrives before split 1,
split 0 uses `digest(P)` while split 1 rebuilds and uses `digest(P,
RF{7,9})`; the late RF is never stored under the stale `P` key.
If any current conjunct returns digest `0`, condition cache is disabled
for that split while runtime-filter pushdown remains enabled. This is
the correctness fallback for expressions that cannot represent their
complete semantics in a stable digest—for example, an RF probe
expression containing a digest-unsupported lambda, or a future/custom
stateful RF whose payload is not digestible. Standalone `TableReader`
callers with an RF also remain conservative unless they provide an
explicit digest for the exact split snapshot.
### Release note
Improve Parquet external scan performance for highly selective
predicates and allow payload-safe condition cache reuse with ready
runtime filters.
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh -j 64 --run
--filter='ParquetScanTest.*:NewParquetReaderTest.PageIndexFilteredGapFlushesPendingOutputSkipOnce:TableReaderTest.ConditionCache*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload'`
- `./run-be-ut.sh -j 64 --run
--filter='TableReaderTest.ConditionCache*'`
- `build-support/check-format.sh`
- clang-tidy attempted but blocked by pre-existing remote toolchain
header and NOLINT errors
- Behavior changed: Yes. Lazy Parquet output readers defer skips across
empty selections, and condition cache supports runtime filters only with
an exact split digest.
- Does this need documentation: No1 parent 7f302a1 commit 30a44d6
16 files changed
Lines changed: 427 additions & 31 deletions
File tree
- be
- src
- exec/scan
- format_v2
- parquet
- reader
- test
- exec/scan
- format_v2
- jni
- parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1942 | 1942 | | |
1943 | 1943 | | |
1944 | 1944 | | |
1945 | | - | |
1946 | | - | |
1947 | | - | |
1948 | | - | |
1949 | | - | |
1950 | | - | |
1951 | | - | |
1952 | | - | |
1953 | | - | |
1954 | | - | |
1955 | | - | |
1956 | | - | |
1957 | | - | |
1958 | | - | |
1959 | | - | |
1960 | | - | |
1961 | | - | |
1962 | | - | |
1963 | | - | |
| 1945 | + | |
1964 | 1946 | | |
1965 | 1947 | | |
1966 | 1948 | | |
| |||
1990 | 1972 | | |
1991 | 1973 | | |
1992 | 1974 | | |
| 1975 | + | |
| 1976 | + | |
| 1977 | + | |
| 1978 | + | |
| 1979 | + | |
| 1980 | + | |
1993 | 1981 | | |
1994 | 1982 | | |
1995 | 1983 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
531 | 531 | | |
532 | 532 | | |
533 | 533 | | |
| 534 | + | |
534 | 535 | | |
535 | 536 | | |
536 | 537 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
270 | 270 | | |
271 | 271 | | |
272 | 272 | | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
273 | 307 | | |
274 | 308 | | |
275 | 309 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
100 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
101 | 113 | | |
102 | 114 | | |
103 | 115 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| 74 | + | |
| 75 | + | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
| |||
198 | 200 | | |
199 | 201 | | |
200 | 202 | | |
| 203 | + | |
201 | 204 | | |
202 | 205 | | |
203 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
103 | 104 | | |
104 | 105 | | |
105 | 106 | | |
| 107 | + | |
106 | 108 | | |
107 | 109 | | |
108 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
655 | 655 | | |
656 | 656 | | |
657 | 657 | | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
658 | 664 | | |
659 | 665 | | |
660 | 666 | | |
| |||
787 | 793 | | |
788 | 794 | | |
789 | 795 | | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
790 | 809 | | |
791 | | - | |
| 810 | + | |
792 | 811 | | |
793 | | - | |
| 812 | + | |
794 | 813 | | |
795 | 814 | | |
796 | 815 | | |
| |||
1395 | 1414 | | |
1396 | 1415 | | |
1397 | 1416 | | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
1398 | 1429 | | |
1399 | 1430 | | |
1400 | 1431 | | |
| |||
1406 | 1437 | | |
1407 | 1438 | | |
1408 | 1439 | | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
1409 | 1443 | | |
1410 | 1444 | | |
1411 | 1445 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
| 155 | + | |
155 | 156 | | |
156 | 157 | | |
157 | 158 | | |
| |||
203 | 204 | | |
204 | 205 | | |
205 | 206 | | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
206 | 211 | | |
207 | 212 | | |
208 | 213 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
| 186 | + | |
186 | 187 | | |
187 | 188 | | |
188 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
527 | 527 | | |
528 | 528 | | |
529 | 529 | | |
530 | | - | |
| 530 | + | |
| 531 | + | |
531 | 532 | | |
532 | 533 | | |
533 | 534 | | |
| |||
631 | 632 | | |
632 | 633 | | |
633 | 634 | | |
634 | | - | |
635 | | - | |
636 | | - | |
637 | | - | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
638 | 640 | | |
639 | 641 | | |
640 | 642 | | |
| |||
828 | 830 | | |
829 | 831 | | |
830 | 832 | | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
831 | 844 | | |
832 | 845 | | |
833 | 846 | | |
| |||
0 commit comments