Repository navigation
Commit 93941b1
authored
[improvement](parquet) Fuse nested sparse selection planning (apache#66263)
Parquet V2 repeated leaves currently rebuild a parent-row filter into a level-entry filter, scan definition levels into null runs plus an ancestor-null hash set, build the leaf selection, and scan the levels again to compact surviving parent shapes. This fixed O(level entries) work is especially visible when only 1% or 10% of parent rows survive.
This change adds a single-pass nested selection builder. It maps parent filters to repeated entries, classifies ancestor and leaf nulls, builds the decoder selection, appends selected nulls, and compacts repetition/definition levels in place. Ancestor-null placeholders remain in the surviving parent shape but do not consume leaf selection entries. Unfiltered and fully filtered paths keep their existing behavior.1 parent af6dcff commit 93941b1
10 files changed
Lines changed: 789 additions & 36 deletions
File tree
- be
- benchmark/parquet
- src/format_v2/parquet/reader/native
- test/format_v2/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
127 | 130 | | |
128 | 131 | | |
129 | | - | |
| 132 | + | |
130 | 133 | | |
131 | 134 | | |
132 | 135 | | |
| |||
299 | 302 | | |
300 | 303 | | |
301 | 304 | | |
302 | | - | |
| 305 | + | |
303 | 306 | | |
304 | 307 | | |
305 | 308 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
43 | 43 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | 54 | | |
52 | 55 | | |
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
56 | 78 | | |
57 | 79 | | |
58 | 80 | | |
| |||
0 commit comments