Repository navigation
Commit 3f9c68b
feat(reader): support execution-time predicates in ArrowReader
Execution engines learn restrictions while a query runs, such as a
completed join build's key range or an improving TopK or MIN/MAX bound.
Add RuntimePredicateSnapshot and RuntimePredicateProvider so engines can
publish them as Iceberg predicates, and
ArrowReaderBuilder::with_runtime_predicate_provider to apply them.
When a data-file task starts, the reader checks the provider's
generation. Only when it changed does the reader take a snapshot, push
NOT down to the leaves and bind the predicate. Refreshes are serialized,
so a generation is snapshotted and bound once, and the cache never moves
to an older generation. The bound predicate is shared, without copying,
by later tasks of the scan with the same generation and schema.
The runtime predicate is planned separately from the planned and
equality-delete predicates, so the shared bound tree is never copied.
They are then combined into one Arrow row filter, which decodes shared
columns once, and their row-group, bloom-filter and page-index
selections are intersected, row groups in linear time. A bloom filter is
read at most once per row group and column, even when both predicates
use it.
Runtime predicates are advisory. Provider, binding and planning
failures, and page-index failures, drop only the runtime part for that
file; the planned and delete filtering are unchanged. A predicate on a
column missing from the file (its initial default or null is invisible
to physical filters) or stored with a type that needs promotion (a
literal cast before promotion could round or overflow) is ignored for
that file. Positional deletes keep their original file positions.
The provider contract documents that data skipped under a published
predicate must never be required later, that a later None only stops
runtime pruning for tasks that start afterwards, that generations
increase monotonically and are never reused, and that provider,
snapshot, binding, planning and page-pruning failures are advisory.
The reader accepts the provider; passing one through TableScan is a
follow-up.
(cherry picked from commit 821fc65)1 parent 5b82ab9 commit 3f9c68b
7 files changed
Lines changed: 2254 additions & 121 deletions
File tree
- crates/iceberg
- src/arrow
- reader
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
82 | 83 | | |
83 | 84 | | |
84 | 85 | | |
| |||
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
96 | 104 | | |
97 | 105 | | |
98 | 106 | | |
| |||
124 | 132 | | |
125 | 133 | | |
126 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
127 | 138 | | |
128 | 139 | | |
129 | 140 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| |||
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
49 | 54 | | |
50 | 55 | | |
51 | 56 | | |
| 57 | + | |
52 | 58 | | |
53 | 59 | | |
54 | 60 | | |
| |||
58 | 64 | | |
59 | 65 | | |
60 | 66 | | |
| 67 | + | |
61 | 68 | | |
62 | 69 | | |
63 | 70 | | |
| |||
74 | 81 | | |
75 | 82 | | |
76 | 83 | | |
| 84 | + | |
77 | 85 | | |
78 | 86 | | |
79 | 87 | | |
| |||
119 | 127 | | |
120 | 128 | | |
121 | 129 | | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
122 | 145 | | |
123 | 146 | | |
124 | 147 | | |
| |||
159 | 182 | | |
160 | 183 | | |
161 | 184 | | |
| 185 | + | |
162 | 186 | | |
163 | 187 | | |
164 | 188 | | |
| |||
177 | 201 | | |
178 | 202 | | |
179 | 203 | | |
| 204 | + | |
180 | 205 | | |
181 | 206 | | |
0 commit comments