Repository navigation
Commit 3398c88
committed
Read only the columns with a choice to make, lazily, when choosing dictionaries
The chooser expanded every nested column's entries into vectors of child
indices before it looked at the leaf, even for leaves with no choice to make:
1000 rows of 8192-element boolean lists, about 1 MiB of input, cost 128 MiB of
temporary allocation. It also kept every column's page state, including one id
per value, until the last column was judged.
It now reads only the leaf columns there is a choice for, one at a time, and
walks each through a lazy iterator that stops where the column's first page
ends. The RLE-hybrid length is counted from whether each id repeats the one
before it, so no per-value ids are kept, and what the chooser holds is one
column's dictionary. Decisions are unchanged; the parquet-mr ground-truth tests
pass as before.1 parent 32f2c0d commit 3398c88
2 files changed
Lines changed: 355 additions & 208 deletions
File tree
- docs/source/contributor-guide
- native/core/src/execution/operators
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
228 | | - | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
229 | 231 | | |
230 | 232 | | |
231 | 233 | | |
| |||
0 commit comments