Skip to content

Commit 3398c88

Browse files
committed
Read only the columns with a choice to make, lazily, when choosing dictionaries
The chooser expanded every nested column's entries into vectors of child indices before it looked at the leaf, even for leaves with no choice to make: 1000 rows of 8192-element boolean lists, about 1 MiB of input, cost 128 MiB of temporary allocation. It also kept every column's page state, including one id per value, until the last column was judged. It now reads only the leaf columns there is a choice for, one at a time, and walks each through a lazy iterator that stops where the column's first page ends. The RLE-hybrid length is counted from whether each id repeats the one before it, so no per-value ids are kept, and what the chooser holds is one column's dictionary. Decisions are unchanged; the parquet-mr ground-truth tests pass as before.
1 parent 32f2c0d commit 3398c88

2 files changed

Lines changed: 355 additions & 208 deletions

File tree

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,9 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
225225
capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before any of them
226226
reaches the writer. `DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting
227227
over those rows and turns dictionary encoding off for the columns parquet-mr would write plain.
228-
The accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
228+
It reads only the columns there is a choice for, one at a time, and walks each lazily only as far
229+
as its first page, so beyond the held rows it keeps no more than one column's dictionary. The
230+
accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
229231
`RunLengthBitPackingHybridEncoder`, and its tests pin answers recorded from parquet-mr itself. The
230232
held rows then go through `RowPacer`, so the roll grid does not move.
231233
- **Field ids and casting.** `decorate_batch_with_field_ids` casts each batch to the

0 commit comments

Comments
 (0)