Repository navigation
Commit 7425780
authored
perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)
* perf: choose dictionary encoding like iceberg-java in the native Iceberg writer
parquet-mr, which iceberg-java writes through, drops dictionary encoding for a
column chunk when the chunk's first data page shows the dictionary saving no
space, and then writes no dictionary page. parquet-rs keeps the dictionary until
it fills and still writes the page, so a high-cardinality column in a
natively written file carried a dictionary page of up to
write.parquet.dict-size-bytes that every selective read had to fetch.
The native writer now makes parquet-mr's decision before it opens a partition's
first file. Each partition's first page of rows is held back, DictionaryChooser
replays parquet-mr's first-page size accounting over those rows, and the
partition's files are opened with dictionary encoding off for the columns
parquet-mr would write plain. Across all of a task's partitions, the held rows
are bounded by the row group size. The roll grid does not move.
Closes #6114.
* Read only the columns with a choice to make, lazily, when choosing dictionaries
The chooser expanded every nested column's entries into vectors of child
indices before it looked at the leaf, even for leaves with no choice to make:
1000 rows of 8192-element boolean lists, about 1 MiB of input, cost 128 MiB of
temporary allocation. It also kept every column's page state, including one id
per value, until the last column was judged.
It now reads only the leaf columns there is a choice for, one at a time, and
walks each through a lazy iterator that stops where the column's first page
ends. The RLE-hybrid length is counted from whether each id repeats the one
before it, so no per-value ids are kept, and what the chooser holds is one
column's dictionary. Decisions are unchanged; the parquet-mr ground-truth tests
pass as before.
* Fix the Iceberg write suite's build after merging main
Main's suite now imports java.nio.file.Path, which made this PR's
org.apache.hadoop.fs.Path import ambiguous. Import the Hadoop class as
HadoopPath instead, and put the Parquet imports in the org.apache group
that spotless expects.
* Give the mid-write retry test a first page that ends before its failure
The test from #6111 needs the native writer to have finalized a data
file by row 7000 of 10000. This PR holds each partition's first data
page of rows back for the dictionary choice, and the default page is
20000 rows, so nothing reached disk before the injected failure. A
1000-row page keeps the test on its original timeline.1 parent 83285bb commit 7425780
6 files changed
Lines changed: 1975 additions & 92 deletions
File tree
- docs/source
- contributor-guide
- user-guide/latest
- native/core/src/execution/operators
- spark/src/test/scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
208 | | - | |
209 | | - | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
210 | 211 | | |
211 | 212 | | |
212 | | - | |
213 | | - | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
214 | 216 | | |
215 | 217 | | |
216 | 218 | | |
| |||
239 | 241 | | |
240 | 242 | | |
241 | 243 | | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
242 | 257 | | |
243 | 258 | | |
244 | 259 | | |
| |||
351 | 366 | | |
352 | 367 | | |
353 | 368 | | |
| 369 | + | |
354 | 370 | | |
355 | 371 | | |
356 | 372 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
70 | 76 | | |
71 | 77 | | |
72 | 78 | | |
| |||
299 | 305 | | |
300 | 306 | | |
301 | 307 | | |
302 | | - | |
303 | | - | |
304 | | - | |
305 | | - | |
306 | | - | |
307 | | - | |
308 | | - | |
309 | | - | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
310 | 321 | | |
311 | 322 | | |
312 | 323 | | |
| |||
0 commit comments