Skip to content

Commit 7425780

Browse files
authored
perf: choose dictionary encoding like iceberg-java in the native Iceberg writer (#6241)
* perf: choose dictionary encoding like iceberg-java in the native Iceberg writer parquet-mr, which iceberg-java writes through, drops dictionary encoding for a column chunk when the chunk's first data page shows the dictionary saving no space, and then writes no dictionary page. parquet-rs keeps the dictionary until it fills and still writes the page, so a high-cardinality column in a natively written file carried a dictionary page of up to write.parquet.dict-size-bytes that every selective read had to fetch. The native writer now makes parquet-mr's decision before it opens a partition's first file. Each partition's first page of rows is held back, DictionaryChooser replays parquet-mr's first-page size accounting over those rows, and the partition's files are opened with dictionary encoding off for the columns parquet-mr would write plain. Across all of a task's partitions, the held rows are bounded by the row group size. The roll grid does not move. Closes #6114. * Read only the columns with a choice to make, lazily, when choosing dictionaries The chooser expanded every nested column's entries into vectors of child indices before it looked at the leaf, even for leaves with no choice to make: 1000 rows of 8192-element boolean lists, about 1 MiB of input, cost 128 MiB of temporary allocation. It also kept every column's page state, including one id per value, until the last column was judged. It now reads only the leaf columns there is a choice for, one at a time, and walks each through a lazy iterator that stops where the column's first page ends. The RLE-hybrid length is counted from whether each id repeats the one before it, so no per-value ids are kept, and what the chooser holds is one column's dictionary. Decisions are unchanged; the parquet-mr ground-truth tests pass as before. * Fix the Iceberg write suite's build after merging main Main's suite now imports java.nio.file.Path, which made this PR's org.apache.hadoop.fs.Path import ambiguous. Import the Hadoop class as HadoopPath instead, and put the Parquet imports in the org.apache group that spotless expects. * Give the mid-write retry test a first page that ends before its failure The test from #6111 needs the native writer to have finalized a data file by row 7000 of 10000. This PR holds each partition's first data page of rows back for the dictionary choice, and the default page is 20000 rows, so nothing reached disk before the injected failure. A 1000-row page keeps the test on its original timeline.
1 parent 83285bb commit 7425780

6 files changed

Lines changed: 1975 additions & 92 deletions

File tree

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -205,12 +205,14 @@ columns.
205205
### Native execution
206206

207207
The planner builds `IcebergWriteExec` (`native/core/src/execution/operators/iceberg_write.rs`) over
208-
the FFI scan of the child's batches. `run_write_task` builds iceberg-rust's writer stack once per
209-
task:
208+
the FFI scan of the child's batches. `run_write_task` builds iceberg-rust's partitioning writer once
209+
per task, and `PartitionWriterBuilder` builds the rest of the stack once per partition, with that
210+
partition's Parquet writer properties:
210211

211212
```text
212-
ParquetWriterBuilder -> RollingFileWriterBuilder -> DataFileWriterBuilder
213-
-> UnpartitionedWriter | FanoutWriter | ClusteredWriter
213+
UnpartitionedWriter | FanoutWriter | ClusteredWriter
214+
-> PartitionWriterBuilder, per partition:
215+
ParquetWriterBuilder -> RollingFileWriterBuilder -> DataFileWriterBuilder
214216
```
215217

216218
Points where Comet adapts iceberg-rust to match iceberg-java:
@@ -239,6 +241,19 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
239241
- **Row pacing.** iceberg-java's rolling writer checks the target file size every 1000 rows of the
240242
current file. iceberg-rust checks once per `write` call. `RowPacer` hands the writer rows in
241243
`ROWS_DIVISOR` (1000) row units per file, so both writers roll on the same grid.
244+
- **Dictionary choice.** parquet-mr writes a column chunk plain, with no dictionary page, when its
245+
first data page shows the dictionary saving no space. parquet-rs has no such check, and fixes a
246+
column's encoding when the file opens
247+
([#6114](https://github.com/apache/datafusion-comet/issues/6114)). So each partition's
248+
`PartitionFeed` holds back the partition's first page of rows (`write.parquet.page-row-limit`,
249+
capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before any of them
250+
reaches the writer. `DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting
251+
over those rows and turns dictionary encoding off for the columns parquet-mr would write plain.
252+
It reads only the columns there is a choice for, one at a time, and walks each lazily only as far
253+
as its first page, so beyond the held rows it keeps no more than one column's dictionary. The
254+
accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
255+
`RunLengthBitPackingHybridEncoder`, and its tests pin answers recorded from parquet-mr itself. The
256+
held rows then go through `RowPacer`, so the roll grid does not move.
242257
- **Field ids and casting.** `decorate_batch_with_field_ids` casts each batch to the
243258
field-id-annotated Arrow schema derived from the Iceberg schema, with `safe: false`, so a type
244259
mismatch fails the task instead of writing NULLs.
@@ -351,6 +366,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
351366
| `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. |
352367
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
353368
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past `chrono`'s calendar. |
369+
| Rust tests in `iceberg_dictionary.rs` | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
354370
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
355371

356372
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,12 @@ the same `TaskCommit` message the JVM writer would have produced. Everything ice
6767
post-write — snapshot assignment, manifest-list aggregation, commit validation and retries —
6868
is untouched: `IcebergCommit` performs the normal `BatchWrite.commit`.
6969

70+
Before a task opens a partition's first file, it holds that partition's first
71+
`write.parquet.page-row-limit` rows in memory, so that it can choose which columns to
72+
dictionary-encode the way iceberg-java would (see the accepted divergences below). The rows a
73+
task holds back this way, across all of its partitions, stay within about
74+
`write.parquet.row-group-size-bytes`.
75+
7076
## Configuration
7177

7278
Standard Comet + Iceberg setup (see [`iceberg.md`](iceberg.md)) plus the write-side toggle:
@@ -299,14 +305,19 @@ a data file but not what any reader computes from it:
299305
- Dictionary-encoded pages are labeled `RLE_DICTIONARY` (parquet-mr v1 files: `PLAIN_DICTIONARY`).
300306
- Fixed-length binary columns (`uuid`, `fixed`, decimals with precision > 18) are not
301307
dictionary-encoded (parquet-mr dictionary-encodes them).
302-
- High-cardinality columns keep a dictionary page. parquet-mr abandons dictionary encoding for a
303-
column chunk, and writes no dictionary page, when the first check shows the dictionary is not
304-
saving space. parquet-rs keeps dictionary encoding until the dictionary reaches
305-
`write.parquet.dict-size-bytes` (2 MB by default), then switches to plain encoding for the rest
306-
of the chunk and still writes the dictionary page. Results are the same, but a selective read of
307-
a native-written file fetches that dictionary page for every column chunk it touches, so it
308-
reads more bytes than it would from an iceberg-java file
309-
([#6114](https://github.com/apache/datafusion-comet/issues/6114)).
308+
- Which columns are dictionary-encoded is decided as parquet-mr decides it: a column whose first
309+
data page shows the dictionary saving no space is written plain, with no dictionary page,
310+
instead of carrying a dictionary page that every selective read of it would have to fetch
311+
([#6114](https://github.com/apache/datafusion-comet/issues/6114)). The native writer decides
312+
once per partition, from that partition's first page of rows in the task, and keeps the
313+
decision for every file and row group it writes for the partition; parquet-mr decides again
314+
for every row group. Where the page size rather than `write.parquet.page-row-limit` ends a
315+
column's first page, the native page ends at the first row past parquet-mr's size threshold,
316+
while parquet-mr only ends it at its next periodic size check, so a column close to the
317+
cut-off can be decided the other way. Close to the cut-off both encodings take about the same
318+
space. A column that keeps its dictionary and later fills it falls back to plain on both
319+
writers, but parquet-rs's dictionary page then holds every entry, where parquet-mr's holds
320+
only the entries earlier pages used.
310321
- Row-group boundaries: parquet-mr flushes by byte size at a record-count check cadence,
311322
parquet-rs buffers by row count. File naming follows the same cadence-style difference
312323
(iceberg-java names files `<partition>-<task>-<operation>-<count>`; iceberg-rust uses a

0 commit comments

Comments
 (0)