Skip to content

Commit 7a4a173

Browse files
andygroveandygrove
committed
revert: pace unconditionally; the Arrow-size gate is not a sound bound
Reverts edc5597. get_array_memory_size() measures Arrow-side allocation while the roll decision uses parquet's encoded-size estimate, so the gate could let the writer roll while pacing was still off, misaligning every later block. Also drops the unsupportable cross-writer row bound from the docs.
1 parent edc5597 commit 7a4a173

2 files changed

Lines changed: 62 additions & 151 deletions

File tree

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,7 @@ A write is eligible only when ALL of the following hold:
157157
| `write.parquet.bloom-filter-enabled.column.<col>` | unset or `false` |
158158
| `write.metadata.metrics.*` | any value (manifest metrics are re-derived on the JVM with Iceberg's own logic) |
159159
| `write.spark.fanout.enabled` | any value (the native writer implements both clustered and fanout modes) |
160-
| `write.target-file-size-bytes` | any value (the roll point can differ by less than 1000 rows; see accepted divergences) |
160+
| `write.target-file-size-bytes` | any value (the two writers can choose different roll points; see accepted divergences) |
161161
| data location URI scheme | `file`, `memory`, `s3`, `s3a`, `gs` |
162162
| partition spec | any |
163163
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) |
@@ -251,13 +251,17 @@ a data file but not what any reader computes from it:
251251
(iceberg-java names files `<partition>-<task>-<operation>-<count>`; iceberg-rust uses a
252252
process-local counter).
253253
- File rolling lands on the same row grid as iceberg-java but not necessarily on the same row.
254-
On both paths a data file only ever rolls on a 1000-row boundary of that file and so overshoots
255-
`write.target-file-size-bytes` by less than 1000 rows: iceberg-java re-checks the size every
256-
1000 rows (`RollingFileWriter.ROWS_DIVISOR`), and Comet hands the iceberg-rust writer rows in
257-
1000-row units, counted per partition file, so its per-`write` check falls on the same
258-
boundaries. What each writer compares against the target still differs — flushed bytes plus
259-
parquet-rs's estimate of the open row group, versus parquet-mr's file position plus its buffered
260-
size — so the same data and target can still roll one 1000-row step earlier or later.
254+
Both writers re-check the current file's size against `write.target-file-size-bytes` once
255+
every 1000 rows of that file (iceberg-java's `RollingFileWriter.ROWS_DIVISOR`; Comet hands the
256+
iceberg-rust writer rows in 1000-row units, per partition file, to get the same grid), so each
257+
writer rolls only on a 1000-row boundary of its own file.
258+
The shared grid is all that is shared. What each writer compares against the target differs —
259+
flushed bytes plus parquet-rs's estimate of the open row group, versus parquet-mr's file
260+
position plus its buffered size — and the two use different threshold comparisons. These are
261+
independent size estimates, so nothing bounds how far apart the two writers' roll points are:
262+
they may cross the target several grid steps apart, and the resulting files can differ in row
263+
count by an arbitrary number of 1000-row blocks. Do not rely on file-layout parity between the
264+
two writers; rely only on each file rolling on its own 1000-row boundary.
261265
- Compressed page bytes are implementation-defined: the codec and any explicit level are
262266
translated, but parquet-rs and parquet-mr embed different encoder implementations and
263267
defaults (zstd default levels, LZ4 framing), so byte-identical output is not achievable even

0 commit comments

Comments
 (0)