Skip to content

Commit a0cbb0f

Browse files
feat: bump iceberg-rust and write float and double identity partitions natively (#6851)
* feat: bump iceberg-rust and write float and double identity partitions natively apache/iceberg-rust#3327 gives `PrimitiveLiteral` an equality and ordering that follow Java's `Float.compare` / `Double.compare`, so iceberg-rust's writers keep -0.0 and 0.0 partitions apart, as iceberg-java does. Move the pin from af1da4c to 1f3bc34, the merge commit of that PR, and drop the fallback that #6531 added for float and double partition fields (#6138). Two other upstream changes in the range need code changes: - apache/iceberg-rust#3145 makes `FileScanTaskDeleteFile`'s fields private behind a validating builder, and holds the deletion-vector offset and size as u64. The planner builds delete files through the builder and rejects negative coordinates, and the scan sizes a delete file by rebuilding it. - apache/iceberg-rust#3323 fixes iceberg-rust's `day` for pre-epoch timestamps (apache/iceberg-rust#3315). Comet computes those partition values itself, so only the canary that pins iceberg-rust's values changes. * test: cover NaN and same-typed decimal identity partitions The lifted gate also admits NaN partition values, so the signed-zero test writes NaN and -NaN through both writers too. The bump changes how iceberg-rust writes a named Avro type that two partition fields share, so a new test writes two decimal identity partitions of the same type, in both encodings, and compares them with iceberg-java. The detection test now uses plain tuples. Scala 2.12 flagged the adapted argument list as an error under -Pstrict-warnings. * refactor: share the delete-file count conversion and trim the new tests - One closure converts a delete file's record count and blob coordinates from the proto's int64, instead of three copies of the same block. - The planner test keeps the cases Comet's own conversion decides, and drops the two that only exercised iceberg-rust's validation messages. - `committedPartitions` sits after `partitionDirs`, replaces the far-date test's local copy of the same query, and is read once per assertion. - The detection test uses the suite's `createTable` table, so `assertSupportLevelIs` keeps its original signature. * test: check the clustered writer rejects interleaved signed zeros like iceberg-java * test: cover a missing or negative deletion vector record count; drop the fixed day difference from the guide * Update native/core/src/execution/operators/iceberg_scan.rs --------- Co-authored-by: Matt Butrovich <[email protected]>
1 parent 79ab485 commit a0cbb0f

12 files changed

Lines changed: 417 additions & 282 deletions

File tree

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -111,8 +111,8 @@ only the latest.
111111
that starts trusting iceberg-rust's metrics, or adds a native-tracked value beyond float and
112112
double NaN counts and bounds, needs a strong reason.
113113
- [ ] **Value semantics match, not only formats.** iceberg-rust compares and hashes partition values
114-
with its own rules: `OrderedFloat` treats `-0.0` and `0.0` as equal where Java's
115-
`Float.compare` does not
114+
with its own rules: until apache/iceberg-rust#3327 its `OrderedFloat` equality treated
115+
`-0.0` and `0.0` as equal where Java's `Float.compare` does not
116116
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check equality, hashing,
117117
ordering and rendering for float, double, timestamp, timestamptz, binary and decimal.
118118
- [ ] **Timestamp partition values are UTC.** Iceberg's `years`, `months`, `days` and `hours`

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -253,9 +253,8 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
253253
NULL past `chrono`'s calendar ([#6145](https://github.com/apache/datafusion-comet/issues/6145)).
254254
All four of its transforms floor a pre-1970 timestamp that lies exactly 999999 microseconds into a
255255
unit, which iceberg-java puts in the unit before
256-
([#6426](https://github.com/apache/datafusion-comet/issues/6426)), and its `day` puts some
257-
timestamps from the last second before a pre-1970 midnight in the next day. Every other transform
258-
stays on iceberg-rust.
256+
([#6426](https://github.com/apache/datafusion-comet/issues/6426)). Every other transform stays on
257+
iceberg-rust.
259258
- **File names.** `file_name_prefix` embeds the partition id, the task attempt id and the operation
260259
id, so a retried or speculative attempt never reuses another attempt's file names.
261260
- **Row pacing.** iceberg-java's rolling writer checks the target file size every 1000 rows of the
@@ -424,8 +423,9 @@ Useful places to look when checking parity:
424423
- `CometIcebergWriteActionSuite` writes the same data through both writers into sibling tables and
425424
compares rows, `readable_metrics` and partition paths.
426425
- iceberg-rust's own code, at the pinned revision. iceberg-rust makes different choices from
427-
iceberg-java in places that matter to the table's contents, for example grouping partition keys
428-
with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
426+
iceberg-java in places that matter to the table's contents. For example, until
427+
[apache/iceberg-rust#3327](https://github.com/apache/iceberg-rust/pull/3327) it grouped
428+
partition keys with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
429429
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check how iceberg-rust
430430
compares, hashes and renders values, not only what it writes.
431431

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 2 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ A write is eligible only when ALL of the following hold:
216216
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` |
217217
| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back, apart from the ignored settings listed below |
218218
| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` / `client.*` properties are also forwarded; unsupported Iceberg-defined S3 settings still fall back |
219-
| partition spec | any, except an identity partition on a `float` or `double` column, or a `void` field whose source column was dropped beside a live field (see below) |
219+
| partition spec | any, except a `void` field whose source column was dropped beside a live field (see below) |
220220
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) and the v3 types `variant`, `unknown`, `timestamp_ns`, `geometry` and `geography` |
221221

222222
Within the namespaces that shape data-file bytes — `write.parquet.*` and `parquet.*` —
@@ -272,13 +272,6 @@ native write. If Iceberg's AWS property classes cannot be loaded, vendor `s3.*`
272272
fall back too and planning still completes. The fall-back reason reports only sorted property
273273
names, never their values, so credentials and tokens do not enter EXPLAIN or plan logs.
274274

275-
An identity partition on a `float` or `double` column falls back. iceberg-rust compares float
276-
partition values with an equality that treats `-0.0` and `0.0` as one value, so the native writer
277-
would put rows with either value in the same partition, where iceberg-java writes two. A read
278-
that prunes on the other value's partition would then miss rows. The fall-back stays until
279-
iceberg-rust distinguishes the two values
280-
([apache/iceberg-rust#3325](https://github.com/apache/iceberg-rust/issues/3325)).
281-
282275
On a format-version 3 table with Iceberg 1.10 or newer, a write that rewrites existing rows falls
283276
back. Copy-on-write `DELETE`, `UPDATE` and `MERGE` and `rewrite_data_files` write the row lineage
284277
columns `_row_id` and `_last_updated_sequence_number` into the new data files, so that rewritten
@@ -429,9 +422,7 @@ a data file but not what any reader computes from it:
429422
per task. The native writer zero-pads the task attempt id and counts from `00000`; iceberg-java
430423
leaves the id unpadded and counts from `00001`.
431424
- Partition directory names match iceberg-java 1.8+'s `PartitionSpec.partitionToPath` for every
432-
partition type the native writer accepts. Identity partitions on `float` and `double` columns
433-
fall back (see [Native Parquet write eligibility](#native-parquet-write-eligibility)), so
434-
iceberg-java names those directories itself. On Iceberg 1.5.x,
425+
partition type the native writer accepts. On Iceberg 1.5.x,
435426
which the Spark 3.4 profile pins, iceberg-java itself spelled `timestamp` and `timestamptz`
436427
directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()`
437428
(`ts=1969-12-31T23:59:58.500Z`) and left the partition field name unescaped; Comet uses the

‎native/Cargo.lock‎

Lines changed: 97 additions & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎native/Cargo.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,8 +85,8 @@ aws-sdk-sts = { version = "1.114.0", default-features = false }
8585
# Pinned to a revision rather than a release because no published iceberg-rust
8686
# version is on our arrow: 0.10.1 requires arrow/parquet ^58. Move to a plain
8787
# version requirement once a release ships on arrow 59. Bump policy: #5645.
88-
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "af1da4c5bc86178c38c1c6db0578bdd9fb7021e3" }
89-
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "af1da4c5bc86178c38c1c6db0578bdd9fb7021e3", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
88+
iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b" }
89+
iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust", rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b", features = ["opendal-memory", "opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
9090
reqsign-core = "3"
9191

9292
[profile.release]

‎native/core/src/execution/operators/iceberg_partition_value.rs‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ use iceberg::{Error, ErrorKind, Result};
3939
/// `timestamp`, or `timestamptz` source, which go through Comet's `iceberg_years` /
4040
/// `iceberg_months` / `iceberg_days` / `iceberg_hours` kernels instead: the ones the sort in front
4141
/// of a clustered write runs, pinned against iceberg-java's `DateTimeUtil` over the whole domain.
42-
/// iceberg-rust's transforms differ from iceberg-java's in three ways:
42+
/// iceberg-rust's transforms differ from iceberg-java's in two ways:
4343
///
4444
/// - `year` and `month` split the calendar with Arrow's `date_part`, which returns NULL for
4545
/// anything `chrono` cannot represent -- past year 262142 -- whereas iceberg-java goes through
@@ -49,9 +49,6 @@ use iceberg::{Error, ErrorKind, Result};
4949
/// - All four floor a pre-epoch timestamp that lies exactly 999999 microseconds into a unit, which
5050
/// iceberg-java puts in the unit before, so `1969-01-01T00:00:00.999999` belongs in the 1968
5151
/// partitions (apache/datafusion-comet#6426).
52-
/// - `day` moves a timestamp from the last second of a day before 1969-12-31 into the next day,
53-
/// unless its microsecond of second is 0 or 999999: it takes the whole seconds with a truncating
54-
/// division and the microseconds with a flooring one (apache/iceberg-rust#3315).
5552
///
5653
/// A partition value that differs from the sort key can also fail a clustered write, which rejects
5754
/// a row whose partition it has already closed. Everywhere else the two implementations agree, so
@@ -393,7 +390,8 @@ mod tests {
393390
Some(-31_535_999_000_001),
394391
// 1969-12-31T23:00:00.999999, where that moves only the hour.
395392
Some(-3_599_000_001),
396-
// 1969-12-30T23:59:59.5 and 1969-12-30T23:59:59.999998.
393+
// 1969-12-30T23:59:59.5 and 1969-12-30T23:59:59.999998, which iceberg-rust's `day`
394+
// moved into 1969-12-31 until apache/iceberg-rust#3315 was fixed.
397395
Some(-86_400_500_000),
398396
Some(-86_400_000_002),
399397
None,
@@ -419,7 +417,7 @@ mod tests {
419417
(
420418
Transform::Day,
421419
day([-366, -1, -2, -2]),
422-
day([-365, -1, -1, -1]),
420+
day([-365, -1, -2, -2]),
423421
),
424422
(
425423
Transform::Hour,
@@ -438,8 +436,7 @@ mod tests {
438436
for (i, (transform, java, rust)) in cases.iter().enumerate() {
439437
let label = format!("{transform} of {}", source.data_type());
440438
assert_eq!(&comet[i], java, "{label}");
441-
// The reason Comet computes these: iceberg-rust floors the first two rows, and its
442-
// `day` moves the last two into 1969-12-31 (apache/iceberg-rust#3315). If this
439+
// The reason Comet computes these: iceberg-rust floors the first two rows. If this
443440
// starts failing, iceberg-rust's transforms have changed and delegating needs
444441
// another look.
445442
assert_eq!(&iceberg_rust[i], rust, "iceberg-rust's {label}");

0 commit comments

Comments
 (0)