You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a0cbb0f
Browse filesBrowse the repository at this point in the historyBrowse files
feat: bump iceberg-rust and write float and double identity partitions natively (#6851)
* feat: bump iceberg-rust and write float and double identity partitions natively
apache/iceberg-rust#3327 gives `PrimitiveLiteral` an equality and ordering that
follow Java's `Float.compare` / `Double.compare`, so iceberg-rust's writers keep
-0.0 and 0.0 partitions apart, as iceberg-java does. Move the pin from af1da4c to
1f3bc34, the merge commit of that PR, and drop the fallback that #6531 added for
float and double partition fields (#6138).
Two other upstream changes in the range need code changes:
- apache/iceberg-rust#3145 makes `FileScanTaskDeleteFile`'s fields private
behind a validating builder, and holds the deletion-vector offset and size as
u64. The planner builds delete files through the builder and rejects negative
coordinates, and the scan sizes a delete file by rebuilding it.
- apache/iceberg-rust#3323 fixes iceberg-rust's `day` for pre-epoch timestamps
(apache/iceberg-rust#3315). Comet computes those partition values itself, so
only the canary that pins iceberg-rust's values changes.
* test: cover NaN and same-typed decimal identity partitions
The lifted gate also admits NaN partition values, so the signed-zero test
writes NaN and -NaN through both writers too. The bump changes how
iceberg-rust writes a named Avro type that two partition fields share, so a
new test writes two decimal identity partitions of the same type, in both
encodings, and compares them with iceberg-java.
The detection test now uses plain tuples. Scala 2.12 flagged the adapted
argument list as an error under -Pstrict-warnings.
* refactor: share the delete-file count conversion and trim the new tests
- One closure converts a delete file's record count and blob coordinates
from the proto's int64, instead of three copies of the same block.
- The planner test keeps the cases Comet's own conversion decides, and
drops the two that only exercised iceberg-rust's validation messages.
- `committedPartitions` sits after `partitionDirs`, replaces the far-date
test's local copy of the same query, and is read once per assertion.
- The detection test uses the suite's `createTable` table, so
`assertSupportLevelIs` keeps its original signature.
* test: check the clustered writer rejects interleaved signed zeros like iceberg-java
* test: cover a missing or negative deletion vector record count; drop the fixed day difference from the guide
* Update native/core/src/execution/operators/iceberg_scan.rs
---------
Co-authored-by: Matt Butrovich <[email protected]>
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/iceberg-writes.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -253,9 +253,8 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
253
253
NULL past `chrono`'s calendar ([#6145](https://github.com/apache/datafusion-comet/issues/6145)).
254
254
All four of its transforms floor a pre-1970 timestamp that lies exactly 999999 microseconds into a
255
255
unit, which iceberg-java puts in the unit before
256
-
([#6426](https://github.com/apache/datafusion-comet/issues/6426)), and its `day` puts some
257
-
timestamps from the last second before a pre-1970 midnight in the next day. Every other transform
258
-
stays on iceberg-rust.
256
+
([#6426](https://github.com/apache/datafusion-comet/issues/6426)). Every other transform stays on
257
+
iceberg-rust.
259
258
-**File names.**`file_name_prefix` embeds the partition id, the task attempt id and the operation
260
259
id, so a retried or speculative attempt never reuses another attempt's file names.
261
260
-**Row pacing.** iceberg-java's rolling writer checks the target file size every 1000 rows of the
@@ -424,8 +423,9 @@ Useful places to look when checking parity:
424
423
-`CometIcebergWriteActionSuite` writes the same data through both writers into sibling tables and
425
424
compares rows, `readable_metrics` and partition paths.
426
425
- iceberg-rust's own code, at the pinned revision. iceberg-rust makes different choices from
427
-
iceberg-java in places that matter to the table's contents, for example grouping partition keys
428
-
with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
426
+
iceberg-java in places that matter to the table's contents. For example, until
427
+
[apache/iceberg-rust#3327](https://github.com/apache/iceberg-rust/pull/3327) it grouped
428
+
partition keys with `OrderedFloat`, which treats `-0.0` and `0.0` as equal
429
429
([#6138](https://github.com/apache/datafusion-comet/issues/6138)). Check how iceberg-rust
430
430
compares, hashes and renders values, not only what it writes.
| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back, apart from the ignored settings listed below |
218
218
| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` / `client.*` properties are also forwarded; unsupported Iceberg-defined S3 settings still fall back |
219
-
| partition spec | any, except an identity partition on a `float` or `double` column, or a `void` field whose source column was dropped beside a live field (see below) |
219
+
| partition spec | any, except a `void` field whose source column was dropped beside a live field (see below)|
220
220
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) and the v3 types `variant`, `unknown`, `timestamp_ns`, `geometry` and `geography`|
221
221
222
222
Within the namespaces that shape data-file bytes — `write.parquet.*` and `parquet.*` —
@@ -272,13 +272,6 @@ native write. If Iceberg's AWS property classes cannot be loaded, vendor `s3.*`
272
272
fall back too and planning still completes. The fall-back reason reports only sorted property
273
273
names, never their values, so credentials and tokens do not enter EXPLAIN or plan logs.
274
274
275
-
An identity partition on a `float` or `double` column falls back. iceberg-rust compares float
276
-
partition values with an equality that treats `-0.0` and `0.0` as one value, so the native writer
277
-
would put rows with either value in the same partition, where iceberg-java writes two. A read
278
-
that prunes on the other value's partition would then miss rows. The fall-back stays until
0 commit comments