You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 27f3629
Browse filesBrowse the repository at this point in the historyBrowse files
iceberg-rust's calculator of the same name, and `PartitionSplitter` replaces its
@@ -264,9 +266,10 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
264
266
column's encoding when the file opens
265
267
([#6114](https://github.com/apache/datafusion-comet/issues/6114)). So each partition's
266
268
`PartitionFeed` holds back the partition's first page of rows (`write.parquet.page-row-limit`,
267
-
capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before any of them
268
-
reaches the writer. `DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting
269
-
over those rows and turns dictionary encoding off for the columns parquet-mr would write plain.
269
+
at least 100, capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before
270
+
any of them reaches the writer, unless no column has a dictionary choice to make.
271
+
`DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting over those rows and
272
+
turns dictionary encoding off for the columns parquet-mr would write plain.
270
273
It reads only the columns there is a choice for, one at a time, and walks each lazily only as far
271
274
as its first page, so beyond the held rows it keeps no more than one column's dictionary. The
272
275
accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
@@ -347,7 +350,8 @@ backend ignores them. A fallback reason must contain property names only, never
347
350
Each task emits exactly one batch with one row and two `BINARY` columns (`build_output_schema`):
348
351
349
352
1. The task's `DataFile`s encoded as an Iceberg v2 data manifest, written by iceberg-rust's
350
-
`ManifestWriter` into an in-process `memory:``FileIO`. Empty when the task wrote no files.
353
+
`ManifestWriter` into an in-process `memory:``FileIO`. A task that wrote no files sends a
354
+
manifest with no entries.
351
355
2. Every location the task's writers were handed (`encode_locations`: a big-endian count, then a
352
356
length-prefixed UTF-8 string per location).
353
357
@@ -368,7 +372,8 @@ Each task emits exactly one batch with one row and two `BINARY` columns (`build_
368
372
Rebuilding metrics on the JVM is the main reason manifest parity holds: metrics modes, truncation,
369
373
the inferred-column cap and list/map bounds suppression are decided by iceberg-java's code. When
370
374
touching the native writer, do not try to make iceberg-rust's own metrics authoritative; the JVM
371
-
discards them. The price is one footer read per written file.
375
+
discards them. The price is reading each written file's footer back from storage, two small reads
376
+
per file.
372
377
373
378
## Failure Handling and Cleanup Ownership
374
379
@@ -438,7 +443,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
|`CometIcebergWriteActionSuite`| End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, writes outgrowing the memory pool, AQE re-planning. |
441
-
|`CometIcebergWriteDetectionSuite`|One case per eligibility rule, accepted and declined. |
446
+
|`CometIcebergWriteDetectionSuite`|Most eligibility rules, each accepted and declined.|
442
447
|`CometIcebergSystemFunctionSuite`| Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
443
448
|`CometIcebergRewriteActionSuite`| Iceberg's `rewrite_data_files` with the split plan and the native writer. |
444
449
|`IcebergWriteProtoTranslationSuite`| Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
@@ -448,9 +453,11 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
448
453
|`CometIcebergEndToEndBenchmark`| An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. |
449
454
450
455
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
451
-
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0 and 4.2, and 1.11.0 for 4.1. Only the default profile
452
-
runs on every pull request. Behaviour that differs between Iceberg versions, such as reflection
453
-
targets, metrics conventions and partition path spelling, needs a test that runs on each.
456
+
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published
457
+
for Spark 4.2, so the 4.2 profile builds against the 4.0 runtime and the Iceberg suites skip there
458
+
(`icebergAvailable` is false on Spark 4.2). Only the default profile runs on every pull request.
459
+
Behaviour that differs between Iceberg versions, such as reflection targets, metrics conventions
460
+
and partition path spelling, needs a test that runs on each.
454
461
455
462
When writing a native-write test:
456
463
@@ -471,9 +478,10 @@ When writing a native-write test:
471
478
472
479
The upstream Iceberg Spark tests also run with the native writer and `localTableScan` enabled (see
473
480
[Running Iceberg Spark Tests](iceberg-spark-tests.md)). They are a broad regression net, but they do
474
-
not assert which writer ran, and Comet's fallback reasons do not appear in their CI logs, so a green
475
-
run is not evidence that the native writer handled a given test
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/s3-credential-provider-design.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -154,7 +154,7 @@ The cache holds the `FileIO` and its bridge. The bridge reuses its credential un
154
154
155
155
## Property-bag handling on the Parquet path
156
156
157
-
The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects `fs.s3a.*` (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present.
157
+
The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects the object store prefixes of the schemes the scan reads (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present.
0 commit comments