Skip to content

Commit 27f3629

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/6264-dpp-stage-reuse
2 parents 4ac0892 + 5e7d918 commit 27f3629

75 files changed

Lines changed: 6673 additions & 586 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,10 +50,11 @@ not, inherits it. There is no query that fails to tell you.
5050
| Shared with the scan | `iceberg_common.rs` (`load_file_io`, `storage_factory_for`, `scheme_of`), `IcebergReflection`, `NativeConfig` S3 translation |
5151

5252
`spark.comet.write.iceberg.enabled` switches both write layers. It is off by default, but once a
53-
user turns it on, a change to the split plan affects every Iceberg write, including writes that
54-
never reach the native writer, and a change to the native writer affects every eligible one. The
55-
testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the
56-
native writer off. A change to a shared file affects the native Iceberg scan too.
53+
user turns it on, a change to the split plan affects every append, overwrite and copy-on-write
54+
Iceberg write, including writes that never reach the native writer, and a change to the native
55+
writer affects every eligible one. The testing-only
56+
`spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the native writer
57+
off. A change to a shared file affects the native Iceberg scan too.
5758

5859
## 2. Direction of the Gate Change
5960

@@ -205,7 +206,7 @@ Read the ownership table in the contributor guide before reviewing any change ne
205206
| Suite | Covers |
206207
| ----------------------------------- | ----------------------------------------------------------------------------------------------- |
207208
| `CometIcebergWriteActionSuite` | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans |
208-
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule |
209+
| `CometIcebergWriteDetectionSuite` | Most eligibility rules, accepted and declined |
209210
| `CometIcebergSystemFunctionSuite` | Native partition transforms keeping partitioned writes native |
210211
| `CometIcebergRewriteActionSuite` | `rewrite_data_files` through the split plan and the native writer |
211212
| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation |
@@ -231,7 +232,8 @@ Ask specifically:
231232
default Spark profile. Reflection, metrics conventions and path spelling differ across
232233
Iceberg versions, so ask for the `run-iceberg-tests` label or a local run where they matter.
233234
- [ ] **Is a green Iceberg Spark test run being cited as evidence?** It does not show which writer
234-
handled a test ([#6148](https://github.com/apache/datafusion-comet/issues/6148)).
235+
handled a test. The job summary counts writes per writer without naming tests (see "Which
236+
writer ran each Iceberg write" in `contributor-guide/iceberg-spark-tests.md`).
235237

236238
## 8. Do the PR Changes Make the Docs Stale?
237239

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -506,6 +506,7 @@ jobs:
506506
org.apache.comet.CometVariantProjectionSuite
507507
org.apache.spark.sql.CometVariantShreddingSuite
508508
org.apache.comet.CometIcebergNativeSuite
509+
org.apache.comet.CometBroadcastBuildSideSuite
509510
org.apache.comet.CometIcebergEncryptionSuite
510511
org.apache.comet.CometIcebergRewriteActionSuite
511512
org.apache.comet.CometIcebergWriteActionSuite
@@ -587,6 +588,7 @@ jobs:
587588
org.apache.comet.rules.CometScanRuleSuite
588589
org.apache.comet.rules.CometScanContribSuite
589590
org.apache.comet.rules.CometScanSchemeFallbackSuite
591+
org.apache.comet.rules.CometMultiStoreScanSuite
590592
org.apache.comet.rules.CometExecRuleSuite
591593
org.apache.spark.sql.comet.CometPlanEqualitySuite
592594
org.apache.comet.rules.CometEmptyRelationExecRuleSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ jobs:
172172
org.apache.comet.CometVariantProjectionSuite
173173
org.apache.spark.sql.CometVariantShreddingSuite
174174
org.apache.comet.CometIcebergNativeSuite
175+
org.apache.comet.CometBroadcastBuildSideSuite
175176
org.apache.comet.CometIcebergEncryptionSuite
176177
org.apache.comet.CometIcebergRewriteActionSuite
177178
org.apache.comet.CometIcebergWriteActionSuite
@@ -253,6 +254,7 @@ jobs:
253254
org.apache.comet.rules.CometScanRuleSuite
254255
org.apache.comet.rules.CometScanContribSuite
255256
org.apache.comet.rules.CometScanSchemeFallbackSuite
257+
org.apache.comet.rules.CometMultiStoreScanSuite
256258
org.apache.comet.rules.CometExecRuleSuite
257259
org.apache.spark.sql.comet.CometPlanEqualitySuite
258260
org.apache.comet.rules.CometEmptyRelationExecRuleSuite

‎dev/ci/check-suites.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ def file_to_class_name(path: Path) -> str | None:
3535
ignore_list = [
3636
"org.apache.comet.parquet.ParquetReadSuite", # abstract
3737
"org.apache.comet.parquet.ParquetReadFromS3Suite", # manual test suite
38+
"org.apache.comet.csv.CsvReadFromS3Suite", # manual test suite
3839
"org.apache.comet.parquet.ParquetReadFromFakeHadoopFsSuite", # manual test suite (loads libhdfs, see #5023)
3940
"org.apache.comet.IcebergReadFromS3Suite", # manual test suite
4041
"org.apache.comet.cloud.s3.CometS3CredentialBridgeSuite", # manual test suite

‎docs/source/contributor-guide/iceberg-spark-tests.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,11 @@ are still the reference, and are what you want when updating a diff.
6565

6666
## 1. Install Comet
6767

68-
Run `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark version.
68+
Run `make release` in Comet to install the Comet JAR into the local Maven repository, specifying the Spark
69+
and Scala versions the Iceberg tests will run against. The example below matches step 3.
6970

7071
```shell
71-
PROFILES="-Pspark-4.1" make release
72+
PROFILES="-Pspark-3.5 -Pscala-2.13" make release
7273
```
7374

7475
## 2. Clone Iceberg and Apply Diff
@@ -121,12 +122,12 @@ diff must be generated against its own tag.
121122

122123
## Running Tests in CI
123124

124-
The `iceberg_spark_test_<version>.yml` workflows apply these diffs and run the three Gradle targets above
125-
against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against Spark 3.4.3; Iceberg 1.9.1 and 1.10.0
126-
run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3. Iceberg 1.11.0 runs in the
127-
merge queue; 1.8.1, 1.9.1 and 1.10.0 run once a night against `main`. All four run earlier on a
128-
pull request labeled `run-iceberg-tests`; none runs on an unlabeled pull request. All caller
129-
workflows delegate to `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See
125+
The `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` and `iceberg_1_11` jobs in `ci.yml` apply these diffs and
126+
run the three Gradle targets above against each Iceberg version, all with Java 17. Iceberg 1.8.1 runs against
127+
Spark 3.4.3; Iceberg 1.9.1 and 1.10.0 run against Spark 3.5.9; Iceberg 1.11.0 runs against Spark 4.1.3.
128+
Iceberg 1.11.0 runs in the merge queue; 1.8.1, 1.9.1 and 1.10.0 run once a night against `main`. All four
129+
run earlier on a pull request labeled `run-iceberg-tests`; none runs on an unlabeled pull request. Each job
130+
calls `iceberg_spark_test_reusable.yml`, which holds the build and test job logic. See
130131
[.github/workflows/README.md](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/README.md)
131132
for how the pull-request, merge-queue and nightly tiers differ.
132133

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 25 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ Two layers, the second built on the first, both switched by `spark.comet.write.i
3838
| The native writer | Who writes the data files. An eligible `IcebergWrite` becomes `CometIcebergWrite`, which writes Parquet with iceberg-rust. |
3939

4040
The flag defaults to `false`, which plans Spark's own write operator. With it on, a change to the
41-
split plan reaches every Iceberg write and a change to the native writer reaches every eligible one.
41+
split plan reaches every append, overwrite and copy-on-write Iceberg write, and a change to the
42+
native writer reaches every eligible one.
4243
The testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the
4344
native writer off, which the suites use to compare the two writers under the same plan. The
4445
criteria for turning the flag on by default are tracked in
@@ -170,7 +171,8 @@ The gate has these properties, and a new rule has to keep them:
170171
`scheme_of` split a location's scheme differently). Prefer sharing one implementation, or pin the
171172
pair with a test that feeds both the same inputs.
172173

173-
Every rule is pinned by `CometIcebergWriteDetectionSuite`. Add a case there for any new rule, for
174+
Most rules are pinned by `CometIcebergWriteDetectionSuite`, and the row lineage rule by
175+
`CometIcebergWriteActionSuite`. Add a case to the detection suite for any new rule, for
174176
both the accepted and the declined side. Update the eligibility table in the user guide in the same
175177
change. Whether each existing restriction is permanent is tracked in
176178
[#5643](https://github.com/apache/datafusion-comet/issues/5643).
@@ -238,9 +240,9 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
238240
- **Location generation.** `CometLocationGenerator` (`iceberg_partition_path.rs`) replaces
239241
iceberg-rust's `DefaultLocationGenerator` and renders partition directories the way iceberg-java's
240242
`PartitionSpec.partitionToPath` does. iceberg-rust's own rendering differs for several types, and
241-
panics for pre-1970 `timestamptz` and for dates past `chrono`'s calendar. Float and double values
242-
use Comet's Java `Double.toString` renderer (`java_float_string`), shared with
243-
`cast(float as string)`. The partition type is resolved once when the generator is built, because
243+
panics for dates past `chrono`'s calendar. Float and double values use Comet's Java
244+
`Double.toString` renderer (`java_float_string`), shared with `cast(float as string)`. The
245+
partition type is resolved once when the generator is built, because
244246
`LocationGenerator::generate_location` cannot return an error.
245247
- **Partition values.** `PartitionValueCalculator` (`iceberg_partition_value.rs`) replaces
246248
iceberg-rust's calculator of the same name, and `PartitionSplitter` replaces its
@@ -264,9 +266,10 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
264266
column's encoding when the file opens
265267
([#6114](https://github.com/apache/datafusion-comet/issues/6114)). So each partition's
266268
`PartitionFeed` holds back the partition's first page of rows (`write.parquet.page-row-limit`,
267-
capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before any of them
268-
reaches the writer. `DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting
269-
over those rows and turns dictionary encoding off for the columns parquet-mr would write plain.
269+
at least 100, capped by `write.parquet.row-group-size-bytes`, which the fanout feeds share) before
270+
any of them reaches the writer, unless no column has a dictionary choice to make.
271+
`DictionaryChooser` (`iceberg_dictionary.rs`) replays parquet-mr's accounting over those rows and
272+
turns dictionary encoding off for the columns parquet-mr would write plain.
270273
It reads only the columns there is a choice for, one at a time, and walks each lazily only as far
271274
as its first page, so beyond the held rows it keeps no more than one column's dictionary. The
272275
accounting follows `FallbackValuesWriter`, `DictionaryValuesWriter` and
@@ -347,7 +350,8 @@ backend ignores them. A fallback reason must contain property names only, never
347350
Each task emits exactly one batch with one row and two `BINARY` columns (`build_output_schema`):
348351

349352
1. The task's `DataFile`s encoded as an Iceberg v2 data manifest, written by iceberg-rust's
350-
`ManifestWriter` into an in-process `memory:` `FileIO`. Empty when the task wrote no files.
353+
`ManifestWriter` into an in-process `memory:` `FileIO`. A task that wrote no files sends a
354+
manifest with no entries.
351355
2. Every location the task's writers were handed (`encode_locations`: a big-endian count, then a
352356
length-prefixed UTF-8 string per location).
353357

@@ -368,7 +372,8 @@ Each task emits exactly one batch with one row and two `BINARY` columns (`build_
368372
Rebuilding metrics on the JVM is the main reason manifest parity holds: metrics modes, truncation,
369373
the inferred-column cap and list/map bounds suppression are decided by iceberg-java's code. When
370374
touching the native writer, do not try to make iceberg-rust's own metrics authoritative; the JVM
371-
discards them. The price is one footer read per written file.
375+
discards them. The price is reading each written file's footer back from storage, two small reads
376+
per file.
372377

373378
## Failure Handling and Cleanup Ownership
374379

@@ -438,7 +443,7 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
438443
| Suite | What it covers |
439444
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
440445
| `CometIcebergWriteActionSuite` | End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, writes outgrowing the memory pool, AQE re-planning. |
441-
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule, accepted and declined. |
446+
| `CometIcebergWriteDetectionSuite` | Most eligibility rules, each accepted and declined. |
442447
| `CometIcebergSystemFunctionSuite` | Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
443448
| `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. |
444449
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
@@ -448,9 +453,11 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
448453
| `CometIcebergEndToEndBenchmark` | An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. |
449454

450455
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
451-
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0 and 4.2, and 1.11.0 for 4.1. Only the default profile
452-
runs on every pull request. Behaviour that differs between Iceberg versions, such as reflection
453-
targets, metrics conventions and partition path spelling, needs a test that runs on each.
456+
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published
457+
for Spark 4.2, so the 4.2 profile builds against the 4.0 runtime and the Iceberg suites skip there
458+
(`icebergAvailable` is false on Spark 4.2). Only the default profile runs on every pull request.
459+
Behaviour that differs between Iceberg versions, such as reflection targets, metrics conventions
460+
and partition path spelling, needs a test that runs on each.
454461

455462
When writing a native-write test:
456463

@@ -471,9 +478,10 @@ When writing a native-write test:
471478

472479
The upstream Iceberg Spark tests also run with the native writer and `localTableScan` enabled (see
473480
[Running Iceberg Spark Tests](iceberg-spark-tests.md)). They are a broad regression net, but they do
474-
not assert which writer ran, and Comet's fallback reasons do not appear in their CI logs, so a green
475-
run is not evidence that the native writer handled a given test
476-
([#6148](https://github.com/apache/datafusion-comet/issues/6148)).
481+
not assert which writer ran, so a green run is not evidence that the native writer handled a given
482+
test. Each job's summary page counts the writes each writer ran and lists the most common reasons
483+
Comet declined the rest (see
484+
[Which writer ran each Iceberg write](iceberg-spark-tests.md#which-writer-ran-each-iceberg-write)).
477485

478486
## Pitfalls
479487

‎docs/source/contributor-guide/s3-credential-provider-design.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ The cache holds the `FileIO` and its bridge. The bridge reuses its credential un
154154

155155
## Property-bag handling on the Parquet path
156156

157-
The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects `fs.s3a.*` (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present.
157+
The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects the object store prefixes of the schemes the scan reads (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present.
158158

159159
## Built-in adapters
160160

0 commit comments

Comments
 (0)