@@ -43,7 +43,8 @@ The highlights:
4343 iceberg-java.
4444- ** Memory management** : Comet now measures the native memory its pools don't track, logs it on every executor,
4545 and fixes several long-standing pool accounting bugs.
46- - ** Native shuffle over Apache Celeborn** , for Celeborn clients that provide a safe push-completion API.
46+ - ** Native shuffle over Apache Celeborn** : Comet's side is in place, but it needs a Celeborn client API that no
47+ released Celeborn version provides yet.
4748- ** Native Parquet writes on Spark 4.0+** (experimental), now built on Spark's own write path.
4849
4950## Native Iceberg Writes (Experimental)
@@ -55,8 +56,9 @@ disabled by default**, and it only engages under fairly strict conditions.
5556### Splitting the write operator
5657
5758Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
58- and validates against the catalog. That operator sits outside Adaptive Query Execution (AQE), so the query
59- feeding it can't be re-planned at runtime, and Comet can't see into it.
59+ and validates against the catalog. Adaptive Query Execution (AQE) re-plans the query feeding that operator, but
60+ the operator itself sits outside AQE, so the file writing can't adapt to how its input actually ran. And because
61+ file writing is bundled with the metadata and commit steps, there is no separate piece for Comet to take over.
6062
6163Setting ` spark.comet.write.iceberg.splitOperator.enabled=true ` splits eligible writes into two operators:
6264
@@ -65,12 +67,12 @@ Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible w
65672 . ** ` IcebergCommit ` ** collects the commit messages on the driver and performs the normal Iceberg commit, once,
6668 outside AQE.
6769
68- On its own, the split doesn't change who writes the files; that's still iceberg-java. It makes the write's input
69- visible to AQE and to Comet, and gives the native writer a place to plug in. It covers ` INSERT INTO ` and
70- DataFrame ` append ` , static and dynamic ` INSERT OVERWRITE ` , and copy-on-write ` DELETE ` , ` UPDATE ` , and ` MERGE ` , on
71- every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
72- unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the
73- write exactly as if Comet weren't there.
70+ On its own, the split doesn't change who writes the files; that's still iceberg-java. It moves file writing inside
71+ AQE and separates it from the commit, which gives the native writer a place to plug in. It covers ` INSERT INTO `
72+ and DataFrame ` append ` , static and dynamic ` INSERT OVERWRITE ` , and copy-on-write ` DELETE ` , ` UPDATE ` , and ` MERGE ` ,
73+ on every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
74+ unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the write
75+ exactly as if Comet weren't there.
7476
7577### Writing Parquet natively
7678
@@ -101,8 +103,9 @@ Eligibility is an allowlist. A write goes native only if its whole configuration
101103supported settings. Anything else, including properties added by future Iceberg versions, falls back to
102104iceberg-java and reports why in Comet's extended ` EXPLAIN ` output. Encryption, object-storage layout, custom
103105location providers, bloom filters, and unrecognized ` parquet.* ` properties all fall back. For a partitioned
104- table, the distribution and sort Iceberg requests on its partition transforms must also run natively, which
105- works because Iceberg's system functions now have native implementations (see below).
106+ table, Iceberg asks Spark to cluster and sort rows by partition value, such as ` bucket(16, id) ` or ` days(ts) ` ,
107+ before writing. That step must also run natively, which it now can because Iceberg's system functions have
108+ native implementations.
106109
107110Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
108111where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
@@ -117,12 +120,13 @@ Iceberg's normal `remove_orphan_files` maintenance.
117120
118121### Known differences
119122
120- Files written by parquet-rs differ from parquet-mr's in a few ways, which enabling the feature accepts. Most are
121- cosmetic, such as footer metadata, ` created_by ` , and page encoding labels. Two are worth knowing about:
123+ Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
124+ parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
125+ Most are cosmetic, such as footer metadata, ` created_by ` , and page encoding labels. Two are worth knowing about:
122126
123127- ** Files won't split at the same points.** Both writers check file size every 1000 rows, but they estimate it
124128 differently, so they can roll over to a new file at different rows.
125- - ** High-cardinality columns keep a dictionary page.** parquet-mr drops dictionary encoding early when it isn't
129+ - ** High-cardinality columns keep a dictionary page.** parquet-java drops dictionary encoding early when it isn't
126130 saving space, while parquet-rs keeps it up to ` write.parquet.dict-size-bytes ` . Results are the same, but
127131 selective reads of native-written files fetch more bytes
128132 ([ #6114 ] ( https://github.com/apache/datafusion-comet/issues/6114 ) ).
@@ -192,6 +196,10 @@ nor the factor is set, except in local mode.
192196
193197### Memory pool fixes
194198
199+ Comet's off-heap memory comes from one of two pools. ` fair_unified ` , the default, caps each operator at an even
200+ share of the task's memory. ` greedy_unified ` gives memory to operators first come, first served. When Spark
201+ grants an operator less memory than it asked for (a partial grant), the operator spills.
202+
195203- ** ` fair_unified ` limited a whole task to one operator's share.** Since 0.15.0, the pool compared all of a
196204 task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
197205 Each operator now gets its own share, so tasks with several operators can use more memory before spilling.
@@ -202,9 +210,9 @@ nor the factor is set, except in local mode.
202210 a failed Arrow import leaked the vectors it had already imported.
203211- ** Configuration units.** 1.0.0 misread three size settings, including ` spark.memory.offHeap.size ` written as
204212 a bare number. See [ Upgrading to 1.1.0] ( #upgrading-to-110 ) for what changes.
205- - ** Less log noise when spilling.** A partial grant is how an operator learns to spill, but 1.0.0 logged a
206- warning and a memory dump for each one, so a spilling query could log hundreds. Partial grants now log at
207- DEBUG, and the dump is gone because it could deadlock the task.
213+ - ** Less log noise when spilling.** 1.0.0 logged a warning and a memory dump for every partial grant, so a
214+ spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
215+ deadlock the task.
208216- ** Metrics.** Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.
209217
210218` spark.comet.exec.memoryPool.fraction ` is deprecated. It was meant to leave room for untracked memory, but Spark
@@ -213,8 +221,8 @@ hands out the whole pool anyway, so it never did. Size `spark.executor.memoryOve
213221### On-heap mode
214222
215223On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
216- Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes it,
217- along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
224+ Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
225+ accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
218226off-heap memory however it's enabled, including when ` CometSparkSessionExtensions ` is registered directly, and
219227disables itself with a warning otherwise.
220228
@@ -231,7 +239,6 @@ fix, and [@sunchao] for the native aggregate spill and memory metrics, with revi
231239
232240- ** V3 deletion vectors** are applied on native scans.
233241- ** Iceberg system functions** (` bucket ` , ` truncate ` , ` years ` , ` months ` , ` days ` , and ` hours ` ) run natively.
234- Besides being faster, this keeps partitioned writes eligible for the native writer.
235242- ** Scan planning metrics and scan time** appear in the Spark UI for native Iceberg scans.
236243- ** A wrong-results fix.** A filter such as ` bucket(4, id) = 2 ` combined with another predicate was pushed to
237244 the native scan as ` id = 2 ` , returning too few rows.
@@ -244,15 +251,16 @@ null-check fix, and [@andygrove] for the native system functions and residual fi
244251
245252## Remote Shuffle with Celeborn
246253
247- Comet's native shuffle can now run over Apache Celeborn, through Comet's composite shuffle manager. In 1.0.0,
248- Celeborn users always got ordinary Spark shuffle. Map tasks push Comet's Arrow data straight to Celeborn, and
249- reducers read it back natively.
254+ 1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
255+ Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.
250256
251- To use it, set ` spark.comet.shuffle.mode=native ` ; the default ` auto ` mode keeps ordinary shuffle. You also need
252- a Celeborn client with a safe push-completion API. Released 0.6.0 and 0.7.0 clients don't have one, so they keep
253- ordinary shuffle even in native mode. Native shuffle over Celeborn doesn't support
254- ` spark.io.encryption.enabled=true ` , and Celeborn isn't bundled with Comet. The [ Celeborn tuning guide] has the
255- full requirements.
257+ It does not run with a released Celeborn yet. Native shuffle needs Celeborn to report reliably when an in-flight
258+ push has completed, and no released client does, including 0.7.0, the latest release. Those clients keep ordinary
259+ shuffle even when native shuffle is requested. Once a Celeborn release provides that API, set
260+ ` spark.shuffle.manager ` to ` org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager ` and
261+ ` spark.comet.shuffle.mode=native ` ; the default ` auto ` mode keeps ordinary shuffle. Native shuffle over Celeborn
262+ doesn't support ` spark.io.encryption.enabled=true ` , and Celeborn isn't bundled with Comet. The
263+ [ Celeborn tuning guide] has the full requirements.
256264
257265Thanks to [ @pingzh ] for this work, with reviews from [ @sunchao ] , [ @ziting-openai ] , and [ @andygrove ] . Related
258266PRs: [ #5473 ] , [ #5481 ] , [ #5513 ] , [ #5531 ] , [ #5537 ] .
@@ -271,7 +279,7 @@ Spark 4.0 made pluggable, and Spark handles the rest. That means:
271279- File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
272280 committer, work.
273281- Column names, nullability, and field IDs come from the target table, not the query.
274- - Byte and row counts are correct on HDFS.
282+ - Bytes-written and rows-written metrics are correct on HDFS.
275283
276284The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
277285` spark.sql.files.maxRecordsPerFile ` . To enable it, set ` spark.comet.parquet.write.enabled=true ` and
@@ -286,7 +294,8 @@ Comet's native S3 access now works with more of the credential setups Spark supp
286294
287295- ** Built-in credential provider adapters.** In 1.0.0, a native Parquet scan failed with
288296 ` Unsupported credential provider ` for classes Spark accepts but Comet didn't reimplement, such as
289- ` DefaultAWSCredentialsProviderChain ` . Two new adapters fix this with one config line.
297+ ` DefaultAWSCredentialsProviderChain ` . Two new adapters fix this by setting
298+ ` spark.hadoop.fs.s3a.comet.credential.provider.class ` .
290299 ` HadoopS3ACredentialProviderAdapter ` (recommended) uses Hadoop S3A's own provider chain, so it supports
291300 everything S3A does, including web identity, assumed roles, and per-bucket settings.
292301 ` AwsSdkCredentialProviderAdapter ` wraps a specific AWS SDK provider class.
@@ -298,14 +307,16 @@ Comet's native S3 access now works with more of the credential setups Spark supp
298307- ** Per-location credentials.** A provider implementing ` CometS3LocationScopedCredentialProvider ` can return
299308 different credentials for different prefixes in the same bucket, such as ` warehouse/sales ` and
300309 ` warehouse/finance ` .
301- - ** S3-compliant filesystems** are supported by the native reader.
310+ - ** Custom S3-compatible schemes.** Vendor filesystems that front an S3-compatible store with their own URL
311+ scheme, such as ` blob:// ` , can be read natively by listing the scheme in
312+ ` spark.hadoop.fs.comet.s3Compliant.schemes ` .
302313
303314See the [ S3 credential providers guide] for details.
304315
305316[ S3 credential providers guide ] : https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html
306317
307318Thanks to [ @parthchandra ] for the credential adapters and STS throttling protection, [ @snmvaughan ] for
308- per-location credentials, and [ @comphead ] for S3-compliant filesystem support , with reviews from [ @sunchao ] and
319+ per-location credentials, and [ @comphead ] for custom S3-compatible schemes , with reviews from [ @sunchao ] and
309320[ @andygrove ] . Related PRs: [ #6023 ] , [ #6025 ] , [ #6031 ] , [ #5314 ] .
310321
311322## Performance
@@ -327,37 +338,43 @@ Thanks to the contributors who drove this work, especially [@peterxcli], [@dwsmi
327338
328339### Planning and Execution
329340
330- - ** Native dynamic filter pushdown** from hash joins into Parquet scans.
331- - ** Adaptive partial aggregation** for eligible native shuffle plans.
341+ - ** Native dynamic filter pushdown** : a hash join's build-side keys filter the probe-side Parquet scan at
342+ runtime, so the scan can skip data that can't match.
343+ - ** Adaptive partial aggregation** for eligible native shuffle plans: when grouping keys are nearly unique, the
344+ partial aggregate passes rows through instead of building a hash table that doesn't reduce them.
332345- ** Parsed plan data is cached** across a stage's tasks.
333346- New ** native Parquet scan I/O and read-amplification metrics** .
334347
335348### Expressions
336349
337- Many expression kernels got faster. ` map_sort ` is up to 3x faster on multi-entry string maps and 18x faster on
338- singleton normalization . The map lookup behind ` element_at ` and ` GetMapValue ` is vectorized, ` collect_list ` and
339- ` collect_set ` have a native ` GroupsAccumulator ` , and regex patterns are compiled once per planned expression.
340- Smaller gains cover date and time field extraction, ` posexplode ` , unnesting, nested hashing, decimal overflow
341- checks, ` list_extract ` , and approximate percentile merges.
350+ Many expression kernels got faster. In kernel microbenchmarks, ` map_sort ` is up to 3x faster on multi-entry string
351+ maps and 18x faster on single-entry maps . The map lookup behind ` element_at ` and ` GetMapValue ` is vectorized,
352+ ` collect_list ` and ` collect_set ` have a native ` GroupsAccumulator ` , and regex patterns are compiled once per
353+ planned expression. Smaller gains cover date and time field extraction, ` posexplode ` , unnesting, nested hashing,
354+ decimal overflow checks, ` list_extract ` , and approximate percentile merges.
342355
343356## Expanded Coverage
344357
345358New native support includes the ` mode ` , ` max_by ` / ` min_by ` , ` listagg ` / ` string_agg ` (Spark 4.0+), and
346359` regr_* ` regression aggregates; ` WindowGroupLimitExec ` ; ` explode_outer ` ; ` make_interval ` ; native
347- ` spark_sequence ` for integral types and ` spark_unbase64 ` ; ` _metadata ` constant columns in the native Parquet
360+ ` sequence ` for integral types and ` unbase64 ` ; ` _metadata ` constant columns in the native Parquet
348361scan; nested types as shuffle hash partitioning keys; ` BinaryType ` in sort-merge joins; and Spark 4's
349362` EmptyRelationExec ` .
350363
351- More expressions go through codegen dispatch by default, including ` translate ` , ` to_csv ` , ` encode ` , ` lpad ` /
352- ` rpad ` , ` round ` on floats, ` abs ` on intervals, ` timestampadd ` / ` timestampdiff ` , and ` next_day ` and
353- ` levenshtein ` on collated input, plus any unrecognized ` StaticInvoke ` / ` Invoke ` . ` rlike ` runs natively by
364+ More expressions now use [ codegen dispatch] by default, where Comet runs Spark's own generated code for an
365+ expression inside the native pipeline, so the result matches Spark without falling back. They include
366+ ` translate ` , ` to_csv ` , ` encode ` , ` lpad ` / ` rpad ` , ` round ` on floats, ` abs ` on intervals, ` timestampadd ` /
367+ ` timestampdiff ` , ` next_day ` and ` levenshtein ` on collated input, and Spark expressions that call a Java or
368+ Scala method directly. ` rlike ` runs natively by
354369default for literal patterns that behave the same as in Java.
355370
371+ [ codegen dispatch ] : https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/
372+
356373Spark 4 ** Variant** support also improved: native Parquet scans can project Variant columns directly. Thanks to
357374[ @peterxcli ] for driving Variant support, with reviews from [ @sunchao ] . Related PRs: [ #5868 ] , [ #5794 ] .
358375
359376There's also ** experimental native support for the in-memory cache** (` spark.comet.exec.inMemoryCache.enabled ` ,
360- off by default), and build gates for contrib ** Delta ** and ** Lance ** scans .
377+ off by default).
361378
362379## Previewing Comet Plans
363380
@@ -428,8 +445,7 @@ affect 1.0.0 users. The [change log] has the rest.
428445
429446## Upgrading to 1.1.0
430447
431- 1.1.0 has no behavior changes that need a ` spark.comet.legacy.* ` key, but a few changes can affect a
432- deployment. The [ Comet Upgrade Guide] has the details.
448+ A few changes in 1.1.0 can affect a deployment. The [ Comet Upgrade Guide] has the details.
433449
434450[ Comet Upgrade Guide ] : https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html
435451
0 commit comments