Skip to content

Commit a082921

Browse files
committed
Address @mbutrovich's review of the Comet 1.1.0 post
- Correct the AQE framing: Spark already re-plans the query feeding a V2 write; what sits outside AQE is the write operator itself. - Define fair_unified, greedy_unified, partial grants, codegen dispatch, dynamic filter pushdown, and adaptive partial aggregation on first use. - Name parquet-rs and parquet-java (formerly parquet-mr). - State plainly that native shuffle over Celeborn does not run with any released Celeborn yet, and name CometCelebornShuffleManager. - Say map_sort numbers come from kernel microbenchmarks. - Use Spark function names (sequence, unbase64), name the S3 adapter and custom-scheme configs, and drop the inert Delta/Lance build gates and the legacy-key clause.
1 parent 0ecd3f3 commit a082921

1 file changed

Lines changed: 62 additions & 46 deletions

File tree

‎content/blog/2026-10-01-datafusion-comet-1.1.0.md‎

Lines changed: 62 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,8 @@ The highlights:
4343
iceberg-java.
4444
- **Memory management**: Comet now measures the native memory its pools don't track, logs it on every executor,
4545
and fixes several long-standing pool accounting bugs.
46-
- **Native shuffle over Apache Celeborn**, for Celeborn clients that provide a safe push-completion API.
46+
- **Native shuffle over Apache Celeborn**: Comet's side is in place, but it needs a Celeborn client API that no
47+
released Celeborn version provides yet.
4748
- **Native Parquet writes on Spark 4.0+** (experimental), now built on Spark's own write path.
4849

4950
## Native Iceberg Writes (Experimental)
@@ -55,8 +56,9 @@ disabled by default**, and it only engages under fairly strict conditions.
5556
### Splitting the write operator
5657

5758
Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
58-
and validates against the catalog. That operator sits outside Adaptive Query Execution (AQE), so the query
59-
feeding it can't be re-planned at runtime, and Comet can't see into it.
59+
and validates against the catalog. Adaptive Query Execution (AQE) re-plans the query feeding that operator, but
60+
the operator itself sits outside AQE, so the file writing can't adapt to how its input actually ran. And because
61+
file writing is bundled with the metadata and commit steps, there is no separate piece for Comet to take over.
6062

6163
Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible writes into two operators:
6264

@@ -65,12 +67,12 @@ Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible w
6567
2. **`IcebergCommit`** collects the commit messages on the driver and performs the normal Iceberg commit, once,
6668
outside AQE.
6769

68-
On its own, the split doesn't change who writes the files; that's still iceberg-java. It makes the write's input
69-
visible to AQE and to Comet, and gives the native writer a place to plug in. It covers `INSERT INTO` and
70-
DataFrame `append`, static and dynamic `INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE`, and `MERGE`, on
71-
every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
72-
unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the
73-
write exactly as if Comet weren't there.
70+
On its own, the split doesn't change who writes the files; that's still iceberg-java. It moves file writing inside
71+
AQE and separates it from the commit, which gives the native writer a place to plug in. It covers `INSERT INTO`
72+
and DataFrame `append`, static and dynamic `INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE`, and `MERGE`,
73+
on every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
74+
unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the write
75+
exactly as if Comet weren't there.
7476

7577
### Writing Parquet natively
7678

@@ -101,8 +103,9 @@ Eligibility is an allowlist. A write goes native only if its whole configuration
101103
supported settings. Anything else, including properties added by future Iceberg versions, falls back to
102104
iceberg-java and reports why in Comet's extended `EXPLAIN` output. Encryption, object-storage layout, custom
103105
location providers, bloom filters, and unrecognized `parquet.*` properties all fall back. For a partitioned
104-
table, the distribution and sort Iceberg requests on its partition transforms must also run natively, which
105-
works because Iceberg's system functions now have native implementations (see below).
106+
table, Iceberg asks Spark to cluster and sort rows by partition value, such as `bucket(16, id)` or `days(ts)`,
107+
before writing. That step must also run natively, which it now can because Iceberg's system functions have
108+
native implementations.
106109

107110
Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
108111
where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
@@ -117,12 +120,13 @@ Iceberg's normal `remove_orphan_files` maintenance.
117120

118121
### Known differences
119122

120-
Files written by parquet-rs differ from parquet-mr's in a few ways, which enabling the feature accepts. Most are
121-
cosmetic, such as footer metadata, `created_by`, and page encoding labels. Two are worth knowing about:
123+
Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
124+
parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
125+
Most are cosmetic, such as footer metadata, `created_by`, and page encoding labels. Two are worth knowing about:
122126

123127
- **Files won't split at the same points.** Both writers check file size every 1000 rows, but they estimate it
124128
differently, so they can roll over to a new file at different rows.
125-
- **High-cardinality columns keep a dictionary page.** parquet-mr drops dictionary encoding early when it isn't
129+
- **High-cardinality columns keep a dictionary page.** parquet-java drops dictionary encoding early when it isn't
126130
saving space, while parquet-rs keeps it up to `write.parquet.dict-size-bytes`. Results are the same, but
127131
selective reads of native-written files fetch more bytes
128132
([#6114](https://github.com/apache/datafusion-comet/issues/6114)).
@@ -192,6 +196,10 @@ nor the factor is set, except in local mode.
192196

193197
### Memory pool fixes
194198

199+
Comet's off-heap memory comes from one of two pools. `fair_unified`, the default, caps each operator at an even
200+
share of the task's memory. `greedy_unified` gives memory to operators first come, first served. When Spark
201+
grants an operator less memory than it asked for (a partial grant), the operator spills.
202+
195203
- **`fair_unified` limited a whole task to one operator's share.** Since 0.15.0, the pool compared all of a
196204
task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
197205
Each operator now gets its own share, so tasks with several operators can use more memory before spilling.
@@ -202,9 +210,9 @@ nor the factor is set, except in local mode.
202210
a failed Arrow import leaked the vectors it had already imported.
203211
- **Configuration units.** 1.0.0 misread three size settings, including `spark.memory.offHeap.size` written as
204212
a bare number. See [Upgrading to 1.1.0](#upgrading-to-110) for what changes.
205-
- **Less log noise when spilling.** A partial grant is how an operator learns to spill, but 1.0.0 logged a
206-
warning and a memory dump for each one, so a spilling query could log hundreds. Partial grants now log at
207-
DEBUG, and the dump is gone because it could deadlock the task.
213+
- **Less log noise when spilling.** 1.0.0 logged a warning and a memory dump for every partial grant, so a
214+
spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
215+
deadlock the task.
208216
- **Metrics.** Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.
209217

210218
`spark.comet.exec.memoryPool.fraction` is deprecated. It was meant to leave room for untracked memory, but Spark
@@ -213,8 +221,8 @@ hands out the whole pool anyway, so it never did. Size `spark.executor.memoryOve
213221
### On-heap mode
214222

215223
On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
216-
Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes it,
217-
along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
224+
Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
225+
accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
218226
off-heap memory however it's enabled, including when `CometSparkSessionExtensions` is registered directly, and
219227
disables itself with a warning otherwise.
220228

@@ -231,7 +239,6 @@ fix, and [@sunchao] for the native aggregate spill and memory metrics, with revi
231239

232240
- **V3 deletion vectors** are applied on native scans.
233241
- **Iceberg system functions** (`bucket`, `truncate`, `years`, `months`, `days`, and `hours`) run natively.
234-
Besides being faster, this keeps partitioned writes eligible for the native writer.
235242
- **Scan planning metrics and scan time** appear in the Spark UI for native Iceberg scans.
236243
- **A wrong-results fix.** A filter such as `bucket(4, id) = 2` combined with another predicate was pushed to
237244
the native scan as `id = 2`, returning too few rows.
@@ -244,15 +251,16 @@ null-check fix, and [@andygrove] for the native system functions and residual fi
244251

245252
## Remote Shuffle with Celeborn
246253

247-
Comet's native shuffle can now run over Apache Celeborn, through Comet's composite shuffle manager. In 1.0.0,
248-
Celeborn users always got ordinary Spark shuffle. Map tasks push Comet's Arrow data straight to Celeborn, and
249-
reducers read it back natively.
254+
1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
255+
Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.
250256

251-
To use it, set `spark.comet.shuffle.mode=native`; the default `auto` mode keeps ordinary shuffle. You also need
252-
a Celeborn client with a safe push-completion API. Released 0.6.0 and 0.7.0 clients don't have one, so they keep
253-
ordinary shuffle even in native mode. Native shuffle over Celeborn doesn't support
254-
`spark.io.encryption.enabled=true`, and Celeborn isn't bundled with Comet. The [Celeborn tuning guide] has the
255-
full requirements.
257+
It does not run with a released Celeborn yet. Native shuffle needs Celeborn to report reliably when an in-flight
258+
push has completed, and no released client does, including 0.7.0, the latest release. Those clients keep ordinary
259+
shuffle even when native shuffle is requested. Once a Celeborn release provides that API, set
260+
`spark.shuffle.manager` to `org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager` and
261+
`spark.comet.shuffle.mode=native`; the default `auto` mode keeps ordinary shuffle. Native shuffle over Celeborn
262+
doesn't support `spark.io.encryption.enabled=true`, and Celeborn isn't bundled with Comet. The
263+
[Celeborn tuning guide] has the full requirements.
256264

257265
Thanks to [@pingzh] for this work, with reviews from [@sunchao], [@ziting-openai], and [@andygrove]. Related
258266
PRs: [#5473], [#5481], [#5513], [#5531], [#5537].
@@ -271,7 +279,7 @@ Spark 4.0 made pluggable, and Spark handles the rest. That means:
271279
- File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
272280
committer, work.
273281
- Column names, nullability, and field IDs come from the target table, not the query.
274-
- Byte and row counts are correct on HDFS.
282+
- Bytes-written and rows-written metrics are correct on HDFS.
275283

276284
The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
277285
`spark.sql.files.maxRecordsPerFile`. To enable it, set `spark.comet.parquet.write.enabled=true` and
@@ -286,7 +294,8 @@ Comet's native S3 access now works with more of the credential setups Spark supp
286294

287295
- **Built-in credential provider adapters.** In 1.0.0, a native Parquet scan failed with
288296
`Unsupported credential provider` for classes Spark accepts but Comet didn't reimplement, such as
289-
`DefaultAWSCredentialsProviderChain`. Two new adapters fix this with one config line.
297+
`DefaultAWSCredentialsProviderChain`. Two new adapters fix this by setting
298+
`spark.hadoop.fs.s3a.comet.credential.provider.class`.
290299
`HadoopS3ACredentialProviderAdapter` (recommended) uses Hadoop S3A's own provider chain, so it supports
291300
everything S3A does, including web identity, assumed roles, and per-bucket settings.
292301
`AwsSdkCredentialProviderAdapter` wraps a specific AWS SDK provider class.
@@ -298,14 +307,16 @@ Comet's native S3 access now works with more of the credential setups Spark supp
298307
- **Per-location credentials.** A provider implementing `CometS3LocationScopedCredentialProvider` can return
299308
different credentials for different prefixes in the same bucket, such as `warehouse/sales` and
300309
`warehouse/finance`.
301-
- **S3-compliant filesystems** are supported by the native reader.
310+
- **Custom S3-compatible schemes.** Vendor filesystems that front an S3-compatible store with their own URL
311+
scheme, such as `blob://`, can be read natively by listing the scheme in
312+
`spark.hadoop.fs.comet.s3Compliant.schemes`.
302313

303314
See the [S3 credential providers guide] for details.
304315

305316
[S3 credential providers guide]: https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html
306317

307318
Thanks to [@parthchandra] for the credential adapters and STS throttling protection, [@snmvaughan] for
308-
per-location credentials, and [@comphead] for S3-compliant filesystem support, with reviews from [@sunchao] and
319+
per-location credentials, and [@comphead] for custom S3-compatible schemes, with reviews from [@sunchao] and
309320
[@andygrove]. Related PRs: [#6023], [#6025], [#6031], [#5314].
310321

311322
## Performance
@@ -327,37 +338,43 @@ Thanks to the contributors who drove this work, especially [@peterxcli], [@dwsmi
327338

328339
### Planning and Execution
329340

330-
- **Native dynamic filter pushdown** from hash joins into Parquet scans.
331-
- **Adaptive partial aggregation** for eligible native shuffle plans.
341+
- **Native dynamic filter pushdown**: a hash join's build-side keys filter the probe-side Parquet scan at
342+
runtime, so the scan can skip data that can't match.
343+
- **Adaptive partial aggregation** for eligible native shuffle plans: when grouping keys are nearly unique, the
344+
partial aggregate passes rows through instead of building a hash table that doesn't reduce them.
332345
- **Parsed plan data is cached** across a stage's tasks.
333346
- New **native Parquet scan I/O and read-amplification metrics**.
334347

335348
### Expressions
336349

337-
Many expression kernels got faster. `map_sort` is up to 3x faster on multi-entry string maps and 18x faster on
338-
singleton normalization. The map lookup behind `element_at` and `GetMapValue` is vectorized, `collect_list` and
339-
`collect_set` have a native `GroupsAccumulator`, and regex patterns are compiled once per planned expression.
340-
Smaller gains cover date and time field extraction, `posexplode`, unnesting, nested hashing, decimal overflow
341-
checks, `list_extract`, and approximate percentile merges.
350+
Many expression kernels got faster. In kernel microbenchmarks, `map_sort` is up to 3x faster on multi-entry string
351+
maps and 18x faster on single-entry maps. The map lookup behind `element_at` and `GetMapValue` is vectorized,
352+
`collect_list` and `collect_set` have a native `GroupsAccumulator`, and regex patterns are compiled once per
353+
planned expression. Smaller gains cover date and time field extraction, `posexplode`, unnesting, nested hashing,
354+
decimal overflow checks, `list_extract`, and approximate percentile merges.
342355

343356
## Expanded Coverage
344357

345358
New native support includes the `mode`, `max_by` / `min_by`, `listagg` / `string_agg` (Spark 4.0+), and
346359
`regr_*` regression aggregates; `WindowGroupLimitExec`; `explode_outer`; `make_interval`; native
347-
`spark_sequence` for integral types and `spark_unbase64`; `_metadata` constant columns in the native Parquet
360+
`sequence` for integral types and `unbase64`; `_metadata` constant columns in the native Parquet
348361
scan; nested types as shuffle hash partitioning keys; `BinaryType` in sort-merge joins; and Spark 4's
349362
`EmptyRelationExec`.
350363

351-
More expressions go through codegen dispatch by default, including `translate`, `to_csv`, `encode`, `lpad` /
352-
`rpad`, `round` on floats, `abs` on intervals, `timestampadd` / `timestampdiff`, and `next_day` and
353-
`levenshtein` on collated input, plus any unrecognized `StaticInvoke` / `Invoke`. `rlike` runs natively by
364+
More expressions now use [codegen dispatch] by default, where Comet runs Spark's own generated code for an
365+
expression inside the native pipeline, so the result matches Spark without falling back. They include
366+
`translate`, `to_csv`, `encode`, `lpad` / `rpad`, `round` on floats, `abs` on intervals, `timestampadd` /
367+
`timestampdiff`, `next_day` and `levenshtein` on collated input, and Spark expressions that call a Java or
368+
Scala method directly. `rlike` runs natively by
354369
default for literal patterns that behave the same as in Java.
355370

371+
[codegen dispatch]: https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/
372+
356373
Spark 4 **Variant** support also improved: native Parquet scans can project Variant columns directly. Thanks to
357374
[@peterxcli] for driving Variant support, with reviews from [@sunchao]. Related PRs: [#5868], [#5794].
358375

359376
There's also **experimental native support for the in-memory cache** (`spark.comet.exec.inMemoryCache.enabled`,
360-
off by default), and build gates for contrib **Delta** and **Lance** scans.
377+
off by default).
361378

362379
## Previewing Comet Plans
363380

@@ -428,8 +445,7 @@ affect 1.0.0 users. The [change log] has the rest.
428445

429446
## Upgrading to 1.1.0
430447

431-
1.1.0 has no behavior changes that need a `spark.comet.legacy.*` key, but a few changes can affect a
432-
deployment. The [Comet Upgrade Guide] has the details.
448+
A few changes in 1.1.0 can affect a deployment. The [Comet Upgrade Guide] has the details.
433449

434450
[Comet Upgrade Guide]: https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html
435451

0 commit comments

Comments
 (0)