Skip to content

Commit e989715

Browse files
committed
Merge remote-tracking branch 'apache/main' into fix/dispatcher-decimal-rescale-6425
2 parents 04492d8 + 9c7fcc5 commit e989715

57 files changed

Lines changed: 4691 additions & 291 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -589,6 +589,7 @@ jobs:
589589
org.apache.comet.rules.CometExecRuleSuite
590590
org.apache.spark.sql.comet.CometPlanEqualitySuite
591591
org.apache.comet.rules.CometEmptyRelationExecRuleSuite
592+
org.apache.comet.rules.EliminateRedundantTransitionsSuite
592593
org.apache.comet.rules.RevertNativeForTransitionHeavyStagesSuite
593594
org.apache.spark.sql.CometTPCDSQuerySuite
594595
org.apache.spark.sql.CometTPCDSQueryTestSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,7 @@ jobs:
246246
org.apache.comet.rules.CometExecRuleSuite
247247
org.apache.spark.sql.comet.CometPlanEqualitySuite
248248
org.apache.comet.rules.CometEmptyRelationExecRuleSuite
249+
org.apache.comet.rules.EliminateRedundantTransitionsSuite
249250
org.apache.comet.rules.RevertNativeForTransitionHeavyStagesSuite
250251
org.apache.spark.sql.CometTPCDSQuerySuite
251252
org.apache.spark.sql.CometTPCDSQueryTestSuite

‎docs/source/contributor-guide/adding_a_new_operator.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,14 @@ remove the inserted local node when restoring Spark execution. Restoring its `or
8585
apply the global TopK twice. A single input partition uses a final native limit and projection;
8686
multiple input partitions still require a final TopK after the shuffle.
8787

88+
With `spark.comet.exec.topK.dynamicFilter.enabled`, only the local sort sets the protobuf
89+
`Sort.dynamic_filter_enabled` flag. The native planner wraps an eligible sort in
90+
`TopKReaderFilterExec`. Its permanent plan contains an unexecuted sort template; each execution
91+
creates a fresh sort and live predicate, then attaches that predicate to the Parquet reader when
92+
eligible. The stream owns the heap and predicate until completion, error, or cancellation. The
93+
final TopK after an exchange does not share this state. Reader work stays on scan metrics while
94+
attachment counters belong to the local TopK.
95+
8896
### Choosing the Right Operator Type
8997

9098
When adding a new operator, choose based on these criteria:

‎docs/source/contributor-guide/expression-audits/conditional_funcs.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,15 @@
2727
- Spark 3.5.8 (audited 2026-05-27): baseline. `Coalesce(children) extends Expression with ComplexTypeMergingExpression`; returns the first non-null child, evaluated left-to-right with short-circuit. Result type is the merged child type. Comet routes via `CometCoalesce`, which serialises as nested `CaseWhen(IsNotNull(c1) -> c1, IsNotNull(c2) -> c2, ..., else cN)` so the native engine preserves the short-circuit semantics.
2828
- Spark 4.0.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
2929
- Spark 4.1.1 (audited 2026-05-27): byte-for-byte identical to 3.5.8.
30+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): runs through the `CASE WHEN` evaluation described under `when`. 12-13x faster for two columns. Benchmark: `benches/conditional.rs`.
3031

3132
## if
3233

3334
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
3435
- Spark 3.5.8 (audited 2026-05-27): baseline. `If(predicate, trueValue, falseValue) extends ComplexTypeMergingExpression`; standard ternary semantics with short-circuit, predicate must be `BooleanType`. Comet routes via `CometIf` to the native `IfExpr` proto.
3536
- Spark 4.0.1 (audited 2026-05-27): adds an `override def withNewAlwaysEvaluatedInputs(...)` hook for the new optimizer phase; semantics unchanged. Error messages reformatted (`paramIndex` -> `ordinalNumber`).
3637
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
38+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): runs through the `CASE WHEN` evaluation described under `when`, with an `IF` or `CASE WHEN` in the ELSE flattened into one merge. 1.5x faster with literal results, 6.6x with column results, 3.3-4.7x with string columns, 3.1-5.9x nested. Benchmark: `benches/conditional.rs`.
3739

3840
## ifnull
3941

@@ -69,5 +71,6 @@
6971
- Spark 3.5.8 (audited 2026-05-27): the `CASE WHEN ... THEN ...` SQL form lowers to `CaseWhen(branches: Seq[(Expression, Expression)], elseValue: Option[Expression])`. Spark evaluates left-to-right with short-circuit; result type is the merged branch type. Comet routes via `CometCaseWhen` to the native `CaseWhen` proto.
7072
- Spark 4.0.1 (audited 2026-05-27): adds the `withNewAlwaysEvaluatedInputs` optimizer hook; semantics unchanged.
7173
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
74+
- Performance (tuned 2026-09-28, PR [#6350](https://github.com/apache/datafusion-comet/pull/6350)): when no branch value and no WHEN after the first can fail, `CaseWhenExpr` evaluates them over the whole batch and takes each row's value with typed merges for primitive, boolean, string and binary results, instead of filtering the batch for each branch and merging the partial results with `MutableArrayData`. A branch is no longer wrapped in a cast to the type it already has, which hid literals and columns from DataFusion's fast paths. 2.8-4x faster with literal results and 2.2-7.8x with column results. When a branch can fail it is about 2% slower, because DataFusion's `CaseExpr` derives the result type of a bare branch for every batch. Benchmark: `benches/conditional.rs`.
7275

7376
[Spark Expression Support]: ../../user-guide/latest/expressions.md

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,8 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
201201
That mode exists only so the Spark SQL tests can run against Comet. See
202202
[Memory Management](memory_management.md).
203203
- Row count still triggers spilling in either mode. `CometDiskBlockWriter` spills at
204-
`min(spark.comet.shuffle.jvm.spillThreshold, spark.comet.shuffle.jvm.batchSize)`.
204+
`min(spark.comet.shuffle.jvm.spillThreshold, batch size)`, where the batch size is
205+
`spark.comet.shuffle.jvm.batchSize` capped at `spark.comet.batchSize`.
205206
`CometShuffleExternalSorter` spills at `spark.comet.shuffle.jvm.spillThreshold` alone, which
206207
defaults to `Int.MaxValue`, so on the sort path that trigger is effectively off by default.
207208
- `CometDiskBlockWriter` coordinates spilling across all partition writers (largest first)

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -340,6 +340,12 @@ forget, and a `createPlan` that fails partway through cleans up on unwind.
340340
insert a replacement before the dying pool reaches the registry lock. The drop therefore compares
341341
pointers and only removes an entry that is still its own.
342342

343+
The pool acquires memory from Spark through the `CometTaskMemoryManager` passed with the plan that
344+
created it, so the JVM side shares one manager per task as well: `CometExecIterator.taskMemory`
345+
hands every native plan in a task the same one and drops it when the task completes. Its `getUsed`
346+
covers the whole task, so `CometExecIterator.close()` warns about memory still in use only when
347+
the task's last open native plan closes.
348+
343349
## How DataFusion consumes the pool
344350

345351
Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservation` API:

‎docs/source/contributor-guide/roadmap.md‎

Lines changed: 51 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -53,67 +53,74 @@ these expressions benefit from vectorized native execution.
5353
## Native Coverage for Codegen-Dispatched Expressions
5454

5555
Beyond lambda bodies, a number of built-in Spark scalar expressions (some regular expression, JSON, and datetime
56-
functions, for example) and aggregate functions route through the same JVM codegen-dispatch bridge by default,
57-
either because their native DataFusion or `datafusion-spark` implementation has known semantic differences from
58-
Spark, or because no native implementation exists yet. See the [compatibility guide] for the current list of
59-
codegen-dispatched expressions. We're exploring closing these gaps so that more expressions run natively by
60-
default, which would reduce JVM round-trips beyond what the lambda and UDF work above already covers.
56+
functions, for example) route through the same JVM codegen-dispatch bridge by default, either because their native
57+
DataFusion or `datafusion-spark` implementation has known semantic differences from Spark, or because no native
58+
implementation exists yet. The [expression reference] records which expressions are codegen-dispatched today. The
59+
bridge covers only scalar expressions, so an aggregate function with no Spark-compatible native implementation falls
60+
back to Spark instead. We're exploring closing these gaps so that more expressions run natively by default, which
61+
would remove JVM round-trips beyond those the lambda work above addresses.
6162

62-
[compatibility guide]: ../user-guide/latest/compatibility/index.md
63+
[expression reference]: ../user-guide/latest/expressions.md
6364

6465
## Iceberg Table Format V3 Support
6566

66-
Comet landed its first Iceberg table format V3 feature (native data file decryption, [#4991]), and we want to
67-
add more V3 features to Comet's native Iceberg scans so they don't fall back to Spark. The work is tracked
68-
phase-by-phase in [#3376]: detecting the V3 format and supporting new V3 data types, deletion vector reads, row
69-
lineage, and table encryption. Native deletion vector reads are prototyped in a draft PR ([#4887]), but are
70-
blocked on upstream `iceberg-rust` support, tracked in [iceberg-rust #2792] and [iceberg-rust #2411]. Native Iceberg
71-
scans don't support HDFS-backed tables today: the storage scheme match in `iceberg_scan.rs` only handles `file`,
72-
`s3`/`s3a`, `gs`, and `oss`, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`. We're
73-
scoping what that work would take.
67+
Comet's native Iceberg scans read V3 tables, including encrypted tables ([#4991]) and tables with deletion vectors
68+
([#5853]). We want to add the remaining V3 features so that these scans don't fall back to Spark: row lineage
69+
metadata columns, column default values, and the new V3 types (`variant`, `geometry`, `geography`, and `unknown`).
70+
The work is tracked in [#3376], and upstream `iceberg-rust` support in [iceberg-rust #2411]. Native Iceberg scans
71+
also don't support HDFS-backed tables today: Comet's native Iceberg storage layer handles only local files, S3 and
72+
S3-compatible stores, GCS, and OSS, and would need an HDFS `StorageFactory` upstream in `iceberg-storage-opendal`.
73+
We're scoping what that work would take.
7474

7575
[#3376]: https://github.com/apache/datafusion-comet/issues/3376
76-
[#4887]: https://github.com/apache/datafusion-comet/pull/4887
7776
[#4991]: https://github.com/apache/datafusion-comet/pull/4991
77+
[#5853]: https://github.com/apache/datafusion-comet/pull/5853
7878
[iceberg-rust #2411]: https://github.com/apache/iceberg-rust/issues/2411
79-
[iceberg-rust #2792]: https://github.com/apache/iceberg-rust/issues/2792
8079

8180
## TPC-H and TPC-DS Performance
8281

8382
Comet already delivers substantial speedups over vanilla Spark on TPC-H and TPC-DS; we publish per-query
8483
results for [TPC-DS] with each release. An independent [AWS Labs benchmark] comparing Comet 0.16.0 with
8584
Gluten 1.6.0 on a 3TB TPC-DS workload found that the two accelerators deliver similar overall performance. Increasing
8685
the speedup further and closing the remaining per-query gaps is an ongoing focus, tracked under [#2004] (TPC-H) and
87-
[#858] (TPC-DS).
86+
[#2551] (TPC-DS).
8887

8988
[TPC-DS]: benchmark-results/tpc-ds.md
9089
[AWS Labs benchmark]: https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark
91-
[#858]: https://github.com/apache/datafusion-comet/issues/858
9290
[#2004]: https://github.com/apache/datafusion-comet/issues/2004
91+
[#2551]: https://github.com/apache/datafusion-comet/issues/2551
9392

9493
## Upstream Work in DataFusion
9594

9695
A growing number of Spark-compatible expressions live in the `datafusion-spark` crate in the core DataFusion
9796
repository. Comet is migrating its expression implementations to that crate so that they can be shared by other
98-
DataFusion-based projects, and is wiring up the functions that crate already provides ([#4150]). Improvements to
97+
DataFusion-based projects, and has wired up nearly every function that crate provides ([#4150]). Improvements to
9998
core DataFusion operators (joins, aggregates, window) made in support of Comet also benefit the wider ecosystem.
10099

101100
[#4150]: https://github.com/apache/datafusion-comet/issues/4150
102101

103102
## Spillable Hash Join
104103

105104
Comet's native hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk
106-
support will allow Comet to handle larger joins without falling back to Spark, improving both reliability and
107-
performance for memory-intensive workloads.
105+
support ([#2545]) will allow Comet to handle larger joins without falling back to Spark, improving both reliability
106+
and performance for memory-intensive workloads. Comet's native hash join uses DataFusion's `HashJoinExec`, and a
107+
design for spilling in that operator, behind a flag that is off by default, is proposed upstream in
108+
[datafusion #24768].
109+
110+
[#2545]: https://github.com/apache/datafusion-comet/issues/2545
111+
[datafusion #24768]: https://github.com/apache/datafusion/issues/24768
108112

109113
## Java/Scala UDF Support
110114

111115
Spark users frequently define custom UDFs in Java or Scala. Comet now dispatches scalar `ScalaUDF` expressions
112116
through a JVM codegen bridge (`CometScalaUDF`) instead of always falling back to Spark. Aggregate UDFs, table
113-
UDFs/generators, Python/Pandas UDFs, and Hive `GenericUDF`/`SimpleUDF` still fall back to Spark entirely.
117+
UDFs/generators, Hive `GenericUDF`/`SimpleUDF`, and Python UDFs other than `mapInArrow` and `mapInPandas` still fall
118+
back to Spark entirely; Comet's support for those two Python APIs is experimental and disabled by default ([#4234]).
114119
Extending the codegen-dispatch approach to cover these remaining categories will reduce fallbacks further and
115120
allow more queries to run end-to-end in Comet.
116121

122+
[#4234]: https://github.com/apache/datafusion-comet/pull/4234
123+
117124
## Memory Management Improvements
118125

119126
Comet coordinates memory between the JVM and native Rust execution through a custom memory pool. Improving
@@ -130,30 +137,37 @@ enabled by default ([#1625]).
130137

131138
## Iceberg Table Writes
132139

133-
Comet's native scans accelerate Iceberg reads today, but writes still run entirely through Spark's Iceberg V2 write
134-
operator. [#4322] tracks exploring native acceleration of Iceberg writes, so an ETL job could run end to end in
135-
native code, falling back only where Comet can't match Iceberg Java's functionality. Draft proposals split the
136-
existing V2 write operator into separate "writer" and "committer" operators so the data-file-write portion can be
137-
planned and accelerated the same way as a native Parquet write ([#4658], [#4487]). No design here is committed yet;
138-
we're exploring whether this approach is worth pursuing.
140+
Comet can now write Iceberg tables natively. The feature is experimental, disabled by default, and controlled by two
141+
settings, the second of which requires the first. The first splits Spark's Iceberg V2 write operator into separate
142+
writer and committer operators, so the query feeding the write becomes visible to AQE and to Comet's columnar rules
143+
([#4658]). The second delegates each task's Parquet write to `iceberg-rust` when the write passes an eligibility
144+
check ([#5361]); writes that don't pass keep Iceberg Java's writer. Merge-on-read writes are not intercepted yet
145+
([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code, is
146+
tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, broader coverage, and enabling
147+
both settings by default ([#5644]). See [Iceberg Writes](iceberg-writes.md) for how the write path works.
139148

140149
[#4322]: https://github.com/apache/datafusion-comet/issues/4322
141-
[#4487]: https://github.com/apache/datafusion-comet/pull/4487
142150
[#4658]: https://github.com/apache/datafusion-comet/pull/4658
151+
[#5361]: https://github.com/apache/datafusion-comet/pull/5361
152+
[#5644]: https://github.com/apache/datafusion-comet/issues/5644
153+
[#5649]: https://github.com/apache/datafusion-comet/issues/5649
154+
[#6240]: https://github.com/apache/datafusion-comet/issues/6240
143155

144156
## Delta Lake Support
145157

146-
Comet currently supports Spark's built-in file formats and Iceberg, but not Delta Lake ([#174]). A draft PR reuses
147-
Comet's existing native Parquet reader to accelerate scans of plain Delta tables that use neither deletion vectors
148-
nor column mapping ([#4669]); a separate, larger effort explores a full native Delta scan built on `delta-kernel-rs`
149-
as an out-of-tree contrib module ([#4366]), landing in inert, gated slices ([#4700], [#4952]). Generalizing Comet's
150-
scan-side APIs so that Delta and other non-Iceberg data sources can plug in more easily is tracked as a Table
151-
Provider API abstraction ([#4706]). None of this is committed yet; we're exploring which approach, if any, is worth
152-
pursuing.
158+
Comet currently supports Spark's built-in file formats and Iceberg, but not Delta Lake ([#174]). The plugin boundary
159+
for an out-of-tree Delta module has landed in inert, gated slices ([#4700], [#4952]), and two read paths build on
160+
it. In the first, which is in review, delta-spark plans the scan and Comet's native Parquet reader executes it,
161+
applying deletion vectors inside the scan; it ships as an opt-in contrib module ([#5365]). The second, still a
162+
draft, is a full native Delta scan built on `delta-kernel-rs` ([#4366]). [#5411] proposes converging the two into
163+
one plugin, with the Parquet path as the default and the kernel path covering what it can't express yet, such as
164+
change data feed. Generalizing Comet's scan-side APIs so that Delta and other non-Iceberg data sources can plug in
165+
more easily is tracked as a Table Provider API abstraction ([#4706]).
153166

154167
[#174]: https://github.com/apache/datafusion-comet/issues/174
155168
[#4366]: https://github.com/apache/datafusion-comet/pull/4366
156-
[#4669]: https://github.com/apache/datafusion-comet/pull/4669
157169
[#4700]: https://github.com/apache/datafusion-comet/pull/4700
158170
[#4706]: https://github.com/apache/datafusion-comet/issues/4706
159171
[#4952]: https://github.com/apache/datafusion-comet/pull/4952
172+
[#5365]: https://github.com/apache/datafusion-comet/pull/5365
173+
[#5411]: https://github.com/apache/datafusion-comet/issues/5411

‎docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,16 @@ If you need to process dates far in the future with accurate timezone handling,
4242
- Using timezone-naive types (`timestamp_ntz`) when timezone conversion is not required
4343
- Falling back to Spark for these specific operations
4444

45+
### Timezone Database Versions
46+
47+
Comet's native code converts between instants and local time with the IANA timezone database that
48+
chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose
49+
version depends on the JDK build and on whether its timezone data has been updated. When the two versions
50+
have different rules for a timezone, local times that Comet computes natively for that timezone can differ
51+
from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates,
52+
`date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions
53+
differ.
54+
4555
<!--BEGIN:EXPR_COMPAT[datetime]-->
4656

4757
<!--END:EXPR_COMPAT-->

0 commit comments

Comments
 (0)