Skip to content

Commit 07c8de2

Browse files
committed
Merge Apache main into array extrema fix
Retain the array-extrema guide alongside upstream signed-zero ordering notes, and keep both sets of benchmark registrations.
2 parents 9cd863d + 525c05b commit 07c8de2

134 files changed

Lines changed: 9280 additions & 1443 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/actions/java-test/action.yaml‎

Lines changed: 50 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,46 @@ runs:
7676
restore-keys: |
7777
${{ runner.os }}-java-maven-
7878
79+
# Maven itself is stored outside the dependency repository. Keep its cache
80+
# independent of pom.xml changes and preserve the macOS cache workaround.
81+
- name: Restore Maven distribution
82+
id: maven-distribution
83+
if: ${{ runner.os != 'macOS' }}
84+
uses: actions/cache/restore@v5
85+
with:
86+
path: |
87+
~/.m2/wrapper/dists
88+
/root/.m2/wrapper/dists
89+
key: ${{ runner.os }}-${{ runner.arch }}-maven-wrapper-${{ hashFiles('.mvn/wrapper/maven-wrapper.properties') }}
90+
91+
# Retry only the wrapper download, never compilation or test execution.
92+
# Delays use exponential backoff (10s, 20s, 40s) plus 0-4s of random jitter.
93+
- name: Bootstrap Maven
94+
shell: bash
95+
run: |
96+
for attempt in 1 2 3 4; do
97+
if ./mvnw -B --version; then
98+
break
99+
fi
100+
if [ "$attempt" -eq 4 ]; then
101+
echo "::error::Maven bootstrap failed after $attempt attempts; tests were not started."
102+
exit 1
103+
fi
104+
delay=$((10 * (1 << (attempt - 1)) + RANDOM % 5))
105+
echo "::warning::Maven bootstrap attempt $attempt failed; retrying in ${delay}s."
106+
sleep "$delay"
107+
done
108+
109+
# Save a successful bootstrap even when the subsequent tests fail.
110+
- name: Save Maven distribution
111+
if: ${{ runner.os != 'macOS' && steps.maven-distribution.outputs.cache-hit != 'true' }}
112+
uses: actions/cache/save@v5
113+
with:
114+
path: |
115+
~/.m2/wrapper/dists
116+
/root/.m2/wrapper/dists
117+
key: ${{ steps.maven-distribution.outputs.cache-primary-key }}
118+
79119
- name: Run all tests
80120
shell: bash
81121
if: ${{ inputs.suites == '' }}
@@ -100,25 +140,32 @@ runs:
100140
with:
101141
name: crash-logs-${{ inputs.artifact_name }}
102142
path: "**/hs_err_pid*.log"
143+
if-no-files-found: ignore
103144
- name: Debug listing
104145
if: failure()
105146
shell: bash
106-
run: |
147+
run: |
107148
echo "CWD: $(pwd)"
108149
ls -lah .
109-
ls -lah target
150+
if [ -d target ]; then
151+
ls -lah target
152+
else
153+
echo "No root target directory; Maven may not have started."
154+
fi
110155
find . -name 'unit-tests.log'
111156
- name: Upload unit-tests.log
112157
if: failure()
113158
uses: actions/upload-artifact@v6
114159
with:
115160
name: unit-tests-${{ inputs.artifact_name }}
116161
path: "**/target/unit-tests.log"
162+
if-no-files-found: ignore
117163
- name: Upload test results
118-
if: ${{ inputs.upload-test-reports == 'true' }}
164+
if: ${{ !cancelled() && inputs.upload-test-reports == 'true' }}
119165
uses: actions/upload-artifact@v6
120166
with:
121167
name: java-test-reports-${{ inputs.artifact_name }}
122168
path: "**/target/surefire-reports/*.txt"
169+
if-no-files-found: ignore
123170
retention-days: 7 # 1 week for test reports
124171
overwrite: true

‎.github/workflows/codeql.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,16 +44,16 @@ jobs:
4444

4545
steps:
4646
- name: Checkout repository
47-
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
47+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
4848
with:
4949
persist-credentials: false
5050

5151
- name: Initialize CodeQL
52-
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
52+
uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4
5353
with:
5454
languages: actions
5555

5656
- name: Perform CodeQL Analysis
57-
uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
57+
uses: github/codeql-action/analyze@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4
5858
with:
5959
category: "/language:actions"

‎.github/workflows/delta_build_gate.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ jobs:
7979
env:
8080
SPARK_LOCAL_IP: localhost
8181
steps:
82-
- uses: actions/checkout@v6
82+
- uses: actions/checkout@v7
8383

8484
- name: Setup Rust & Java toolchain
8585
uses: ./.github/actions/setup-builder

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -350,6 +350,7 @@ jobs:
350350
org.apache.comet.CometSetOpWithGroupBySuite
351351
org.apache.comet.CometSparkSessionExtensionsSuite
352352
org.apache.spark.CometPluginsSuite
353+
org.apache.spark.CometTaskMemoryManagerSuite
353354
org.apache.spark.CometPluginsDefaultSuite
354355
org.apache.spark.CometPluginsNonOverrideSuite
355356
org.apache.spark.CometPluginsUnifiedModeOverrideSuite
@@ -412,6 +413,7 @@ jobs:
412413
org.apache.comet.CometWidthBucketSuite
413414
org.apache.comet.CometUuidExpressionSuite
414415
org.apache.comet.serde.CometScalarFunctionSuite
416+
org.apache.comet.CometFallbackInvarianceSuite
415417
fail-fast: false
416418
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]
417419
runs-on: ubuntu-24.04

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,7 @@ jobs:
166166
org.apache.comet.CometSetOpWithGroupBySuite
167167
org.apache.comet.CometSparkSessionExtensionsSuite
168168
org.apache.spark.CometPluginsSuite
169+
org.apache.spark.CometTaskMemoryManagerSuite
169170
org.apache.spark.CometPluginsDefaultSuite
170171
org.apache.spark.CometPluginsNonOverrideSuite
171172
org.apache.spark.CometPluginsUnifiedModeOverrideSuite
@@ -228,6 +229,7 @@ jobs:
228229
org.apache.comet.CometWidthBucketSuite
229230
org.apache.comet.CometUuidExpressionSuite
230231
org.apache.comet.serde.CometScalarFunctionSuite
232+
org.apache.comet.CometFallbackInvarianceSuite
231233
232234
fail-fast: false
233235
name: ${{ matrix.os }}/${{ matrix.profile.name }} [${{ matrix.suite.name }}]

‎Makefile‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,11 @@ clean:
5656
bench:
5757
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo bench $(FEATURES_ARG) $(filter-out $@,$(MAKECMDGOALS))
5858
format:
59+
@if command -v prettier >/dev/null 2>&1; then \
60+
prettier -w "**/*.md"; \
61+
else \
62+
echo "prettier not found, skipping markdown formatting (npm i -g prettier)"; \
63+
fi
5964
cd native && cargo fmt
6065
./mvnw compile test-compile scalafix:scalafix -Psemanticdb $(PROFILES)
6166
./mvnw spotless:apply $(PROFILES)

‎dev/diffs/3.5.9.diff‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1446,6 +1446,30 @@ index 005e764cc30..92ec088efab 100644
14461446
}
14471447

14481448
private def checkSorts(query: String, enabledCount: Int, disabledCount: Int): Unit = {
1449+
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1450+
index 46ed8fdfd21..4585dbab5b8 100644
1451+
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1452+
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1453+
@@ -18,6 +18,7 @@
1454+
package org.apache.spark.sql.execution
1455+
1456+
import org.apache.spark.sql.{DataFrame, QueryTest}
1457+
+import org.apache.spark.sql.comet.CometWindowGroupLimitExec
1458+
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
1459+
import org.apache.spark.sql.execution.window.WindowGroupLimitExec
1460+
import org.apache.spark.sql.functions.lit
1461+
@@ -30,7 +31,10 @@ abstract class RemoveRedundantWindowGroupLimitsSuiteBase
1462+
1463+
private def checkNumWindowGroupLimits(df: DataFrame, count: Int): Unit = {
1464+
val plan = df.queryExecution.executedPlan
1465+
- assert(collectWithSubqueries(plan) { case exec: WindowGroupLimitExec => exec }.length == count)
1466+
+ assert(collectWithSubqueries(plan) {
1467+
+ case exec: WindowGroupLimitExec => exec
1468+
+ case exec: CometWindowGroupLimitExec => exec
1469+
+ }.length == count)
1470+
}
1471+
1472+
private def checkWindowGroupLimits(query: String, count: Int): Unit = {
14491473
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
14501474
index 47679ed7865..9ffbaecb98e 100644
14511475
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala

‎dev/diffs/4.0.4.diff‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1862,6 +1862,30 @@ index 005e764cc30..92ec088efab 100644
18621862
}
18631863

18641864
private def checkSorts(query: String, enabledCount: Int, disabledCount: Int): Unit = {
1865+
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1866+
index 46ed8fdfd21..4585dbab5b8 100644
1867+
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1868+
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1869+
@@ -18,6 +18,7 @@
1870+
package org.apache.spark.sql.execution
1871+
1872+
import org.apache.spark.sql.{DataFrame, QueryTest}
1873+
+import org.apache.spark.sql.comet.CometWindowGroupLimitExec
1874+
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
1875+
import org.apache.spark.sql.execution.window.WindowGroupLimitExec
1876+
import org.apache.spark.sql.functions.lit
1877+
@@ -30,7 +31,10 @@ abstract class RemoveRedundantWindowGroupLimitsSuiteBase
1878+
1879+
private def checkNumWindowGroupLimits(df: DataFrame, count: Int): Unit = {
1880+
val plan = df.queryExecution.executedPlan
1881+
- assert(collectWithSubqueries(plan) { case exec: WindowGroupLimitExec => exec }.length == count)
1882+
+ assert(collectWithSubqueries(plan) {
1883+
+ case exec: WindowGroupLimitExec => exec
1884+
+ case exec: CometWindowGroupLimitExec => exec
1885+
+ }.length == count)
1886+
}
1887+
1888+
private def checkWindowGroupLimits(query: String, count: Int): Unit = {
18651889
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
18661890
index 47679ed7865..9ffbaecb98e 100644
18671891
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala

‎dev/diffs/4.1.3.diff‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1985,6 +1985,30 @@ index 005e764cc30..92ec088efab 100644
19851985
}
19861986

19871987
private def checkSorts(query: String, enabledCount: Int, disabledCount: Int): Unit = {
1988+
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1989+
index 46ed8fdfd21..4585dbab5b8 100644
1990+
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1991+
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimitsSuite.scala
1992+
@@ -18,6 +18,7 @@
1993+
package org.apache.spark.sql.execution
1994+
1995+
import org.apache.spark.sql.{DataFrame, QueryTest}
1996+
+import org.apache.spark.sql.comet.CometWindowGroupLimitExec
1997+
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanHelper, DisableAdaptiveExecutionSuite, EnableAdaptiveExecutionSuite}
1998+
import org.apache.spark.sql.execution.window.WindowGroupLimitExec
1999+
import org.apache.spark.sql.functions.lit
2000+
@@ -30,7 +31,10 @@ abstract class RemoveRedundantWindowGroupLimitsSuiteBase
2001+
2002+
private def checkNumWindowGroupLimits(df: DataFrame, count: Int): Unit = {
2003+
val plan = df.queryExecution.executedPlan
2004+
- assert(collectWithSubqueries(plan) { case exec: WindowGroupLimitExec => exec }.length == count)
2005+
+ assert(collectWithSubqueries(plan) {
2006+
+ case exec: WindowGroupLimitExec => exec
2007+
+ case exec: CometWindowGroupLimitExec => exec
2008+
+ }.length == count)
2009+
}
2010+
2011+
private def checkWindowGroupLimits(query: String, count: Int): Unit = {
19882012
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala
19892013
index 47679ed7865..9ffbaecb98e 100644
19902014
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala

‎docs/source/contributor-guide/expression-audits/agg_funcs.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,19 @@
5757
- Spark 3.5.8 (audited 2026-06-24): identical to 3.4.3.
5858
- Spark 4.0.1 (audited 2026-06-24): only structural change is adding `with UnaryLike[Expression]` to the case class (no behavior change).
5959
- Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.
60-
- Comet implementation: native side delegates to `datafusion_spark::function::aggregate::collect::SparkCollectList`, which wraps `ArrayAggAccumulator` with `ignore_nulls = true` and converts a final NULL accumulator state to an empty array (matching Spark's `defaultResult`). The native return type is `List(Field, containsNull = true)`, while Spark uses `containsNull = false`. Because nulls are filtered before insertion, no nulls actually appear in the array, so this is a schema-shape difference only and tests using `checkSparkAnswerAndOperator` accept it (same pattern already in use for `collect_set`).
60+
- Comet implementation: native side delegates to `datafusion_spark::function::aggregate::collect::SparkCollectList`, which wraps `ArrayAggAccumulator` with `ignore_nulls = true` and converts a final NULL accumulator state to an empty array (matching Spark's `defaultResult`). The native return type is `List(Field, containsNull = true)`, while Spark uses `containsNull = false`. Because nulls are filtered before insertion, no nulls actually appear in the array, so this is a schema-shape difference only and tests using `checkSparkAnswerAndOperator` accept it (the same pattern applies to [collect_set](#collect-set)).
61+
- Buffer shape (applies equally to `collect_set`): both are `TypedImperativeAggregate`s, so Spark's `aggBufferAttributes` declares the intermediate buffer as `BinaryType` (serialized state) while the native accumulator's `state_fields` is a `List`. `CometBaseAggregate.adjustOutputForNativeState` rewrites the Comet-side Partial output to the list shape. Neither collector can split a Partial and Final across Comet and Spark, and a multi-stage distinct rewrite (which inserts a `PartialMerge` stage) forces the whole chain back to Spark ([#4724](https://github.com/apache/datafusion-comet/issues/4724)).
6162
- Spark 4.2 (preview): `CollectList` and `CollectSet` gain an `ignoreNulls` field (default `true`); `RESPECT NULLS` sets it to `false` and keeps null elements. The native path always drops nulls, so `CometCollectShim` reads the field per Spark version (always `true` on 3.4-4.1) and `CometCollectList` / `CometCollectSet` report `Unsupported` when it is `false`, falling back to Spark.
6263

64+
## collect_set
65+
66+
- Spark 3.4.3 (audited 2026-07-27): `CollectSet` extends `Collect[mutable.HashSet[Any]]`, returns `ArrayType(child.dataType, containsNull = false)`, ignores NULL inputs in `update()` (the same Hive-compatible semantics as `collect_list`), and yields an empty array as `defaultResult`. `nullable = false`. Unlike `CollectList` it overrides `checkInputDataTypes` and rejects any input whose type recursively contains a `MapType` (`UNSUPPORTED_INPUT_TYPE`). `convertToBufferElement` copies the value with `InternalRow.copyValue`, except for `BinaryType`, which is wrapped in an `UnsafeArrayData` so that byte arrays dedup by content rather than by identity. Deduplication is Scala `mutable.HashSet` equality on the boxed value, which for floating-point types is numeric `==`: repeated `NaN`s are each kept as separate elements, while `0.0` and `-0.0` collapse to one. Registered only as `collect_set` in `FunctionRegistry` (there is no second alias, unlike `collect_list`/`array_agg`).
67+
- Spark 3.5.8 (audited 2026-07-27): identical to 3.4.3.
68+
- Spark 4.0.1 (audited 2026-07-27): adds `with UnaryLike[Expression]` to the case class, and `checkInputDataTypes` additionally requires `UnsafeRowUtils.isBinaryStable(child.dataType)`, so non-default-collation strings are rejected along with maps. Deduplication semantics unchanged.
69+
- Spark 4.1.1 (audited 2026-07-27): identical to 4.0.1.
70+
- Comet implementation: the native side delegates to `datafusion_spark::function::aggregate::collect::SparkCollectSet`, which wraps `DistinctArrayAggAccumulator` with `ignore_nulls = true` in a `NullToEmptyListAccumulator` so a final NULL accumulator state becomes an empty array. The `containsNull` mismatch against Spark's declared output type, and its rationale, are identical to [collect_list](#collect-list).
71+
- `CometCollectSet` reports `Incompatible` for float and double input when `spark.comet.exec.strictFloatingPoint=true`, because the native distinct comparison treats `NaN == NaN` and collapses repeated `NaN`s into a single element while Spark keeps each one. The native path for floating-point input is then opt-in via `spark.comet.expression.CollectSet.allowIncompatible=true`. All other input types are `Compatible`.
72+
6373
## median
6474

6575
- Spark 3.4.3 (audited 2026-06-24): `Median(child)` is a `RuntimeReplaceableAggregate` with `replacement = Percentile(child, Literal(0.5))`. Catalyst rewrites `median(x)` to `percentile(x, 0.5)` before Comet sees the plan, so it is served by `CometPercentile`.

0 commit comments

Comments
 (0)