Skip to content

Commit 46faab0

Browse files
committed
Merge remote-tracking branch 'origin/main' into feat/array-contains-float-native
# Conflicts: # docs/source/user-guide/latest/compatibility/floating-point.md
2 parents 964c9c8 + fe5c41a commit 46faab0

124 files changed

Lines changed: 9569 additions & 1214 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-expression-pr/SKILL.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ Location: `spark/src/main/scala/org/apache/comet/serde/`
7474
- [ ] `getSupportLevel` reflects true compatibility rather than the happy path
7575
- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on)
7676
- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md`
77+
- [ ] A change that routes a whole class of expressions to the JVM codegen dispatcher lists the new
78+
shapes it admits and tests one of each, including decimal results whose scale differs from the
79+
declared type and boolean inputs from sliced batches (#6424, #6425)
7780

7881
### Registration in `QueryPlanSerde.scala`
7982

@@ -103,6 +106,12 @@ Location: `native/spark-expr/src/`, registered in `comet_scalar_funcs.rs`.
103106
- [ ] No panics. Use `Result`.
104107
- [ ] Batch operations rather than row-by-row where a kernel exists
105108
- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy`
109+
- [ ] An aggregate merges partial states in Spark's order of operations, not just with Spark's
110+
buffer layout. A different floating-point order fails Spark's exact checks, such as `m2 == 0`,
111+
when a constant is merged from several partitions (#6423).
112+
- [ ] Comparisons, grouping and set operations treat `-0.0` and `0.0`, and NaN payloads, as Spark
113+
does, including inside arrays and structs. Where a DataFusion kernel treats them differently,
114+
the serde falls back (#5507, #5701).
106115

107116
Before accepting a hand-written kernel, ask whether the function already exists upstream in
108117
DataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying
@@ -163,6 +172,11 @@ single file by appending a substring of its name to the suite argument.
163172
- [ ] Timezone handling tested for timestamp and datetime expressions, including a non-UTC session
164173
timezone and timestamps with and without timezone
165174
- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark
175+
- [ ] For a function from another project, such as an Iceberg transform, the expected values come
176+
from that project's Java implementation, which is what Spark runs, not from iceberg-rust
177+
(#6426)
178+
- [ ] A `query ignore(...)` for a result that differs from Spark comes with a fallback for that
179+
case, so the divergence doesn't ship natively by default (#5701)
166180
- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw
167181
- [ ] One expression per SQL file
168182
- [ ] Comet Scala literal tests disable constant folding:

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,10 @@ what test was added. Ask for at least one of:
182182
- A trace comparing `jemalloc_allocated` against the summed
183183
`thread_NNN_comet_memory_reserved` values, which is the only way to see the accounting gap
184184
- Spill counts from `spark.comet.explain.native.enabled=true`, before and after
185+
- For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills
186+
under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the
187+
ability to spill again during that replay, and only a run at a small off-heap size showed it
188+
(#6254).
185189
- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or
186190
`OOMKilled` is the cgroup. 52 with `java.lang.OutOfMemoryError` is JVM heap. A failed task with
187191
`SparkOutOfMemoryError` and a surviving executor is Spark's pool, the only one of the three that

‎.ai/skills/review-comet-pr/SKILL.md‎

Lines changed: 115 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -97,34 +97,142 @@ A PR that quietly marks something `Compatible` while the diff shows a known dive
9797
most important thing to catch. Keep reasons concise and link a tracking issue when the behavior is
9898
known to differ.
9999

100+
Two disguised forms of the same thing:
101+
102+
- A divergence documented in the compatibility guide on a path that is enabled by default. If the
103+
guide says the results differ, the default has to fall back for that case (#5469, #5507).
104+
- A test switched to `ignore` in the PR because Comet's answer now differs from Spark's. An ignored
105+
test is a known divergence, so the code needs a fallback for that case (#5701).
106+
107+
### Behavior change against the latest release
108+
109+
Users upgrade from a release, not from `main`. So the question is not only what the PR changes
110+
relative to `main`, but whether a user moving from the latest release to a build with this PR sees
111+
different behavior. `main` may already carry unreleased changes in the same code, and a PR that
112+
looks harmless against `main` can finish turning a released behavior into a different one.
113+
114+
Find the latest release branch, the highest `branch-X.Y`:
115+
116+
```shell
117+
git ls-remote --heads https://github.com/apache/datafusion-comet 'branch-*' | sort -t- -k2 -V | tail -1
118+
```
119+
120+
Fetch the PR head and diff the files the PR touches against that branch, not against `main`:
121+
122+
```shell
123+
git fetch https://github.com/apache/datafusion-comet branch-X.Y:release-X.Y pull/<pr>/head:pr-<pr>
124+
git diff release-X.Y pr-<pr> -- <changed files>
125+
```
126+
127+
Then ask, for each code path the PR touches, whether any of these differ from the release:
128+
129+
- the result for some input, including null, NaN, `-0.0`, empty, overflow, and timezone cases
130+
- whether a query raises an error, and which error
131+
- whether an operator or expression runs natively or falls back to Spark
132+
- a config default, a config name, or a support level
133+
- performance or memory use on an existing path
134+
135+
The tests the PR adds are a good probe. If a new test would fail on the release branch, the PR
136+
changes released behavior, and you should know which of those two cases it is. Running the test
137+
against the release branch is the cheapest way to find out when the answer is not obvious from
138+
the code.
139+
140+
Every behavior change against the release is one of two things:
141+
142+
- **Intended.** A bug fix that makes Comet match Spark, or a deliberate change. The PR description
143+
should say so, user-facing changes need a note in the user guide or compatibility docs, and a
144+
correctness fix should be considered for backport to the release branches per
145+
`docs/source/contributor-guide/backporting.md`.
146+
- **Unintended.** The PR, alone or together with unreleased changes already on `main`, makes Comet
147+
diverge from Spark where the release did not, or makes a released path slower. This is a
148+
correctness or performance regression, and it falls under the request-changes rule below.
149+
150+
Look hardest when the PR is one of the three kinds of change behind most of the regressions that the
151+
1.1.0 audit found ([#6399](https://github.com/apache/datafusion-comet/issues/6399)):
152+
153+
- **A path that becomes native by default.** The query used to fall back to Spark and was right.
154+
Find the inputs where the native path differs from Spark, and either test them or fall back for
155+
them. Making map and struct literals native exposed a type mismatch in the native `IF` (#6334).
156+
Constant metadata columns went native with a per-split value that DataFusion and Spark assign
157+
differently (#6505).
158+
- **A removed fallback or guard.** List everything the fallback was shielding, not just the case the
159+
PR is about. Removing the Iceberg complex-type null-check fallback also exposed every `explode` of
160+
an Iceberg array to a schema-evolution bug in iceberg-rust (#6504).
161+
- **A broad routing change.** A catch-all that sends more expressions down a path, such as the JVM
162+
codegen dispatcher, admits shapes the path never handled. List the new shapes and test one of each
163+
(#6424, #6425).
164+
165+
Report the comparison in the review even when nothing changed, so the reviewer knows it was done.
166+
100167
### Spark version coverage
101168

102169
Comet supports several Spark versions. Version-specific behavior belongs in the shims under
103170
`spark/src/main/spark-{3.4,3.5,3.x,4.0,4.1,4.2}/org/apache/comet/shims/`, not in branches on a
104171
version string in shared code, and not in native Rust. If the PR adds a shim for one 4.x version,
105172
check that the sibling 4.x source sets got it too.
106173

174+
When Spark changed the behavior in a patch release, such as SPARK-55969 or SPARK-54918, a check on
175+
the minor version is wrong for every earlier patch. CI builds only the newest patch of each line, so
176+
it can't catch that (#6042, #5701). The pull request CI also runs only the default Spark profile, so
177+
logic that depends on the Spark version needs the matching `run-spark-*` labels (#6156).
178+
107179
### Configuration
108180

109181
New configs go in `CometConf.scala` and must follow
110182
`docs/source/contributor-guide/config_conventions.md`. Check the naming, the default, the category,
111183
and that the description reads as user-facing documentation, because it is. `configs.md` is
112184
generated from it.
113185

186+
A new behavior that is on by default and trades performance for some workloads needs a supported
187+
config, not a testing one, to turn it off before merge (#6466).
188+
114189
### Tests
115190

116191
- Does the PR test the thing it changed, or only that nothing else broke?
117192
- Are the tests in the right framework for the area? The area skills say which.
118193
- Does a bug fix come with a test that fails without the fix?
194+
- Do the tests compare against Spark? A comparison with Comet's own accumulator, or with a second
195+
implementation such as iceberg-rust instead of the Iceberg Java that Spark runs, can agree on the
196+
wrong answer (#6423, #6426).
119197
- New suites must be registered in both `.github/workflows/pr_build_linux.yml` and
120198
`pr_build_macos.yml`.
121199

200+
For a change on a default path, look for tests with the inputs that broke earlier changes:
201+
202+
- `-0.0` next to `0.0`, and NaN with different payloads, including inside arrays and structs (#5469,
203+
#5507, #5701)
204+
- Values at a type or unit boundary, including negative timestamps before 1970 (#6426)
205+
- A constant that binary floating point can't represent, such as 0.1, aggregated across partitions
206+
(#6423)
207+
- A batch where every row takes the same branch (#6334)
208+
- Output that the native side slices, so it reaches the JVM past the first batch with a non-zero
209+
offset (#6464)
210+
- A Parquet file that Spark splits into several partitions, an empty file, and an Iceberg file
211+
written before a nested field was added (#6504, #6505, #6506)
212+
- Data that defeats a heuristic, such as distinct keys at the start of a task followed by repeats
213+
(#6466)
214+
- One of a pair of aliases configured without the other (#5825)
215+
122216
### CI
123217

124218
`gh pr checks <pr> --repo apache/datafusion-comet`, and again with `--failed` for detail.
125219

126220
Summarize failures in the review. Do not compare against failures on `main`.
127221

222+
Some paths have no CI coverage at all. Nothing in CI reads from a real object store, runs Iceberg's
223+
forward-compatibility tables, or makes a spilled operator replay under a tight memory pool. A change
224+
on one of those paths needs a local run, and the review should ask for one (#5759, #6254).
225+
226+
### Dependency upgrades
227+
228+
Review a dependency bump as a set of behavior changes, not an API migration. For DataFusion, arrow,
229+
parquet and iceberg-rust, diff the upstream source of what Comet calls on default paths. Both
230+
versions are in the local Cargo registry after a build of each side. A test that the bump changes to
231+
`ignore` or to a new expected value marks a behavior change that needs a fallback or an explicit
232+
decision. The DataFusion 55 upgrade changed signed-zero handling in `array_distinct` and
233+
`array_union` (#5701), the spill replay of the final aggregate (#6254), and, through iceberg-rust,
234+
task validation (#5759).
235+
128236
## 6. Documentation Freshness
129237

130238
Every PR carries a documentation question, and it has two halves.
@@ -192,13 +300,16 @@ Present your review as guidance for the reviewer:
192300
1. **PR Summary**, brief description of what the PR does
193301
2. **Areas**, which sibling skills you loaded and why
194302
3. **CI Status**, summary of CI check results
195-
4. **Findings**, organized by area
196-
5. **Suggested Review Comments**, specific comments the reviewer could leave, with file and line
303+
4. **Behavior vs Release**, which release branch you compared against, and every behavior change
304+
you found, each marked intended or unintended. Say "no change" when there is none.
305+
5. **Findings**, organized by area
306+
6. **Suggested Review Comments**, specific comments the reviewer could leave, with file and line
197307
references. Everything here is something you expect the author to address. Anything that did not
198308
clear the bar above should not appear.
199-
6. **Review State**, how to submit the review. Use the first case that applies:
309+
7. **Review State**, how to submit the review. Use the first case that applies:
200310
- **Request changes** when any finding is a correctness problem that the PR introduces or a
201-
performance regression. Name those findings.
311+
performance regression, including an unintended behavior change against the latest release.
312+
Name those findings.
202313
- **Approve**, or dismiss the earlier review, when this is a re-review and the findings behind the
203314
reviewer's earlier **Request changes** review have all been addressed.
204315
- **Comment** otherwise.

‎.ai/skills/review-comet-shuffle-pr/SKILL.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,14 @@ keys**, and it is not the same rule for the two partitionings:
5555
and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar
5656
float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
5757
[#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
58-
normalizes its comparison keys and its sampled boundary rows the same way the native sort does,
59-
and `CometSortOrder.getSupportLevel` returns `Compatible()` for them regardless of strict mode.
60-
What strict mode still governs is floating point _nested_ in an array, struct, or map
61-
([#5507](https://github.com/apache/datafusion-comet/issues/5507)), which is already out as a
62-
range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on floating-point
63-
uses native shuffle" under both settings of the config.
58+
normalizes its comparison keys and its sampled boundary rows the same way the native sort does.
59+
`CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats
60+
nested in arrays and structs too, because the native sort normalizes them, unless the key's type
61+
can hold a null element or field
62+
([#6476](https://github.com/apache/datafusion-comet/issues/6476),
63+
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), which strict mode declines.
64+
A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs
65+
"range partitioning on floating-point uses native shuffle" under both settings of the config.
6466
- **`HashPartitioning` is primitive-only by default only.** With
6567
`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),
6668
`supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,7 @@ jobs:
512512
org.apache.comet.exec.CometExecSuite
513513
org.apache.comet.exec.CometTopKSuite
514514
org.apache.comet.exec.CometEmptyRelationExecSuite
515+
org.apache.comet.exec.CometRangeExecSuite
515516
org.apache.comet.exec.CometInMemoryCacheSuite
516517
org.apache.comet.exec.CometInMemoryCachePruningSuite
517518
org.apache.comet.exec.CometInMemoryCacheKryoSuite
@@ -571,6 +572,7 @@ jobs:
571572
org.apache.spark.sql.CometSortCollationSuite
572573
org.apache.comet.CometFuzzAggregateSuite
573574
org.apache.spark.sql.comet.execution.arrow.CometArrowStreamSuite
575+
org.apache.spark.sql.comet.execution.arrow.CometArrowWriterSuite
574576
org.apache.spark.sql.comet.execution.arrow.CometStringWriterSuite
575577
org.apache.spark.sql.CometSparkInternalFunctionsSuite
576578
- name: "expressions"

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ jobs:
216216
org.apache.comet.exec.CometExecSuite
217217
org.apache.comet.exec.CometTopKSuite
218218
org.apache.comet.exec.CometEmptyRelationExecSuite
219+
org.apache.comet.exec.CometRangeExecSuite
219220
org.apache.comet.exec.CometInMemoryCacheSuite
220221
org.apache.comet.exec.CometInMemoryCachePruningSuite
221222
org.apache.comet.exec.CometInMemoryCacheKryoSuite
@@ -275,6 +276,7 @@ jobs:
275276
org.apache.spark.sql.CometSortCollationSuite
276277
org.apache.comet.CometFuzzAggregateSuite
277278
org.apache.spark.sql.comet.execution.arrow.CometArrowStreamSuite
279+
org.apache.spark.sql.comet.execution.arrow.CometArrowWriterSuite
278280
org.apache.spark.sql.comet.execution.arrow.CometStringWriterSuite
279281
org.apache.spark.sql.CometSparkInternalFunctionsSuite
280282
- name: "expressions"

‎dev/release/build-release-comet.sh‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ This script builds comet native binaries inside a docker image. The image is nam
3434
Options are:
3535
3636
-r [repo] : git repo (default: ${REPO})
37-
-b [branch] : git branch (default: ${BRANCH})
37+
-b [ref] : git branch or tag to build (required)
3838
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
3939
EOF
4040
exit 1
@@ -59,10 +59,10 @@ function cleanup()
5959

6060
trap cleanup SIGINT SIGTERM EXIT
6161

62-
CLEANUP=1
62+
CLEANUP=0
6363

6464
REPO="https://github.com/apache/datafusion-comet.git"
65-
BRANCH="release"
65+
BRANCH=
6666
MACOS_SDK=
6767
HAS_MACOS_SDK="false"
6868
IMGTAG=latest
@@ -77,6 +77,26 @@ while getopts "b:hr:t:" opt; do
7777
esac
7878
done
7979

80+
if [ -z "$BRANCH" ]
81+
then
82+
echo "Error: -b must specify the branch or tag to build"
83+
usage
84+
fi
85+
86+
if git ls-remote --exit-code "$REPO" "refs/heads/$BRANCH" "refs/tags/$BRANCH" >/dev/null
87+
then
88+
echo "Found branch or tag $BRANCH in $REPO"
89+
else
90+
STATUS=$?
91+
if [ "$STATUS" -eq 2 ]
92+
then
93+
echo "Error: branch or tag '$BRANCH' does not exist in $REPO"
94+
else
95+
echo "Error: unable to check branch or tag '$BRANCH' in $REPO"
96+
fi
97+
exit 1
98+
fi
99+
80100
echo "Building binaries from $REPO/$BRANCH"
81101

82102
# Check Java version
@@ -135,6 +155,7 @@ pushd $COMET_HOME_DIR/native && cargo clean && popd
135155
# Run the builder container for each architecture. The entrypoint script will build the binaries
136156

137157
# AMD64
158+
CLEANUP=1
138159
echo "Building amd64 binary"
139160
docker run \
140161
--name comet-amd64-builder-container \

‎docs/source/contributor-guide/expression-audits/array_funcs.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,10 @@
5353
## array_distinct
5454

5555
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
56-
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0` are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
56+
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet`, which merges NaNs but keeps `+0.0` and `-0.0` distinct. Wired as `CometScalarFunction("array_distinct")`.
5757
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
5858
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
59-
- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them like Spark's `SQLOpenHashSet`.
59+
- Float/double element types run natively only on Spark 4.2.0, which normalizes the arguments in the plan (SPARK-54918); every other version falls back by default. See [Array distinct and union](../../user-guide/latest/compatibility/floating-point.md#array-distinct-and-union).
6060

6161
## array_except
6262

@@ -141,7 +141,7 @@
141141
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
142142
- Spark 4.0.1 (audited 2026-05-27): `nullIntolerant = true` moves into `ArrayBinaryLike`; overflow path uses `arrayFunctionWithElementsExceedLimitError`.
143143
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
144-
- Float/double NaN and signed-zero canonicalization matches `array_distinct`. Result element ordering also matches Spark (left-side distinct elements followed by new right-side elements).
144+
- Float/double element types follow the same Spark 4.2.0-only gate as `array_distinct`. Result element ordering matches Spark (left-side distinct elements followed by new right-side elements).
145145

146146
## arrays_overlap
147147

0 commit comments

Comments
 (0)