Skip to content

Commit b1cc9fc

Browse files
committed
Merge remote-tracking branch 'apache/main' into normalize-float-comparison-operands
2 parents 44c18d6 + f980d6f commit b1cc9fc

149 files changed

Lines changed: 10695 additions & 1753 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-expression-pr/SKILL.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ Location: `spark/src/main/scala/org/apache/comet/serde/`
7474
- [ ] `getSupportLevel` reflects true compatibility rather than the happy path
7575
- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on)
7676
- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md`
77+
- [ ] A change that routes a whole class of expressions to the JVM codegen dispatcher lists the new
78+
shapes it admits and tests one of each, including decimal results whose scale differs from the
79+
declared type and boolean inputs from sliced batches (#6424, #6425)
7780

7881
### Registration in `QueryPlanSerde.scala`
7982

@@ -103,6 +106,12 @@ Location: `native/spark-expr/src/`, registered in `comet_scalar_funcs.rs`.
103106
- [ ] No panics. Use `Result`.
104107
- [ ] Batch operations rather than row-by-row where a kernel exists
105108
- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy`
109+
- [ ] An aggregate merges partial states in Spark's order of operations, not just with Spark's
110+
buffer layout. A different floating-point order fails Spark's exact checks, such as `m2 == 0`,
111+
when a constant is merged from several partitions (#6423).
112+
- [ ] Comparisons, grouping and set operations treat `-0.0` and `0.0`, and NaN payloads, as Spark
113+
does, including inside arrays and structs. Where a DataFusion kernel treats them differently,
114+
the serde falls back (#5507, #5701).
106115

107116
Before accepting a hand-written kernel, ask whether the function already exists upstream in
108117
DataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying
@@ -163,6 +172,11 @@ single file by appending a substring of its name to the suite argument.
163172
- [ ] Timezone handling tested for timestamp and datetime expressions, including a non-UTC session
164173
timezone and timestamps with and without timezone
165174
- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark
175+
- [ ] For a function from another project, such as an Iceberg transform, the expected values come
176+
from that project's Java implementation, which is what Spark runs, not from iceberg-rust
177+
(#6426)
178+
- [ ] A `query ignore(...)` for a result that differs from Spark comes with a fallback for that
179+
case, so the divergence doesn't ship natively by default (#5701)
166180
- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw
167181
- [ ] One expression per SQL file
168182
- [ ] Comet Scala literal tests disable constant folding:

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,10 @@ what test was added. Ask for at least one of:
182182
- A trace comparing `jemalloc_allocated` against the summed
183183
`thread_NNN_comet_memory_reserved` values, which is the only way to see the accounting gap
184184
- Spill counts from `spark.comet.explain.native.enabled=true`, before and after
185+
- For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills
186+
under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the
187+
ability to spill again during that replay, and only a run at a small off-heap size showed it
188+
(#6254).
185189
- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or
186190
`OOMKilled` is the cgroup. 52 with `java.lang.OutOfMemoryError` is JVM heap. A failed task with
187191
`SparkOutOfMemoryError` and a surviving executor is Spark's pool, the only one of the three that

‎.ai/skills/review-comet-pr/SKILL.md‎

Lines changed: 115 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -97,34 +97,142 @@ A PR that quietly marks something `Compatible` while the diff shows a known dive
9797
most important thing to catch. Keep reasons concise and link a tracking issue when the behavior is
9898
known to differ.
9999

100+
Two disguised forms of the same thing:
101+
102+
- A divergence documented in the compatibility guide on a path that is enabled by default. If the
103+
guide says the results differ, the default has to fall back for that case (#5469, #5507).
104+
- A test switched to `ignore` in the PR because Comet's answer now differs from Spark's. An ignored
105+
test is a known divergence, so the code needs a fallback for that case (#5701).
106+
107+
### Behavior change against the latest release
108+
109+
Users upgrade from a release, not from `main`. So the question is not only what the PR changes
110+
relative to `main`, but whether a user moving from the latest release to a build with this PR sees
111+
different behavior. `main` may already carry unreleased changes in the same code, and a PR that
112+
looks harmless against `main` can finish turning a released behavior into a different one.
113+
114+
Find the latest release branch, the highest `branch-X.Y`:
115+
116+
```shell
117+
git ls-remote --heads https://github.com/apache/datafusion-comet 'branch-*' | sort -t- -k2 -V | tail -1
118+
```
119+
120+
Fetch the PR head and diff the files the PR touches against that branch, not against `main`:
121+
122+
```shell
123+
git fetch https://github.com/apache/datafusion-comet branch-X.Y:release-X.Y pull/<pr>/head:pr-<pr>
124+
git diff release-X.Y pr-<pr> -- <changed files>
125+
```
126+
127+
Then ask, for each code path the PR touches, whether any of these differ from the release:
128+
129+
- the result for some input, including null, NaN, `-0.0`, empty, overflow, and timezone cases
130+
- whether a query raises an error, and which error
131+
- whether an operator or expression runs natively or falls back to Spark
132+
- a config default, a config name, or a support level
133+
- performance or memory use on an existing path
134+
135+
The tests the PR adds are a good probe. If a new test would fail on the release branch, the PR
136+
changes released behavior, and you should know which of those two cases it is. Running the test
137+
against the release branch is the cheapest way to find out when the answer is not obvious from
138+
the code.
139+
140+
Every behavior change against the release is one of two things:
141+
142+
- **Intended.** A bug fix that makes Comet match Spark, or a deliberate change. The PR description
143+
should say so, user-facing changes need a note in the user guide or compatibility docs, and a
144+
correctness fix should be considered for backport to the release branches per
145+
`docs/source/contributor-guide/backporting.md`.
146+
- **Unintended.** The PR, alone or together with unreleased changes already on `main`, makes Comet
147+
diverge from Spark where the release did not, or makes a released path slower. This is a
148+
correctness or performance regression, and it falls under the request-changes rule below.
149+
150+
Look hardest when the PR is one of the three kinds of change behind most of the regressions that the
151+
1.1.0 audit found ([#6399](https://github.com/apache/datafusion-comet/issues/6399)):
152+
153+
- **A path that becomes native by default.** The query used to fall back to Spark and was right.
154+
Find the inputs where the native path differs from Spark, and either test them or fall back for
155+
them. Making map and struct literals native exposed a type mismatch in the native `IF` (#6334).
156+
Constant metadata columns went native with a per-split value that DataFusion and Spark assign
157+
differently (#6505).
158+
- **A removed fallback or guard.** List everything the fallback was shielding, not just the case the
159+
PR is about. Removing the Iceberg complex-type null-check fallback also exposed every `explode` of
160+
an Iceberg array to a schema-evolution bug in iceberg-rust (#6504).
161+
- **A broad routing change.** A catch-all that sends more expressions down a path, such as the JVM
162+
codegen dispatcher, admits shapes the path never handled. List the new shapes and test one of each
163+
(#6424, #6425).
164+
165+
Report the comparison in the review even when nothing changed, so the reviewer knows it was done.
166+
100167
### Spark version coverage
101168

102169
Comet supports several Spark versions. Version-specific behavior belongs in the shims under
103170
`spark/src/main/spark-{3.4,3.5,3.x,4.0,4.1,4.2}/org/apache/comet/shims/`, not in branches on a
104171
version string in shared code, and not in native Rust. If the PR adds a shim for one 4.x version,
105172
check that the sibling 4.x source sets got it too.
106173

174+
When Spark changed the behavior in a patch release, such as SPARK-55969 or SPARK-54918, a check on
175+
the minor version is wrong for every earlier patch. CI builds only the newest patch of each line, so
176+
it can't catch that (#6042, #5701). The pull request CI also runs only the default Spark profile, so
177+
logic that depends on the Spark version needs the matching `run-spark-*` labels (#6156).
178+
107179
### Configuration
108180

109181
New configs go in `CometConf.scala` and must follow
110182
`docs/source/contributor-guide/config_conventions.md`. Check the naming, the default, the category,
111183
and that the description reads as user-facing documentation, because it is. `configs.md` is
112184
generated from it.
113185

186+
A new behavior that is on by default and trades performance for some workloads needs a supported
187+
config, not a testing one, to turn it off before merge (#6466).
188+
114189
### Tests
115190

116191
- Does the PR test the thing it changed, or only that nothing else broke?
117192
- Are the tests in the right framework for the area? The area skills say which.
118193
- Does a bug fix come with a test that fails without the fix?
194+
- Do the tests compare against Spark? A comparison with Comet's own accumulator, or with a second
195+
implementation such as iceberg-rust instead of the Iceberg Java that Spark runs, can agree on the
196+
wrong answer (#6423, #6426).
119197
- New suites must be registered in both `.github/workflows/pr_build_linux.yml` and
120198
`pr_build_macos.yml`.
121199

200+
For a change on a default path, look for tests with the inputs that broke earlier changes:
201+
202+
- `-0.0` next to `0.0`, and NaN with different payloads, including inside arrays and structs (#5469,
203+
#5507, #5701)
204+
- Values at a type or unit boundary, including negative timestamps before 1970 (#6426)
205+
- A constant that binary floating point can't represent, such as 0.1, aggregated across partitions
206+
(#6423)
207+
- A batch where every row takes the same branch (#6334)
208+
- Output that the native side slices, so it reaches the JVM past the first batch with a non-zero
209+
offset (#6464)
210+
- A Parquet file that Spark splits into several partitions, an empty file, and an Iceberg file
211+
written before a nested field was added (#6504, #6505, #6506)
212+
- Data that defeats a heuristic, such as distinct keys at the start of a task followed by repeats
213+
(#6466)
214+
- One of a pair of aliases configured without the other (#5825)
215+
122216
### CI
123217

124218
`gh pr checks <pr> --repo apache/datafusion-comet`, and again with `--failed` for detail.
125219

126220
Summarize failures in the review. Do not compare against failures on `main`.
127221

222+
Some paths have no CI coverage at all. Nothing in CI reads from a real object store, runs Iceberg's
223+
forward-compatibility tables, or makes a spilled operator replay under a tight memory pool. A change
224+
on one of those paths needs a local run, and the review should ask for one (#5759, #6254).
225+
226+
### Dependency upgrades
227+
228+
Review a dependency bump as a set of behavior changes, not an API migration. For DataFusion, arrow,
229+
parquet and iceberg-rust, diff the upstream source of what Comet calls on default paths. Both
230+
versions are in the local Cargo registry after a build of each side. A test that the bump changes to
231+
`ignore` or to a new expected value marks a behavior change that needs a fallback or an explicit
232+
decision. The DataFusion 55 upgrade changed signed-zero handling in `array_distinct` and
233+
`array_union` (#5701), the spill replay of the final aggregate (#6254), and, through iceberg-rust,
234+
task validation (#5759).
235+
128236
## 6. Documentation Freshness
129237

130238
Every PR carries a documentation question, and it has two halves.
@@ -192,13 +300,16 @@ Present your review as guidance for the reviewer:
192300
1. **PR Summary**, brief description of what the PR does
193301
2. **Areas**, which sibling skills you loaded and why
194302
3. **CI Status**, summary of CI check results
195-
4. **Findings**, organized by area
196-
5. **Suggested Review Comments**, specific comments the reviewer could leave, with file and line
303+
4. **Behavior vs Release**, which release branch you compared against, and every behavior change
304+
you found, each marked intended or unintended. Say "no change" when there is none.
305+
5. **Findings**, organized by area
306+
6. **Suggested Review Comments**, specific comments the reviewer could leave, with file and line
197307
references. Everything here is something you expect the author to address. Anything that did not
198308
clear the bar above should not appear.
199-
6. **Review State**, how to submit the review. Use the first case that applies:
309+
7. **Review State**, how to submit the review. Use the first case that applies:
200310
- **Request changes** when any finding is a correctness problem that the PR introduces or a
201-
performance regression. Name those findings.
311+
performance regression, including an unintended behavior change against the latest release.
312+
Name those findings.
202313
- **Approve**, or dismiss the earlier review, when this is a re-review and the findings behind the
203314
reviewer's earlier **Request changes** review have all been addressed.
204315
- **Comment** otherwise.

‎.ai/skills/review-comet-shuffle-pr/SKILL.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,14 @@ keys**, and it is not the same rule for the two partitionings:
5555
and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar
5656
float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
5757
[#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
58-
normalizes its comparison keys and its sampled boundary rows the same way the native sort does,
59-
and `CometSortOrder.getSupportLevel` returns `Compatible()` for them regardless of strict mode.
60-
What strict mode still governs is floating point _nested_ in an array, struct, or map
61-
([#5507](https://github.com/apache/datafusion-comet/issues/5507)), which is already out as a
62-
range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on floating-point
63-
uses native shuffle" under both settings of the config.
58+
normalizes its comparison keys and its sampled boundary rows the same way the native sort does.
59+
`CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats
60+
nested in arrays and structs too, because the native sort normalizes them, unless the key's type
61+
can hold a null element or field
62+
([#6476](https://github.com/apache/datafusion-comet/issues/6476),
63+
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), which strict mode declines.
64+
A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs
65+
"range partitioning on floating-point uses native shuffle" under both settings of the config.
6466
- **`HashPartitioning` is primitive-only by default only.** With
6567
`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),
6668
`supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark

‎.github/workflows/ci_label.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,12 @@ jobs:
4747
# A called workflow's jobs can hold no more than the caller grants, and
4848
# GitHub checks that for every job when the run starts, including the
4949
# ones this event skips. This is the union of what ci.yml's jobs request:
50-
# `changes` and `nightly_report` read the Actions API, and
51-
# `nightly_report` files issues (it only runs on the schedule).
50+
# `changes` and `nightly_report` read the Actions API, `nightly_report`
51+
# files issues (it only runs on the schedule), and `docs` pushes the built
52+
# site to asf-site (it only runs on main). Leaving any of them out fails
53+
# every label run at startup, before a single job runs.
5254
permissions:
5355
actions: read
54-
contents: read
56+
contents: write
5557
issues: write
5658
uses: ./.github/workflows/ci.yml

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,7 @@ jobs:
512512
org.apache.comet.exec.CometExecSuite
513513
org.apache.comet.exec.CometTopKSuite
514514
org.apache.comet.exec.CometEmptyRelationExecSuite
515+
org.apache.comet.exec.CometRangeExecSuite
515516
org.apache.comet.exec.CometInMemoryCacheSuite
516517
org.apache.comet.exec.CometInMemoryCachePruningSuite
517518
org.apache.comet.exec.CometInMemoryCacheKryoSuite

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ jobs:
216216
org.apache.comet.exec.CometExecSuite
217217
org.apache.comet.exec.CometTopKSuite
218218
org.apache.comet.exec.CometEmptyRelationExecSuite
219+
org.apache.comet.exec.CometRangeExecSuite
219220
org.apache.comet.exec.CometInMemoryCacheSuite
220221
org.apache.comet.exec.CometInMemoryCachePruningSuite
221222
org.apache.comet.exec.CometInMemoryCacheKryoSuite

‎dev/ci/check-ci-config.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1202,11 +1202,57 @@ def check_label_runs_separate():
12021202
f"and nothing else; any other type would run ci.yml twice per push"
12031203
)
12041204

1205+
# ci_label.yml calls ci.yml with an explicit `permissions:` ceiling, and
1206+
# GitHub refuses to start the run at all if any job in ci.yml asks for more,
1207+
# even a job the `labeled` event would skip. That is how the `docs` job's
1208+
# `contents: write` stopped every label run on 2026-09-29.
1209+
if label_workflow.exists():
1210+
granted = permission_blocks(label_workflow.read_text(encoding="utf-8").splitlines())
1211+
ceiling = {}
1212+
for block in granted:
1213+
for scope, level in block.items():
1214+
ceiling[scope] = max(ceiling.get(scope, 0), PERMISSION_RANK[level])
1215+
ci = (WORKFLOWS / "ci.yml").read_text(encoding="utf-8").splitlines()
1216+
for block in permission_blocks(ci):
1217+
for scope, level in block.items():
1218+
if PERMISSION_RANK[level] > ceiling.get(scope, 0):
1219+
failures.append(
1220+
f"ci.yml requests `{scope}: {level}`, but ci_label.yml grants "
1221+
f"the call at most `{scope}: "
1222+
f"{PERMISSION_NAME[ceiling.get(scope, 0)]}`, so every label "
1223+
f"run fails at startup. Raise the grant in ci_label.yml"
1224+
)
1225+
12051226
for failure in failures:
12061227
print(f"label runs: {failure}")
12071228
return not failures
12081229

12091230

1231+
PERMISSION_RANK = {"none": 0, "read": 1, "write": 2}
1232+
PERMISSION_NAME = {rank: name for name, rank in PERMISSION_RANK.items()}
1233+
PERMISSIONS_KEY = re.compile(r"^(\s*)permissions:\s*(\{\s*\})?\s*$")
1234+
PERMISSION_ENTRY = re.compile(r"^(\s*)([a-z-]+):\s*(read|write|none)\s*$")
1235+
1236+
1237+
def permission_blocks(lines):
1238+
"""Every `permissions:` mapping in a workflow, as a list of {scope: level}."""
1239+
blocks = []
1240+
for index, line in enumerate(lines):
1241+
match = PERMISSIONS_KEY.match(line)
1242+
if not match:
1243+
continue
1244+
indent, block = len(match.group(1)), {}
1245+
for entry in lines[index + 1:]:
1246+
if not entry.strip() or entry.lstrip().startswith("#"):
1247+
continue
1248+
found = PERMISSION_ENTRY.match(entry)
1249+
if not found or len(found.group(1)) <= indent:
1250+
break
1251+
block[found.group(2)] = found.group(3)
1252+
blocks.append(block)
1253+
return blocks
1254+
1255+
12101256
def caller_passes_cache_refresh(ci_lines, workflow_name):
12111257
"""Does the ci.yml job that `uses:` this workflow set `cache-refresh-only:`?"""
12121258
job_start = None

0 commit comments

Comments
 (0)