Skip to content

Commit 2923064

Browse files
authored
Merge branch 'main' into fix/analyze-trace-pool-total
2 parents b6d146c + d62d338 commit 2923064

50 files changed

Lines changed: 3662 additions & 504 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
---
2+
name: review-comet-expression-pr
3+
description: Use when reviewing a DataFusion Comet pull request that adds or changes a Spark expression, its Scala serde, its protobuf message, or its native Rust implementation. Load alongside review-comet-pr, which covers the parts of the review that apply to every PR.
4+
argument-hint: <pr-number>
5+
---
6+
7+
<!--
8+
Licensed to the Apache Software Foundation (ASF) under one
9+
or more contributor license agreements. See the NOTICE file
10+
distributed with this work for additional information
11+
regarding copyright ownership. The ASF licenses this file
12+
to you under the Apache License, Version 2.0 (the
13+
"License"); you may not use this file except in compliance
14+
with the License. You may obtain a copy of the License at
15+
16+
http://www.apache.org/licenses/LICENSE-2.0
17+
18+
Unless required by applicable law or agreed to in writing,
19+
software distributed under the License is distributed on an
20+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
21+
KIND, either express or implied. See the License for the
22+
specific language governing permissions and limitations
23+
under the License.
24+
-->
25+
26+
Expression-specific review for Comet PR #$ARGUMENTS.
27+
28+
**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review
29+
bar, and the output format. This skill only covers what is specific to expressions.
30+
31+
## Read the Contributor Guide First
32+
33+
| Doc | What you need from it |
34+
| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
35+
| `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming |
36+
| `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports |
37+
| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel |
38+
39+
Hold the diff against these. If the PR does something a guide says to do differently, either the PR
40+
is wrong or the guide is out of date, and you need to say which.
41+
42+
## 1. Read the Spark Source
43+
44+
**For any PR that adds or modifies an expression, you must read the Spark source to understand the
45+
canonical behavior.** This is the authoritative reference for what Comet must match. Do not review
46+
an expression PR from the diff alone.
47+
48+
Clone Spark into `/tmp/spark` if it is not already there, using a shallow clone, then find the
49+
expression class under `sql/catalyst/src/main/scala` by searching for its `case class` declaration.
50+
51+
Read the Spark implementation carefully. Pay attention to:
52+
53+
- `eval` and `doGenCode` / `nullSafeEval`. These define the exact behavior.
54+
- `inputTypes` and `dataType`. These define which types Spark accepts and what it returns.
55+
- Null handling. Does it use `nullable = true`? Does `nullSafeEval` handle nulls implicitly?
56+
- Special cases, guards, and `require` assertions.
57+
- ANSI mode branches, usually `SQLConf.get.ansiEnabled` or `failOnError`.
58+
- Behavior differences across the Spark versions Comet supports. If the expression changed between
59+
3.5 and 4.0, the serde needs a shim, not a single implementation.
60+
61+
Then read the Spark tests for the expression, under the test source trees in `sql/`.
62+
63+
Compare Spark's behavior against the PR and identify edge cases Spark tests that the PR does not,
64+
types Spark supports that the PR does not handle, and behavioral differences that should be marked
65+
`Incompatible`.
66+
67+
## 2. Scala Serde
68+
69+
Location: `spark/src/main/scala/org/apache/comet/serde/`
70+
71+
- [ ] Expression class correctly identified
72+
- [ ] All child expressions converted via `exprToProtoInternal`
73+
- [ ] Return type correctly serialized
74+
- [ ] `getSupportLevel` reflects true compatibility rather than the happy path
75+
- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on)
76+
- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md`
77+
78+
### Registration in `QueryPlanSerde.scala`
79+
80+
- [ ] Added to the correct map (temporal, string, arithmetic, and so on)
81+
- [ ] No duplicate registrations
82+
- [ ] Import statement added
83+
84+
### Return type collisions with DataFusion built-ins
85+
86+
If the PR registers a Spark function whose name is also defined by `datafusion-functions`, such as
87+
`levenshtein`, `concat`, `coalesce`, `sha2`, or `regexp_replace`, check that the serde sets the
88+
return type explicitly via `scalarFunctionExprToProtoWithReturnType` rather than
89+
`scalarFunctionExprToProto` or the bare `CometScalarFunction(name)` shortcut.
90+
91+
Without an explicit return type the native planner consults DataFusion's UDF registry first for
92+
type resolution, and any arity or input-type difference between the Spark and DataFusion versions
93+
fails native execution with `Error from DataFusion: Function 'X' expects N arguments but received
94+
M`. The Comet UDF is only swapped in after DataFusion's signature validation passes. See "When to
95+
set the return type explicitly" in `adding_a_new_expression.md`.
96+
97+
## 3. Rust Implementation
98+
99+
Location: `native/spark-expr/src/`, registered in `comet_scalar_funcs.rs`.
100+
101+
- [ ] Matches DataFusion and Arrow conventions
102+
- [ ] Null handling is correct, including nulls inside nested types
103+
- [ ] No panics. Use `Result`.
104+
- [ ] Batch operations rather than row-by-row where a kernel exists
105+
- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy`
106+
107+
Before accepting a hand-written kernel, ask whether the function already exists upstream in
108+
DataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying
109+
its own copy.
110+
111+
## 4. Tests
112+
113+
**Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) where possible.** The
114+
framework runs each query through both Spark and Comet and compares results, with no Scala code.
115+
Fall back to Comet Scala Tests in `CometExpressionSuite` only when the SQL framework cannot express
116+
the test, for example complex `DataFrame` setup or programmatic data generation.
117+
118+
Test location: `spark/src/test/resources/sql-tests/expressions/<category>/`, with categories
119+
`aggregate/`, `array/`, `string/`, `math/`, `struct/`, `map/`, `datetime/`, `hash/`, and others.
120+
121+
```sql
122+
statement
123+
CREATE TABLE test_crc32(col string, a int, b float) USING parquet
124+
125+
statement
126+
INSERT INTO test_crc32 VALUES ('Spark', 10, 1.5), (NULL, NULL, NULL), ('', 0, 0.0)
127+
128+
-- default mode: verifies native Comet execution and that the result matches Spark
129+
query
130+
SELECT crc32(col) FROM test_crc32
131+
132+
-- compares results without requiring native execution
133+
query spark_answer_only
134+
SELECT crc32(cast(a as string)) FROM test_crc32
135+
136+
-- allows numeric variance for floating-point results
137+
query tolerance=0.0001
138+
SELECT cos(v) FROM test_trig
139+
140+
-- asserts fallback to Spark occurs
141+
query expect_fallback(unsupported expression)
142+
SELECT unsupported_func(v) FROM test_table
143+
144+
-- verifies both engines throw matching exceptions
145+
query expect_error(ARITHMETIC_OVERFLOW)
146+
SELECT 2147483647 + 1
147+
148+
-- skip a query with a known bug, with the issue link
149+
query ignore(https://github.com/apache/datafusion-comet/issues/NNNN)
150+
SELECT known_buggy_expr(v) FROM test_table
151+
```
152+
153+
Run the whole suite with `-Dsuites="org.apache.comet.CometSqlFileTestSuite" -Dtest=none`, or a
154+
single file by appending a substring of its name to the suite argument.
155+
156+
**Verify all test requirements, whichever framework is used:**
157+
158+
- [ ] Basic functionality tested against column data, not only literals
159+
- [ ] Null handling tested
160+
- [ ] Edge cases tested: empty input, overflow, boundary values, negative values
161+
- [ ] Both literal and column arguments tested, in every combination for multi-argument
162+
expressions. They take different code paths.
163+
- [ ] Timezone handling tested for timestamp and datetime expressions, including a non-UTC session
164+
timezone and timestamps with and without timezone
165+
- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark
166+
- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw
167+
- [ ] One expression per SQL file
168+
- [ ] Comet Scala literal tests disable constant folding:
169+
170+
```scala
171+
withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
172+
"org.apache.spark.sql.catalyst.optimizer.ConstantFolding") {
173+
checkSparkAnswerAndOperator("SELECT func(literal)")
174+
}
175+
```
176+
177+
## 5. Performance
178+
179+
**For PRs that add a new native expression, performance is not optional.** The point of a native
180+
implementation is to be faster than Spark's codegen. If it is not faster, it may not be worth
181+
carrying.
182+
183+
1. Does the PR description contain benchmark numbers comparing Comet against Spark? If not, ask for
184+
them.
185+
2. Does the PR add a benchmark under `spark/src/test/scala/org/apache/spark/sql/benchmark/`?
186+
3. Are the benchmarks representative, with realistic data sizes and the data shapes that break
187+
optimizations, such as nulls, dictionaries, and wide strings?
188+
4. Does the Rust implementation allocate or copy more than it needs to, process row by row where a
189+
kernel exists, redo type conversions, or revalidate UTF-8 repeatedly?
190+
5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is
191+
acceptable or include a plan to fix it.
192+
193+
For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule
194+
and the requirement to record a performance audit. Hold the PR to both.
195+
196+
## 6. Documentation
197+
198+
### Generated, review the source instead
199+
200+
The compatibility pages under `docs/source/user-guide/latest/compatibility/expressions/` and
201+
`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on
202+
merge. Never ask the contributor to hand-edit them. Check what feeds them:
203+
204+
- Expression marked `Incompatible(Some("..."))` but `getIncompatibleReasons()` is empty, so the
205+
compat page shows it as supported with no caveats.
206+
- `Unsupported(Some("..."))` for specific types or argument shapes with no
207+
`getUnsupportedReasons()` to surface the limitation.
208+
- Reason strings that drift from the `notes` passed to `Compatible` / `Incompatible` /
209+
`Unsupported`. They need not match exactly, but consistency helps.
210+
- Reason strings too terse to be useful in user-facing docs: a single word, no context, no link to
211+
a tracking issue when behavior is known to differ.
212+
213+
### Hand-edited, the PR should update these
214+
215+
- `docs/source/user-guide/latest/expressions.md`, the supported-expressions list. A new expression
216+
belongs here.
217+
- Other `latest/compatibility/` pages such as `floating-point.md`, `regex.md`, `operators.md`.
218+
219+
### Does the PR make the contributor guide stale?
220+
221+
Check `adding_a_new_expression.md` against the diff. It goes stale when a PR:
222+
223+
- Changes the `CometExpressionSerde` interface, the support-level API, or the
224+
`getIncompatibleReasons` / `getUnsupportedReasons` contract
225+
- Adds or renames an expression map in `QueryPlanSerde.scala`, since the guide names them
226+
- Changes how the return type is resolved, which would invalidate the "when to set the return type
227+
explicitly" section
228+
- Changes the shimming pattern or adds a Spark version to the set the guide lists
229+
- Changes how scalar functions are registered in `comet_scalar_funcs.rs`
230+
231+
Check `sql-file-tests.md` when the PR adds, renames, or changes a test directive. The directive
232+
reference in that doc is the only place they are documented.
233+
234+
## Common Expression Review Findings
235+
236+
1. **Incomplete type support**, Spark supports types the PR does not handle
237+
2. **Missing edge cases**, null, overflow, empty string, negative values
238+
3. **Wrong return type**, it must match Spark exactly
239+
4. **Tests in the wrong framework**, Scala tests where a SQL file test would do
240+
5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible`
241+
6. **Version-specific Spark behavior implemented once**, with no shim
242+
7. **Name collides with a DataFusion built-in** and no explicit return type

0 commit comments

Comments
 (0)