|
| 1 | +--- |
| 2 | +name: review-comet-expression-pr |
| 3 | +description: Use when reviewing a DataFusion Comet pull request that adds or changes a Spark expression, its Scala serde, its protobuf message, or its native Rust implementation. Load alongside review-comet-pr, which covers the parts of the review that apply to every PR. |
| 4 | +argument-hint: <pr-number> |
| 5 | +--- |
| 6 | + |
| 7 | +<!-- |
| 8 | +Licensed to the Apache Software Foundation (ASF) under one |
| 9 | +or more contributor license agreements. See the NOTICE file |
| 10 | +distributed with this work for additional information |
| 11 | +regarding copyright ownership. The ASF licenses this file |
| 12 | +to you under the Apache License, Version 2.0 (the |
| 13 | +"License"); you may not use this file except in compliance |
| 14 | +with the License. You may obtain a copy of the License at |
| 15 | +
|
| 16 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 17 | +
|
| 18 | +Unless required by applicable law or agreed to in writing, |
| 19 | +software distributed under the License is distributed on an |
| 20 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 21 | +KIND, either express or implied. See the License for the |
| 22 | +specific language governing permissions and limitations |
| 23 | +under the License. |
| 24 | +--> |
| 25 | + |
| 26 | +Expression-specific review for Comet PR #$ARGUMENTS. |
| 27 | + |
| 28 | +**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review |
| 29 | +bar, and the output format. This skill only covers what is specific to expressions. |
| 30 | + |
| 31 | +## Read the Contributor Guide First |
| 32 | + |
| 33 | +| Doc | What you need from it | |
| 34 | +| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | |
| 35 | +| `docs/source/contributor-guide/adding_a_new_expression.md` | The serde contract, support levels, when to set the return type explicitly, shimming | |
| 36 | +| `docs/source/contributor-guide/sql-file-tests.md` | The test framework expression PRs are expected to use, and every directive it supports | |
| 37 | +| `docs/source/contributor-guide/optimizing_expressions.md` | The benchmark workflow and the no-regression rule, for PRs that change an existing kernel | |
| 38 | + |
| 39 | +Hold the diff against these. If the PR does something a guide says to do differently, either the PR |
| 40 | +is wrong or the guide is out of date, and you need to say which. |
| 41 | + |
| 42 | +## 1. Read the Spark Source |
| 43 | + |
| 44 | +**For any PR that adds or modifies an expression, you must read the Spark source to understand the |
| 45 | +canonical behavior.** This is the authoritative reference for what Comet must match. Do not review |
| 46 | +an expression PR from the diff alone. |
| 47 | + |
| 48 | +Clone Spark into `/tmp/spark` if it is not already there, using a shallow clone, then find the |
| 49 | +expression class under `sql/catalyst/src/main/scala` by searching for its `case class` declaration. |
| 50 | + |
| 51 | +Read the Spark implementation carefully. Pay attention to: |
| 52 | + |
| 53 | +- `eval` and `doGenCode` / `nullSafeEval`. These define the exact behavior. |
| 54 | +- `inputTypes` and `dataType`. These define which types Spark accepts and what it returns. |
| 55 | +- Null handling. Does it use `nullable = true`? Does `nullSafeEval` handle nulls implicitly? |
| 56 | +- Special cases, guards, and `require` assertions. |
| 57 | +- ANSI mode branches, usually `SQLConf.get.ansiEnabled` or `failOnError`. |
| 58 | +- Behavior differences across the Spark versions Comet supports. If the expression changed between |
| 59 | + 3.5 and 4.0, the serde needs a shim, not a single implementation. |
| 60 | + |
| 61 | +Then read the Spark tests for the expression, under the test source trees in `sql/`. |
| 62 | + |
| 63 | +Compare Spark's behavior against the PR and identify edge cases Spark tests that the PR does not, |
| 64 | +types Spark supports that the PR does not handle, and behavioral differences that should be marked |
| 65 | +`Incompatible`. |
| 66 | + |
| 67 | +## 2. Scala Serde |
| 68 | + |
| 69 | +Location: `spark/src/main/scala/org/apache/comet/serde/` |
| 70 | + |
| 71 | +- [ ] Expression class correctly identified |
| 72 | +- [ ] All child expressions converted via `exprToProtoInternal` |
| 73 | +- [ ] Return type correctly serialized |
| 74 | +- [ ] `getSupportLevel` reflects true compatibility rather than the happy path |
| 75 | +- [ ] Serde lives in the appropriate file (`datetime.scala`, `strings.scala`, `arithmetic.scala`, and so on) |
| 76 | +- [ ] ANSI and `fail_on_error` handling matches the constraints in `adding_a_new_expression.md` |
| 77 | + |
| 78 | +### Registration in `QueryPlanSerde.scala` |
| 79 | + |
| 80 | +- [ ] Added to the correct map (temporal, string, arithmetic, and so on) |
| 81 | +- [ ] No duplicate registrations |
| 82 | +- [ ] Import statement added |
| 83 | + |
| 84 | +### Return type collisions with DataFusion built-ins |
| 85 | + |
| 86 | +If the PR registers a Spark function whose name is also defined by `datafusion-functions`, such as |
| 87 | +`levenshtein`, `concat`, `coalesce`, `sha2`, or `regexp_replace`, check that the serde sets the |
| 88 | +return type explicitly via `scalarFunctionExprToProtoWithReturnType` rather than |
| 89 | +`scalarFunctionExprToProto` or the bare `CometScalarFunction(name)` shortcut. |
| 90 | + |
| 91 | +Without an explicit return type the native planner consults DataFusion's UDF registry first for |
| 92 | +type resolution, and any arity or input-type difference between the Spark and DataFusion versions |
| 93 | +fails native execution with `Error from DataFusion: Function 'X' expects N arguments but received |
| 94 | +M`. The Comet UDF is only swapped in after DataFusion's signature validation passes. See "When to |
| 95 | +set the return type explicitly" in `adding_a_new_expression.md`. |
| 96 | + |
| 97 | +## 3. Rust Implementation |
| 98 | + |
| 99 | +Location: `native/spark-expr/src/`, registered in `comet_scalar_funcs.rs`. |
| 100 | + |
| 101 | +- [ ] Matches DataFusion and Arrow conventions |
| 102 | +- [ ] Null handling is correct, including nulls inside nested types |
| 103 | +- [ ] No panics. Use `Result`. |
| 104 | +- [ ] Batch operations rather than row-by-row where a kernel exists |
| 105 | +- [ ] Invalid UTF-8 going into a native `StringType` goes through `decode_utf8_spark_lossy` |
| 106 | + |
| 107 | +Before accepting a hand-written kernel, ask whether the function already exists upstream in |
| 108 | +DataFusion or the `datafusion-spark` crate. Comet prefers wiring an upstream function over carrying |
| 109 | +its own copy. |
| 110 | + |
| 111 | +## 4. Tests |
| 112 | + |
| 113 | +**Expression tests should use Comet SQL Tests (`CometSqlFileTestSuite`) where possible.** The |
| 114 | +framework runs each query through both Spark and Comet and compares results, with no Scala code. |
| 115 | +Fall back to Comet Scala Tests in `CometExpressionSuite` only when the SQL framework cannot express |
| 116 | +the test, for example complex `DataFrame` setup or programmatic data generation. |
| 117 | + |
| 118 | +Test location: `spark/src/test/resources/sql-tests/expressions/<category>/`, with categories |
| 119 | +`aggregate/`, `array/`, `string/`, `math/`, `struct/`, `map/`, `datetime/`, `hash/`, and others. |
| 120 | + |
| 121 | +```sql |
| 122 | +statement |
| 123 | +CREATE TABLE test_crc32(col string, a int, b float) USING parquet |
| 124 | + |
| 125 | +statement |
| 126 | +INSERT INTO test_crc32 VALUES ('Spark', 10, 1.5), (NULL, NULL, NULL), ('', 0, 0.0) |
| 127 | + |
| 128 | +-- default mode: verifies native Comet execution and that the result matches Spark |
| 129 | +query |
| 130 | +SELECT crc32(col) FROM test_crc32 |
| 131 | + |
| 132 | +-- compares results without requiring native execution |
| 133 | +query spark_answer_only |
| 134 | +SELECT crc32(cast(a as string)) FROM test_crc32 |
| 135 | + |
| 136 | +-- allows numeric variance for floating-point results |
| 137 | +query tolerance=0.0001 |
| 138 | +SELECT cos(v) FROM test_trig |
| 139 | + |
| 140 | +-- asserts fallback to Spark occurs |
| 141 | +query expect_fallback(unsupported expression) |
| 142 | +SELECT unsupported_func(v) FROM test_table |
| 143 | + |
| 144 | +-- verifies both engines throw matching exceptions |
| 145 | +query expect_error(ARITHMETIC_OVERFLOW) |
| 146 | +SELECT 2147483647 + 1 |
| 147 | + |
| 148 | +-- skip a query with a known bug, with the issue link |
| 149 | +query ignore(https://github.com/apache/datafusion-comet/issues/NNNN) |
| 150 | +SELECT known_buggy_expr(v) FROM test_table |
| 151 | +``` |
| 152 | + |
| 153 | +Run the whole suite with `-Dsuites="org.apache.comet.CometSqlFileTestSuite" -Dtest=none`, or a |
| 154 | +single file by appending a substring of its name to the suite argument. |
| 155 | + |
| 156 | +**Verify all test requirements, whichever framework is used:** |
| 157 | + |
| 158 | +- [ ] Basic functionality tested against column data, not only literals |
| 159 | +- [ ] Null handling tested |
| 160 | +- [ ] Edge cases tested: empty input, overflow, boundary values, negative values |
| 161 | +- [ ] Both literal and column arguments tested, in every combination for multi-argument |
| 162 | + expressions. They take different code paths. |
| 163 | +- [ ] Timezone handling tested for timestamp and datetime expressions, including a non-UTC session |
| 164 | + timezone and timestamps with and without timezone |
| 165 | +- [ ] SQL syntax gated with `MinSparkVersion` when it only parses on newer Spark |
| 166 | +- [ ] `expect_error` patterns substring-match what both Spark and Comet actually throw |
| 167 | +- [ ] One expression per SQL file |
| 168 | +- [ ] Comet Scala literal tests disable constant folding: |
| 169 | + |
| 170 | +```scala |
| 171 | +withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> |
| 172 | + "org.apache.spark.sql.catalyst.optimizer.ConstantFolding") { |
| 173 | + checkSparkAnswerAndOperator("SELECT func(literal)") |
| 174 | +} |
| 175 | +``` |
| 176 | + |
| 177 | +## 5. Performance |
| 178 | + |
| 179 | +**For PRs that add a new native expression, performance is not optional.** The point of a native |
| 180 | +implementation is to be faster than Spark's codegen. If it is not faster, it may not be worth |
| 181 | +carrying. |
| 182 | + |
| 183 | +1. Does the PR description contain benchmark numbers comparing Comet against Spark? If not, ask for |
| 184 | + them. |
| 185 | +2. Does the PR add a benchmark under `spark/src/test/scala/org/apache/spark/sql/benchmark/`? |
| 186 | +3. Are the benchmarks representative, with realistic data sizes and the data shapes that break |
| 187 | + optimizations, such as nulls, dictionaries, and wide strings? |
| 188 | +4. Does the Rust implementation allocate or copy more than it needs to, process row by row where a |
| 189 | + kernel exists, redo type conversions, or revalidate UTF-8 repeatedly? |
| 190 | +5. If a benchmark shows Comet slower than Spark, flag it. The PR needs to explain why that is |
| 191 | + acceptable or include a plan to fix it. |
| 192 | + |
| 193 | +For PRs optimizing an existing kernel, `optimizing_expressions.md` defines the no-regression rule |
| 194 | +and the requirement to record a performance audit. Hold the PR to both. |
| 195 | + |
| 196 | +## 6. Documentation |
| 197 | + |
| 198 | +### Generated, review the source instead |
| 199 | + |
| 200 | +The compatibility pages under `docs/source/user-guide/latest/compatibility/expressions/` and |
| 201 | +`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on |
| 202 | +merge. Never ask the contributor to hand-edit them. Check what feeds them: |
| 203 | + |
| 204 | +- Expression marked `Incompatible(Some("..."))` but `getIncompatibleReasons()` is empty, so the |
| 205 | + compat page shows it as supported with no caveats. |
| 206 | +- `Unsupported(Some("..."))` for specific types or argument shapes with no |
| 207 | + `getUnsupportedReasons()` to surface the limitation. |
| 208 | +- Reason strings that drift from the `notes` passed to `Compatible` / `Incompatible` / |
| 209 | + `Unsupported`. They need not match exactly, but consistency helps. |
| 210 | +- Reason strings too terse to be useful in user-facing docs: a single word, no context, no link to |
| 211 | + a tracking issue when behavior is known to differ. |
| 212 | + |
| 213 | +### Hand-edited, the PR should update these |
| 214 | + |
| 215 | +- `docs/source/user-guide/latest/expressions.md`, the supported-expressions list. A new expression |
| 216 | + belongs here. |
| 217 | +- Other `latest/compatibility/` pages such as `floating-point.md`, `regex.md`, `operators.md`. |
| 218 | + |
| 219 | +### Does the PR make the contributor guide stale? |
| 220 | + |
| 221 | +Check `adding_a_new_expression.md` against the diff. It goes stale when a PR: |
| 222 | + |
| 223 | +- Changes the `CometExpressionSerde` interface, the support-level API, or the |
| 224 | + `getIncompatibleReasons` / `getUnsupportedReasons` contract |
| 225 | +- Adds or renames an expression map in `QueryPlanSerde.scala`, since the guide names them |
| 226 | +- Changes how the return type is resolved, which would invalidate the "when to set the return type |
| 227 | + explicitly" section |
| 228 | +- Changes the shimming pattern or adds a Spark version to the set the guide lists |
| 229 | +- Changes how scalar functions are registered in `comet_scalar_funcs.rs` |
| 230 | + |
| 231 | +Check `sql-file-tests.md` when the PR adds, renames, or changes a test directive. The directive |
| 232 | +reference in that doc is the only place they are documented. |
| 233 | + |
| 234 | +## Common Expression Review Findings |
| 235 | + |
| 236 | +1. **Incomplete type support**, Spark supports types the PR does not handle |
| 237 | +2. **Missing edge cases**, null, overflow, empty string, negative values |
| 238 | +3. **Wrong return type**, it must match Spark exactly |
| 239 | +4. **Tests in the wrong framework**, Scala tests where a SQL file test would do |
| 240 | +5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible` |
| 241 | +6. **Version-specific Spark behavior implemented once**, with no shim |
| 242 | +7. **Name collides with a DataFusion built-in** and no explicit return type |
0 commit comments