Skip to content

Commit 38f394e

Browse files
authored
docs: prefer behavior-named parameters over Spark versions (#6776)
* docs: prefer behavior-named parameters over Spark versions When behavior varies between Spark versions, resolve the version in Scala and pass native code a parameter that describes the behavior. Add the guidance to the contributor guide and the expression skills. * docs: address review on behavior-named parameters Reconcile the review skill's shim paragraph with the new rule: the version check may live in a shim or a named predicate in Scala, and protobuf and native code see only the behavior. Extend the rule to native function names, point to RegrSparkVersions for fixes that land in patch releases, ask comments to cite the Spark JIRA, state the fork benefit as it is today, and replace the removed isSpark33Plus helper. The two existing version-named names are tracked in #6891.
1 parent 695ca3a commit 38f394e

5 files changed

Lines changed: 35 additions & 7 deletions

File tree

‎.ai/skills/implement-comet-expression/SKILL.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,8 @@ Follow `adding_a_new_expression.md`:
7777
2. Register it in the matching map in `QueryPlanSerde.scala`.
7878
3. If the function name collides with a DataFusion built-in that has a different signature, use `scalarFunctionExprToProtoWithReturnType` (see "When to set the return type explicitly").
7979
4. For a new scalar function, add a match case in `native/spark-expr/src/comet_scalar_funcs.rs::create_comet_physical_fun`. If step 2 found an upstream implementation, wire that in. Otherwise implement the function under `native/spark-expr/src/`.
80-
5. Add at least one Comet SQL Test at `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql` exercising column references, literals, and `NULL`.
80+
5. If Spark's behavior differs between versions, resolve the version in the serde or a shim and pass native code a parameter named for the behavior (`wrap_second_millisecond_overflow`), not the version (`spark_420_plus`). See "Name the behavior, not the Spark version" in `adding_a_new_expression.md`.
81+
6. Add at least one Comet SQL Test at `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql` exercising column references, literals, and `NULL`.
8182

8283
Build and smoke-test:
8384

‎.ai/skills/review-comet-expression-pr/SKILL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,9 @@ reference in that doc is the only place they are documented.
266266
3. **Wrong return type**, it must match Spark exactly
267267
4. **Tests in the wrong framework**, Scala tests where a SQL file test would do
268268
5. **Missing `getSupportLevel`**, divergences left undeclared rather than marked `Incompatible`
269-
6. **Version-specific Spark behavior implemented once**, with no shim
269+
6. **Version-specific Spark behavior implemented once**, with no shim or version check, or with a
270+
protobuf field, native argument, or native function named for the Spark version
271+
(`spark_420_plus`) instead of the behavior (`wrap_second_millisecond_overflow`)
270272
7. **Name collides with a DataFusion built-in** and no explicit return type
271273
8. **Timestamp result mislabelled**, with the session timezone or no timezone instead of `"UTC"`.
272274
It only shows once the result is compared or fed to another expression.

‎.ai/skills/review-comet-pr/SKILL.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -171,10 +171,19 @@ Report the comparison in the review even when nothing changed, so the reviewer k
171171

172172
### Spark version coverage
173173

174-
Comet supports several Spark versions. Version-specific behavior belongs in the shims under
175-
`spark/src/main/spark-{3.4,3.5,3.x,4.0,4.1,4.2}/org/apache/comet/shims/`, not in branches on a
176-
version string in shared code, and not in native Rust. If the PR adds a shim for one 4.x version,
177-
check that the sibling 4.x source sets got it too.
174+
Comet supports several Spark versions. The check on the Spark version belongs in Scala, either in a
175+
shim under `spark/src/main/spark-*/` or in one named predicate such as `isSpark42Plus` or
176+
`RegrSparkVersions.slopeFiltersVarByPairNulls`. It does not belong in an ad hoc comparison on the
177+
version string in shared code, or in native Rust. If the PR adds a shim for one 4.x version, check
178+
that the sibling 4.x source sets got it too.
179+
180+
Protobuf fields, native function arguments, and native function names only ever see the behavior,
181+
such as `wrap_second_millisecond_overflow`, never the version, such as `spark_420_plus` (#6740). A
182+
fork that backports a Spark fix then changes the one Scala check that sets the flag and leaves the
183+
protobuf and Rust alone. Flag a version-named field, argument, or function, and comments that say
184+
"Spark 4.2" for behavior later releases inherit or that leave out the Spark JIRA.
185+
`Cast.is_spark4_plus` and `get_json_object_spark34` predate the rule and #6891 tracks renaming them,
186+
so flag them only when a PR adds behavior behind them.
178187

179188
When Spark changed the behavior in a patch release, such as SPARK-55969 or SPARK-54918, a check on
180189
the minor version is wrong for every earlier patch. CI builds only the newest patch of each line, so

‎.ai/skills/wire-datafusion-function/SKILL.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,8 @@ Helpers from `QueryPlanSerde`:
102102
- `optExprWithInfo(optExpr, expr, children*)` — wrap final result; propagates "why we couldn't convert" tags.
103103
- `withInfo(expr, "reason")` — tag a fallback when returning `None`.
104104

105+
If the Spark behavior you are matching varies by Spark version, resolve the version in Scala and pass the native side a parameter, or pick a function name, that describes the behavior rather than the version (see "Name the behavior, not the Spark version" in `adding_a_new_expression.md`).
106+
105107
`getSupportLevel` returning `Incompatible(Some("…"))` gates behind `spark.comet.expr.<name>.allowIncompatible=true`. `Unsupported(…)` always falls back.
106108

107109
### 4. Register the UDF (Pattern B only)

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -582,7 +582,21 @@ Spark.
582582
If the expression you're adding has different behavior across different Spark versions, you'll need to account for that in your implementation. There are two tools at your disposal to help with this:
583583

584584
1. Shims that exist in `spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala` for each Spark version. These shims are used to provide compatibility between different Spark versions.
585-
2. Variables that correspond to the Spark version, such as `isSpark33Plus`, which can be used to conditionally execute code based on the Spark version.
585+
2. Variables that correspond to the Spark version, such as `isSpark42Plus`, which can be used to conditionally execute code based on the Spark version.
586+
587+
#### Name the behavior, not the Spark version
588+
589+
When native code has to match a behavior that differs between Spark versions, decide which behavior applies in Scala and pass the result to native code under a name that describes the behavior, not the Spark version that introduced it. This covers protobuf fields, native function arguments, and native function names. For example, `TruncTimestamp` carries a `wrap_second_millisecond_overflow` flag, which the serde sets from `!isSpark42Plus`. It is not called `spark_420_plus`. The `filter_var_by_pair_nulls` field of `Regr` is another example.
590+
591+
Set the flag in one place on the Scala side, in a shim or in the serde, so that a fork overrides only that. A helper such as `isSpark42Plus` is enough when the behavior changed in a minor release. When Spark changed it in patch releases, compare the full version in a named predicate, because a check on the minor version is wrong for every earlier patch. `RegrSparkVersions.r2DegenerateCasesSwapped` in `serde/aggregates.scala` is the model. It resolves SPARK-55969, which shipped in 3.5.9, 4.0.3, 4.1.2, and 4.2.0.
592+
593+
Reasons to prefer behavior names:
594+
595+
- Some deployments run a Spark fork that backports fixes from newer open-source releases. Such a fork changes the one Scala check that sets the flag and leaves the protobuf and native code alone. A `spark_420_plus` flag cannot describe a fork that has one 4.2 fix but not the others.
596+
- Spark occasionally changes behavior in a patch release or between minor releases, so a version number is a poor proxy for the behavior.
597+
- The native code stays free of Spark version logic, and a reader of the Rust or proto definition can tell what the flag does without looking up a release.
598+
599+
Write comments the same way. Cite the Spark JIRA that changed the behavior, as the `TruncTimestamp` serde does with SPARK-56663, because forks track backports by JIRA rather than by release. Say "Spark 4.2 and later" for a behavior that future releases inherit, rather than "Spark 4.2". If a later release changes the behavior again, add a new parameter or shim for that change.
586600

587601
## Shimming to Support Different Spark Versions
588602

0 commit comments

Comments
 (0)