You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 38f394e
Browse filesBrowse the repository at this point in the historyBrowse files
docs: prefer behavior-named parameters over Spark versions (#6776)
* docs: prefer behavior-named parameters over Spark versions
When behavior varies between Spark versions, resolve the version in Scala
and pass native code a parameter that describes the behavior. Add the
guidance to the contributor guide and the expression skills.
* docs: address review on behavior-named parameters
Reconcile the review skill's shim paragraph with the new rule: the
version check may live in a shim or a named predicate in Scala, and
protobuf and native code see only the behavior. Extend the rule to
native function names, point to RegrSparkVersions for fixes that land
in patch releases, ask comments to cite the Spark JIRA, state the fork
benefit as it is today, and replace the removed isSpark33Plus helper.
The two existing version-named names are tracked in #6891.
2. Register it in the matching map in `QueryPlanSerde.scala`.
78
78
3. If the function name collides with a DataFusion built-in that has a different signature, use `scalarFunctionExprToProtoWithReturnType` (see "When to set the return type explicitly").
79
79
4. For a new scalar function, add a match case in `native/spark-expr/src/comet_scalar_funcs.rs::create_comet_physical_fun`. If step 2 found an upstream implementation, wire that in. Otherwise implement the function under `native/spark-expr/src/`.
80
-
5. Add at least one Comet SQL Test at `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql` exercising column references, literals, and `NULL`.
80
+
5. If Spark's behavior differs between versions, resolve the version in the serde or a shim and pass native code a parameter named for the behavior (`wrap_second_millisecond_overflow`), not the version (`spark_420_plus`). See "Name the behavior, not the Spark version" in `adding_a_new_expression.md`.
81
+
6. Add at least one Comet SQL Test at `spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sql` exercising column references, literals, and `NULL`.
Copy file name to clipboardExpand all lines: .ai/skills/wire-datafusion-function/SKILL.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -102,6 +102,8 @@ Helpers from `QueryPlanSerde`:
102
102
-`optExprWithInfo(optExpr, expr, children*)` — wrap final result; propagates "why we couldn't convert" tags.
103
103
-`withInfo(expr, "reason")` — tag a fallback when returning `None`.
104
104
105
+
If the Spark behavior you are matching varies by Spark version, resolve the version in Scala and pass the native side a parameter, or pick a function name, that describes the behavior rather than the version (see "Name the behavior, not the Spark version" in `adding_a_new_expression.md`).
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/adding_a_new_expression.md
+15-1Lines changed: 15 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -582,7 +582,21 @@ Spark.
582
582
If the expression you're adding has different behavior across different Spark versions, you'll need to account for that in your implementation. There are two tools at your disposal to help with this:
583
583
584
584
1. Shims that exist in `spark/src/main/spark-$SPARK_VERSION/org/apache/comet/shims/CometExprShim.scala` for each Spark version. These shims are used to provide compatibility between different Spark versions.
585
-
2. Variables that correspond to the Spark version, such as `isSpark33Plus`, which can be used to conditionally execute code based on the Spark version.
585
+
2. Variables that correspond to the Spark version, such as `isSpark42Plus`, which can be used to conditionally execute code based on the Spark version.
586
+
587
+
#### Name the behavior, not the Spark version
588
+
589
+
When native code has to match a behavior that differs between Spark versions, decide which behavior applies in Scala and pass the result to native code under a name that describes the behavior, not the Spark version that introduced it. This covers protobuf fields, native function arguments, and native function names. For example, `TruncTimestamp` carries a `wrap_second_millisecond_overflow` flag, which the serde sets from `!isSpark42Plus`. It is not called `spark_420_plus`. The `filter_var_by_pair_nulls` field of `Regr` is another example.
590
+
591
+
Set the flag in one place on the Scala side, in a shim or in the serde, so that a fork overrides only that. A helper such as `isSpark42Plus` is enough when the behavior changed in a minor release. When Spark changed it in patch releases, compare the full version in a named predicate, because a check on the minor version is wrong for every earlier patch. `RegrSparkVersions.r2DegenerateCasesSwapped` in `serde/aggregates.scala` is the model. It resolves SPARK-55969, which shipped in 3.5.9, 4.0.3, 4.1.2, and 4.2.0.
592
+
593
+
Reasons to prefer behavior names:
594
+
595
+
- Some deployments run a Spark fork that backports fixes from newer open-source releases. Such a fork changes the one Scala check that sets the flag and leaves the protobuf and native code alone. A `spark_420_plus` flag cannot describe a fork that has one 4.2 fix but not the others.
596
+
- Spark occasionally changes behavior in a patch release or between minor releases, so a version number is a poor proxy for the behavior.
597
+
- The native code stays free of Spark version logic, and a reader of the Rust or proto definition can tell what the flag does without looking up a release.
598
+
599
+
Write comments the same way. Cite the Spark JIRA that changed the behavior, as the `TruncTimestamp` serde does with SPARK-56663, because forks track backports by JIRA rather than by release. Say "Spark 4.2 and later" for a behavior that future releases inherit, rather than "Spark 4.2". If a later release changes the behavior again, add a new parameter or shim for that change.
0 commit comments