Skip to content

Commit 1e2ce22

Browse files
committed
docs: derive Spark versions from pom.xml, move audit lines to a separate PR
Review feedback from @comphead: - Stop hardcoding the Spark version list. The versions in the skill (3.4.3, 3.5.8, 4.0.1, 4.1.1, inherited from audit-comet-expression) were already stale against pom.xml, which pins 3.4.3, 3.5.9, 4.0.4, 4.1.3 and has a spark-4.2 profile. Derive the list from pom.xml at run time instead, so it tracks support being added and dropped. - Move the two audit-log lines to their own PR so the skill can be reviewed separately from the assessments it produced.
1 parent 399c36b commit 1e2ce22

3 files changed

Lines changed: 21 additions & 15 deletions

File tree

‎.claude/skills/suggest-native-expression/SKILL.md‎

Lines changed: 21 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,8 @@ on them.
5050
section the filed issue should reference.
5151
- `docs/source/user-guide/latest/compatibility/index.md`, section "Native and codegen-dispatch
5252
implementations". Defines the two paths and which one is the default.
53-
- `docs/source/user-guide/latest/expressions.md`, the Implementation legend and tables. The
54-
`Codegen dispatch` rows are the candidate pool.
53+
- `docs/source/user-guide/latest/expressions.md`, the Implementation legend and tables. Useful for a
54+
candidate's SQL name, category, and status, but see Step 1: this table is not the pool.
5555
- `docs/source/contributor-guide/optimizing_expressions.md`, the "Proven techniques" table. If
5656
no technique in that table applies to the candidate, the native upside claim is weak.
5757
- `spark/src/main/scala/org/apache/comet/serde/CometExpressionSerde.scala` for the
@@ -209,10 +209,24 @@ dispatcher it would replace is exact.
209209

210210
### Read the Spark implementation across versions
211211

212-
Reuse the clone recipe from `audit-comet-expression` (Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1) and read
213-
`eval` / `nullSafeEval` / `doGenCode`, `inputTypes`, `dataType`, ANSI branches, and any
214-
config the expression reads. Note behavior that differs across versions: each divergence is a
215-
shim the native path would need.
212+
Derive the version list from the build rather than hardcoding it, so the assessment does not go stale
213+
as Comet drops and adds Spark support:
214+
215+
```bash
216+
grep -oE "<spark.version>[0-9.]+</spark.version>" pom.xml | sed -E 's/<[^>]+>//g' | sort -u
217+
```
218+
219+
Read `eval` / `nullSafeEval` / `doGenCode`, `inputTypes`, `dataType`, ANSI branches, and any config
220+
the expression reads, for each of those versions. The class body is usually short enough to fetch
221+
directly instead of cloning:
222+
223+
```bash
224+
curl -sSfL "https://raw.githubusercontent.com/apache/spark/v<version>/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/<file>.scala"
225+
```
226+
227+
Diff the class body between adjacent versions rather than eyeballing it. Note behavior that differs:
228+
each divergence is a shim the native path would need, and "identical across all supported versions"
229+
is itself a finding worth stating, because it means no shim.
216230

217231
### Check upstream first
218232

@@ -248,7 +262,7 @@ run rather than skimmed.
248262
| Lambda bodies or arbitrary child expression trees (`transform`, `aggregate`, `zip_with`) | The native path would need to evaluate an arbitrary Catalyst tree, not a fixed kernel |
249263
| Dependence on JVM session state beyond a scalar config (SQLConf lookups, user classes, reflection) | State that cannot be serialized into the plan cannot cross into native code |
250264
| Nondeterminism or ordering-sensitivity (hash iteration order, unstable sorts) | Output order must match Spark's, which is often an implementation detail rather than a spec |
251-
| Behavior that differs across 3.4.3 / 3.5.8 / 4.0.1 / 4.1.1 | Each divergence is a shim; several divergences means the native path carries version branches forever |
265+
| Behavior that differs across the Spark versions pinned in `pom.xml` | Each divergence is a shim; several divergences means the native path carries version branches forever |
252266

253267
### Rate it
254268

‎docs/source/contributor-guide/expression-audits/array_funcs.md‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -175,10 +175,6 @@
175175
- Spark 4.0.1 (audited 2026-05-27): semantics unchanged; ANSI default flips to `true`.
176176
- Spark 4.1.1 (audited 2026-05-27): `inputTypes` tightened to `Seq(ArrayType, IntegralType)` (analysis-time only); runtime unchanged.
177177

178-
## sequence
179-
180-
- Native candidate (assessed 2026-08-13): Recommended for integral element types only ([#5349](https://github.com/apache/datafusion-comet/issues/5349)). Compatibility: Medium (behaviour stable across 3.4.3, 3.5.8, 4.0.1, and 4.1.1, and the integral path is enumerable integer arithmetic with two error conditions, but `TemporalSequenceImpl` / `PeriodSequenceImpl` / `DurationSequenceImpl` carry a `zoneId` and step through `DateTimeUtils.timestampAddInterval`, so DST, month-length arithmetic, and the legacy calendar flags apply; the guard is `start.dataType`, knowable at plan time, with date and timestamp sequences staying on `CodegenDispatchFallback`). Upside: High, measured from the emitted kernel via `CometBatchKernelCodegen.generateSource` (3 heap allocations per non-null row: `new long[n]`, plus the internal `new long[...]` and `new UnsafeArrayData()` in `UnsafeArrayData.fromPrimitiveArray`, two of them scaling with sequence length, plus three passes over every element from the fill loop, the `copyMemory`, and the output-side `getLong`/`setSafe` loop; roughly 48 MB of transient `long[]` per 8192-row batch for a 365-element date spine). Janino compiles the kernel, so there is no hidden whole-operator Spark fallback. Upstream is unusable: no `sequence` in `datafusion-spark`, and `datafusion-functions-nested::range` has different messages and a scalar-argument shape.
181-
182178
## shuffle
183179

184180
- Spark 3.4.3 (audited 2026-07-02): `Shuffle(child, randomSeed: Option[Long])`; `inputTypes = Seq(ArrayType)`, `dataType = child.dataType`, non-deterministic and stateful. Seeds a Commons Math3 `MersenneTwister` with `randomSeed + partitionIndex` and applies the "inside-out" Fisher-Yates from `RandomIndicesGenerator`. Only the one-argument `shuffle(array)` form exists in SQL. NULL input returns NULL without advancing the RNG.

‎docs/source/contributor-guide/expression-audits/string_funcs.md‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -245,10 +245,6 @@
245245
- Spark 3.5.8 (audited 2026-05-27): registry alias of `Upper`. Same support as `upper`.
246246
- Spark 4.0.1 (audited 2026-05-27): unchanged alias of `Upper`.
247247

248-
## unbase64
249-
250-
- Native candidate (assessed 2026-08-13): Recommended ([#5347](https://github.com/apache/datafusion-comet/issues/5347)). Compatibility: High (semantics identical across 3.4.3, 3.5.8, 4.0.1, and 4.1.1; no collation, timezone, locale, or decimal dependence; the `java.util.Base64.getMimeDecoder()` skip rule and its four `IllegalArgumentException` conditions are fully enumerable from the JDK source, and Comet already ports the mirror-image MIME encoder in `spark_base64`). Upside: High (5 JVM heap allocations per non-null row, roughly 41k per 8192-row batch, plus a double scan from the MIME `decodedOutLength` pre-scan, all removable with the `spark_unhex` pattern of a 256-entry table plus a preallocated `BinaryBuilder`; counts derived from source, not measured, and workload presence is unmeasured). Upstream `datafusion-spark::SparkUnBase64` is not usable: it lowers to `decode(bin, 'base64pad')`, whose strict engine errors on the non-alphabet bytes Spark skips, so it cannot round-trip Comet's own CRLF-chunked `base64` output.
251-
252248
## upper
253249

254250
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.

0 commit comments

Comments
 (0)