You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1e2ce22
Browse filesBrowse the repository at this point in the historyBrowse files
docs: derive Spark versions from pom.xml, move audit lines to a separate PR
Review feedback from @comphead:
- Stop hardcoding the Spark version list. The versions in the skill (3.4.3,
3.5.8, 4.0.1, 4.1.1, inherited from audit-comet-expression) were already
stale against pom.xml, which pins 3.4.3, 3.5.9, 4.0.4, 4.1.3 and has a
spark-4.2 profile. Derive the list from pom.xml at run time instead, so it
tracks support being added and dropped.
- Move the two audit-log lines to their own PR so the skill can be reviewed
separately from the assessments it produced.
Diff the class body between adjacent versions rather than eyeballing it. Note behavior that differs:
228
+
each divergence is a shim the native path would need, and "identical across all supported versions"
229
+
is itself a finding worth stating, because it means no shim.
216
230
217
231
### Check upstream first
218
232
@@ -248,7 +262,7 @@ run rather than skimmed.
248
262
| Lambda bodies or arbitrary child expression trees (`transform`, `aggregate`, `zip_with`) | The native path would need to evaluate an arbitrary Catalyst tree, not a fixed kernel |
249
263
| Dependence on JVM session state beyond a scalar config (SQLConf lookups, user classes, reflection) | State that cannot be serialized into the plan cannot cross into native code |
250
264
| Nondeterminism or ordering-sensitivity (hash iteration order, unstable sorts) | Output order must match Spark's, which is often an implementation detail rather than a spec |
251
-
| Behavior that differs across 3.4.3 / 3.5.8 / 4.0.1 / 4.1.1 | Each divergence is a shim; several divergences means the native path carries version branches forever |
265
+
| Behavior that differs across the Spark versions pinned in `pom.xml`| Each divergence is a shim; several divergences means the native path carries version branches forever |
- Native candidate (assessed 2026-08-13): Recommended for integral element types only ([#5349](https://github.com/apache/datafusion-comet/issues/5349)). Compatibility: Medium (behaviour stable across 3.4.3, 3.5.8, 4.0.1, and 4.1.1, and the integral path is enumerable integer arithmetic with two error conditions, but `TemporalSequenceImpl` / `PeriodSequenceImpl` / `DurationSequenceImpl` carry a `zoneId` and step through `DateTimeUtils.timestampAddInterval`, so DST, month-length arithmetic, and the legacy calendar flags apply; the guard is `start.dataType`, knowable at plan time, with date and timestamp sequences staying on `CodegenDispatchFallback`). Upside: High, measured from the emitted kernel via `CometBatchKernelCodegen.generateSource` (3 heap allocations per non-null row: `new long[n]`, plus the internal `new long[...]` and `new UnsafeArrayData()` in `UnsafeArrayData.fromPrimitiveArray`, two of them scaling with sequence length, plus three passes over every element from the fill loop, the `copyMemory`, and the output-side `getLong`/`setSafe` loop; roughly 48 MB of transient `long[]` per 8192-row batch for a 365-element date spine). Janino compiles the kernel, so there is no hidden whole-operator Spark fallback. Upstream is unusable: no `sequence` in `datafusion-spark`, and `datafusion-functions-nested::range` has different messages and a scalar-argument shape.
181
-
182
178
## shuffle
183
179
184
180
- Spark 3.4.3 (audited 2026-07-02): `Shuffle(child, randomSeed: Option[Long])`; `inputTypes = Seq(ArrayType)`, `dataType = child.dataType`, non-deterministic and stateful. Seeds a Commons Math3 `MersenneTwister` with `randomSeed + partitionIndex` and applies the "inside-out" Fisher-Yates from `RandomIndicesGenerator`. Only the one-argument `shuffle(array)` form exists in SQL. NULL input returns NULL without advancing the RNG.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/string_funcs.md
-4Lines changed: 0 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -245,10 +245,6 @@
245
245
- Spark 3.5.8 (audited 2026-05-27): registry alias of `Upper`. Same support as `upper`.
246
246
- Spark 4.0.1 (audited 2026-05-27): unchanged alias of `Upper`.
247
247
248
-
## unbase64
249
-
250
-
- Native candidate (assessed 2026-08-13): Recommended ([#5347](https://github.com/apache/datafusion-comet/issues/5347)). Compatibility: High (semantics identical across 3.4.3, 3.5.8, 4.0.1, and 4.1.1; no collation, timezone, locale, or decimal dependence; the `java.util.Base64.getMimeDecoder()` skip rule and its four `IllegalArgumentException` conditions are fully enumerable from the JDK source, and Comet already ports the mirror-image MIME encoder in `spark_base64`). Upside: High (5 JVM heap allocations per non-null row, roughly 41k per 8192-row batch, plus a double scan from the MIME `decodedOutLength` pre-scan, all removable with the `spark_unhex` pattern of a 256-entry table plus a preallocated `BinaryBuilder`; counts derived from source, not measured, and workload presence is unmeasured). Upstream `datafusion-spark::SparkUnBase64` is not usable: it lowers to `decode(bin, 'base64pad')`, whose strict engine errors on the non-alphabet bytes Spark skips, so it cannot round-trip Comet's own CRLF-chunked `base64` output.
251
-
252
248
## upper
253
249
254
250
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
0 commit comments