You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ae98cef
Browse filesBrowse the repository at this point in the historyBrowse files
Preserve the upstream RSS callback and protocol tests alongside the
wide-decimal native shuffle routing expectation. Combine the conflicting
DecimalType and StructType imports without changing either implementation.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/string_funcs.md
+8Lines changed: 8 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -245,6 +245,14 @@
245
245
- Spark 3.5.8 (audited 2026-05-27): registry alias of `Upper`. Same support as `upper`.
246
246
- Spark 4.0.1 (audited 2026-05-27): unchanged alias of `Upper`.
247
247
248
+
## unbase64
249
+
250
+
- Spark 3.4.3 (audited 2026-08-24): identical to 3.5.8.
251
+
- Spark 3.5.8 (audited 2026-08-24): baseline. `doGenCode` emits `java.util.Base64.getMimeDecoder().decode(child.toString())`. The MIME decoder skips every byte outside the base64 alphabet, so CRLF-wrapped output from Spark's own `base64` round-trips cleanly; four terminal-shape errors surface as `IllegalArgumentException`. `failOnError = true` is set only when the node is constructed from `to_binary('base64')` / `try_to_binary`, which use a stricter RFC 4648 validator.
252
+
- Spark 4.0.1 (audited 2026-08-24): `NullIntolerant` becomes `override def nullIntolerant: Boolean = true`; `inputTypes` widens to `StringTypeWithCollation(supportsTrimCollation = true)`. Behaviour is byte-level and collation-independent, so no divergence for `UTF8_BINARY` and no shim is needed.
253
+
- Spark 4.1.1 (audited 2026-08-24): adds `contextIndependentFoldable`; no behavioural change on the decode path.
254
+
- Comet native implementation (`spark_unbase64`, `native/spark-expr/src/string_funcs/unbase64.rs`) ports the JDK MIME decoder rules: 256-entry decode LUT, a reused per-batch scratch `Vec<u8>` copied into a preallocated `BinaryBuilder`, all four error messages reproduced verbatim. `CometUnBase64` handles `failOnError = false` natively; `failOnError = true` (reachable from `to_binary('base64')` / `try_to_binary`) requires strict RFC 4648 validation and is not yet implemented natively, so those cases stay on the JVM codegen dispatcher via `CodegenDispatchFallback`.
255
+
248
256
## upper
249
257
250
258
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
0 commit comments