You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 26994c9
Browse filesBrowse the repository at this point in the historyBrowse files
Replace the hand-rolled getSupportLevel/convert branching with the
`CodegenDispatchFallback` mixin, which is what the framework already provides
for exactly this shape. `exprToProtoInternal` routes an `Unsupported` result to
the dispatcher plus a logDebug, a non-opted-in `Incompatible` to the dispatcher
plus the [COMET-INFO] opt-in hint, and an opted-in `Incompatible` to the native
proto with a logWarning.
For `CometStringTranslate` that makes the whole diff one mixin on the original
declaration: the pre-existing `Incompatible(Some(incompatReason))` was already
the right classification. This also restores the `CometScalarFunction` base, so
the function name "translate" stops being duplicated and the opted-in logWarning
comes back.
For `CometStructsToCsv` the original three-way `getSupportLevel` was likewise
already correct -- `Unsupported` for complex field types, `Incompatible` for the
#3232 types, `Incompatible()` otherwise -- so the +30/-30 rewrite collapses to
the mixin. That restores `getUnsupportedReasons`, drops the duplicated
`nativeSupported` predicate that had to stay in sync between two call sites, and
picks up the logDebug on the opted-in-but-unsupported path for free. Also drops
`CometCodegenDispatch` as the base, which is for expressions with no native
path at all.
Tests:
- string_translate.sql gains the two rows that actually distinguish the paths:
"e" + U+0301 (one grapheme, two code points) and U+0000 as the `to` argument.
Built with decode(X'65CC81') / decode(X'00') to keep the fixture ASCII.
Verified these discriminate: adding them to string_translate_enabled.sql,
which opts into the native path, fails.
- to_csv.sql gains options coverage (`sep`, and `timestampFormat`/`dateFormat`
over date and timestamp fields), which the dispatcher path had nowhere.
- CometCsvExpressionSuite "to_csv - default options" drops the
allowIncompatible=true wrapper and covers c6 Double, c10 Timestamp and c11
TimestampNTZ, which it previously had to skip. It now tests what its name
says.
Two field types are deliberately left unasserted, both Spark behavior rather
than Comet's, confirmed against Spark 3.5 with Comet disabled:
- Binary: Spark's CSV converter renders it with Java's default toString, e.g.
"[B@731af74", an identity hash that differs between any two evaluations, so
it is not comparable by any engine including Spark against itself.
- Complex types: a non-null array/map/struct renders as
"org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" and any null complex
value throws NullPointerException inside Spark's UnsafeWriter.write. Before
this change these schemas fell back to Spark; now they reach the dispatcher,
which runs the same Spark code, so the observable result is unchanged.
0 commit comments