Skip to content

Commit 1dec4c0

Browse files
authored
fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171)
* fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), change two results that Comet reproduces natively: - SPARK-58821: when stop - start overflows a long but the length is small, sequence() returns the elements, where earlier releases raise INTERNAL_ERROR("Unreachable code reached."). A new native variant, spark_sequence_recomputed, does the same, and CometSequence.nativeFunction picks it on those releases. - SPARK-60118: casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent ("0e79") casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI). CometCast sets a new Cast proto field, decimal_zero_any_exponent, and parse_string_to_decimal skips the zero limit when it is set. SparkFixes holds the version checks. The internal-error case in sequence.sql moves to CometArrayExpressionSuite, because SQL file version directives compare only major.minor. * docs: describe SPARK-60119 and the version-number caveat, and share one version table Review follow-up. - SPARK-60119, which is in the same release candidates, makes Spark return 0.00 for a string far below the target scale's ulp, as the native cast already does. The cast audit entry and the string.rs comments now say that #6966's positive-scale case applies only before 4.0.5, 4.1.4 and 4.2.1, and that its negative-scale case still differs on them. - The cast compatibility page and CometSequence's compatibility notes say that Comet chooses by the Spark version number, and what a vendor build that reports an older version but includes SPARK-60118 or SPARK-58821 returns, as floating-point.md does for SPARK-59602. - recomputed_length_after_overflow keeps only the new element checks. length_limit_and_overflow_edges and the length grid already cover the internal error and both variants' limit errors. - SparkFixesSuite holds one version table for both fixes and CometSequence.nativeFunction, replacing the near-identical tables in CometArrayExpressionSuite and CometNativeCastSuite, which keep their convert checks. It is listed in both PR build workflows. * fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test Review follow-up. - On the released path, parse_string_to_decimal counts a zero's integral digits as 1 - scale with a wrapping i32 subtraction, as Spark's int arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers it, and two inputs whose counts don't wrap. - The sequence overflow test's released-Spark branch uses checkSparkError, which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather than the message. - The cast audit entry describes SPARK-60118's long digit count: released Spark throws for '1e2147483647' and similar strings, the fixed releases return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's ANSI error is #7188. - SparkFixes' scaladoc names the two fixes it covers and points to the other version checks.
1 parent c4a0b1f commit 1dec4c0

19 files changed

Lines changed: 457 additions & 92 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -676,6 +676,7 @@ jobs:
676676
org.apache.comet.serde.CometScalarFunctionSuite
677677
org.apache.comet.serde.CometLiteralSuite
678678
org.apache.comet.serde.QueryPlanSerdeSuite
679+
org.apache.comet.serde.SparkFixesSuite
679680
org.apache.comet.CometFallbackInvarianceSuite
680681
fail-fast: false
681682
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,6 +342,7 @@ jobs:
342342
org.apache.comet.serde.CometScalarFunctionSuite
343343
org.apache.comet.serde.CometLiteralSuite
344344
org.apache.comet.serde.QueryPlanSerdeSuite
345+
org.apache.comet.serde.SparkFixesSuite
345346
org.apache.comet.CometFallbackInvarianceSuite
346347
347348
fail-fast: false

‎docs/source/contributor-guide/expression-audits/array_funcs.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,7 @@
187187
- Spark 3.5.8 (audited 2026-08-29): internal refactors only (`DataTypeUtils.sameType`, `PhysicalIntegralType.integral`); runtime semantics identical to 3.4.3.
188188
- Spark 4.0.1 (audited 2026-08-29): boundary error becomes `SparkIllegalArgumentException(_LEGACY_ERROR_TEMP_3243)` and the length error becomes `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` (now carrying the function name); adds `throwable` optimizer hint. `sequenceLength` itself is unchanged.
189189
- Spark 4.1.1 (audited 2026-08-29): byte-identical `Sequence` class body to 4.0.1.
190+
- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates; in 4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3): `sequenceLength` returns the exact length when `stop - start` overflows a long but the length is within the limit, instead of raising `internalError("Unreachable code reached.")` (SPARK-58821). Where `SparkFixes.sequenceRecomputesOverflowedLength` holds, `CometSequence.nativeFunction` picks `spark_sequence_recomputed`, which returns that length; otherwise `spark_sequence`, which raises the internal error.
190191
- Comet routes integral element types (`ByteType`/`ShortType`/`IntegerType`/`LongType`) via `CometSequence` to the native `spark_sequence` kernel ([#5349](https://github.com/apache/datafusion-comet/issues/5349)): one pass over the generated elements, child buffer reserved once per batch, no per-row allocation. The two-argument form is evaluated with Spark's per-row default step inside the kernel. Both error conditions and the internal-error edge are reproduced through `SparkError` and mapped per Spark version by `ShimSparkErrorConverter`. Date/timestamp/timestamp_ntz sequences return `Unsupported` and run on the JVM codegen dispatcher (`CodegenDispatchFallback`), pending the timezone/DST/legacy-calendar work.
191192
- Per-batch capacity ceiling: the native kernel writes every row's generated elements into one Arrow child buffer whose offsets are `i32`, so the sum of every row's length in a single Arrow batch must fit in `i32::MAX`. Spark itself has no equivalent limit because it stores each row as its own `long[]`. If the total is exceeded, or if the allocator refuses the reservation, the query fails with a `SparkError::SequenceBatchTooLarge` message that names `spark.comet.batchSize` as the actionable knob (lower it to group fewer rows per batch). The `try_reserve` path guarantees the failure surfaces as a query error rather than an allocator abort.
192193
- Argument-shape restriction: `CometSequence` reports `Unsupported` for any `Sequence` whose `start`, `stop`, or `step` is not a leaf expression, and routes those through the JVM codegen dispatcher (`CodegenDispatchFallback`). DataFusion evaluates each scalar-UDF argument over the whole batch before calling the outer kernel, so a non-leaf argument would run on rows that Spark's per-row null short-circuit (or a `CASE` branch) would have discarded, and could raise where Spark would have returned `NULL`.

‎docs/source/contributor-guide/expression-audits/conversion_funcs.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
- Spark 3.5.8 (audited 2026-05-27): baseline. `Cast(child, dataType, timeZoneId, evalMode)`; eval modes are `LEGACY`, `ANSI`, `TRY`. The legacy `Cast.canCast` matrix and the `Cast.canAnsiCast` matrix decide acceptance per type pair. Comet routes via `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala`) using a per-source-type support matrix that returns `Compatible`, `Incompatible(reason)`, or `Unsupported(reason)`; literal children are short-circuited to `Compatible()` so `CometLiteral` validates them. The serialized `Cast` proto carries `datatype`, `evalMode`, `timezone` (default `UTC`), and `isSpark4Plus`; `allowIncompatible` is enforced only by the JVM planner and is not sent to the native cast kernel. The native side (`native/spark-expr/src/conversion_funcs/cast.rs`) implements explicit per-eval-mode branches for narrowing numeric casts that match Spark's overflow exceptions, and falls through to DataFusion `cast_with_options(safe = !ANSI)` for the rest.
2828
- Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
2929
- Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
30+
- Spark 4.0.5, 4.1.4 and 4.2.1 (release candidates; in 4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3): `Decimal.fromString` and `fromStringANSI` no longer count integral digits for zero (SPARK-60118), so a zero with a large exponent such as `'0e39'` casts to 0 instead of NULL (or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI). `CometCast` sets the `decimal_zero_any_exponent` cast option from `SparkFixes.castsZeroWithAnyExponentToDecimal`, and `parse_string_to_decimal` skips the zero limit when it is set. The same fix counts the integral digits in a long. Released Spark counts them in an int, which wraps for an exponent near the int limit, so `'1e2147483647'`, `'12e2147483647'` and `'1.5e2147483647'` skip the limit and throw a raw `ArithmeticException` in every eval mode, and `'0e2147483647'` casts to 0. On the fixed releases the first three return NULL, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI. The native cast returns NULL for them outside ANSI, as the fixed releases do, and raises `CAST_INVALID_INPUT` under ANSI ([#7188](https://github.com/apache/datafusion-comet/issues/7188)). Without `decimal_zero_any_exponent` it lets `'0e2147483647'` through as released Spark does. The same releases skip `BigDecimal.setScale` in `Decimal.changePrecision` when the value's magnitude already decides the result (SPARK-60119), so a string far below the target scale's ulp casts to the rounded value instead of throwing `ArithmeticException`. `'1e-2147483647'` to `DECIMAL(10,2)` gives `0.00`, which the native cast already returns, so that case of [#6966](https://github.com/apache/datafusion-comet/issues/6966) applies only to earlier releases. Its negative-scale case (`spark.sql.legacy.allowNegativeScaleOfDecimal=true`) still differs: `'1e-2147483647'` to `DECIMAL(10,-2)` gives 0 on these releases, while the native cast raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise.
3031
- Known divergences and gaps:
3132
- `CAST(<binary> AS STRING)` decodes the bytes with `decode_utf8_spark_lossy`, replacing invalid UTF-8 with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does (the same decoder as the native shuffle). It matches Spark's rendered output, but decoding is not byte-preserving, so it diverges from Spark for operations that read the underlying bytes: round-trips such as `CAST(CAST(x AS STRING) AS BINARY)`, and value identity, since distinct ill-formed byte sequences all decode to `U+FFFD` and can therefore compare equal in Comet where Spark compares the raw bytes ([#4764](https://github.com/apache/datafusion-comet/issues/4764)).
3233
- Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test ([#4489](https://github.com/apache/datafusion-comet/issues/4489); umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)).

‎docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,12 @@ including:
7575
- Scientific notation (e.g. `1.23E+5`) is supported.
7676
- Special values (`inf`, `infinity`, `nan`) produce `NULL`.
7777

78+
A zero with a large exponent, such as `'0e39'`, casts to `0` on Spark 4.0.5, 4.1.4, 4.2.1 and later
79+
(SPARK-60118). Earlier releases return `NULL` for it, or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE`
80+
under ANSI mode. Comet chooses the behavior by the Spark version number. On a vendor build that
81+
reports an older version but includes SPARK-60118, Comet returns `NULL` (or raises) where Spark
82+
returns `0`.
83+
7884
## String to Date
7985

8086
Comet's native `CAST(string AS DATE)` implementation matches Apache Spark's behavior for years

‎native/core/src/execution/planner.rs‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -776,14 +776,16 @@ impl PhysicalPlanner {
776776
registry.get(expr_id)
777777
});
778778

779+
let mut cast_options = SparkCastOptions::new_with_version(
780+
eval_mode,
781+
&expr.timezone,
782+
expr.is_spark4_plus,
783+
);
784+
cast_options.decimal_zero_any_exponent = expr.decimal_zero_any_exponent;
779785
Ok(Arc::new(Cast::new(
780786
child,
781787
datatype,
782-
SparkCastOptions::new_with_version(
783-
eval_mode,
784-
&expr.timezone,
785-
expr.is_spark4_plus,
786-
),
788+
cast_options,
787789
spark_expr.expr_id,
788790
query_context,
789791
)))

‎native/proto/src/proto/expr.proto‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -462,6 +462,9 @@ message Cast {
462462
// True when running against Spark 4.0+. Controls version-specific cast behaviour
463463
// such as the handling of leading whitespace before T-prefixed time-only strings.
464464
bool is_spark4_plus = 6;
465+
// True when the running Spark casts a zero string with any exponent ("0e39") to decimal as
466+
// 0, instead of applying the integral-digit limit to it (SPARK-60118).
467+
bool decimal_zero_any_exponent = 7;
465468
}
466469

467470
// Spark's internal PreciseTimestampConversion, used by time-window grouping to convert

‎native/spark-expr/src/array_funcs/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ pub use get_array_struct_fields::GetArrayStructFields;
4747
pub use list_extract::ListExtract;
4848
pub use list_positions::ListPositionsExpr;
4949
pub use nested_comparison::{spark_comparison, spark_in_list, FloatOperands, SparkComparison};
50-
pub use sequence::spark_sequence;
50+
pub use sequence::{spark_sequence, spark_sequence_recomputed};
5151
pub use size::{spark_size, SparkSizeFunc};
5252
pub use sort_array::{SparkArraySort, SparkSortArray};
5353

0 commit comments

Comments
 (0)