Repository navigation
Commit 1dec4c0
authored
fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast (#7171)
* fix: match SPARK-58821 and SPARK-60118 in sequence and string to decimal cast
Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3,
4.1.4-rc4 and 4.2.1-rc3), change two results that Comet reproduces
natively:
- SPARK-58821: when stop - start overflows a long but the length is small,
sequence() returns the elements, where earlier releases raise
INTERNAL_ERROR("Unreachable code reached."). A new native variant,
spark_sequence_recomputed, does the same, and
CometSequence.nativeFunction picks it on those releases.
- SPARK-60118: casting a string to decimal no longer applies the
integral-digit limit to zero, so a zero with a large exponent ("0e79")
casts to 0 instead of NULL (or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI).
CometCast sets a new Cast proto field, decimal_zero_any_exponent, and
parse_string_to_decimal skips the zero limit when it is set.
SparkFixes holds the version checks. The internal-error case in
sequence.sql moves to CometArrayExpressionSuite, because SQL file version
directives compare only major.minor.
* docs: describe SPARK-60119 and the version-number caveat, and share one version table
Review follow-up.
- SPARK-60119, which is in the same release candidates, makes Spark return
0.00 for a string far below the target scale's ulp, as the native cast
already does. The cast audit entry and the string.rs comments now say
that #6966's positive-scale case applies only before 4.0.5, 4.1.4 and
4.2.1, and that its negative-scale case still differs on them.
- The cast compatibility page and CometSequence's compatibility notes say
that Comet chooses by the Spark version number, and what a vendor build
that reports an older version but includes SPARK-60118 or SPARK-58821
returns, as floating-point.md does for SPARK-59602.
- recomputed_length_after_overflow keeps only the new element checks.
length_limit_and_overflow_edges and the length grid already cover the
internal error and both variants' limit errors.
- SparkFixesSuite holds one version table for both fixes and
CometSequence.nativeFunction, replacing the near-identical tables in
CometArrayExpressionSuite and CometNativeCastSuite, which keep their
convert checks. It is listed in both PR build workflows.
* fix: follow released Spark for '0e2147483647', and check Spark's error in the sequence test
Review follow-up.
- On the released path, parse_string_to_decimal counts a zero's integral
digits as 1 - scale with a wrapping i32 subtraction, as Spark's int
arithmetic does, so '0e2147483647' casts to 0 there. The zero test covers
it, and two inputs whose counts don't wrap.
- The sequence overflow test's released-Spark branch uses checkSparkError,
which checks Spark's INTERNAL_ERROR, exception class and SQLSTATE rather
than the message.
- The cast audit entry describes SPARK-60118's long digit count: released
Spark throws for '1e2147483647' and similar strings, the fixed releases
return NULL or raise NUMERIC_OUT_OF_SUPPORTED_RANGE, and the native cast's
ANSI error is #7188.
- SparkFixes' scaladoc names the two fixes it covers and points to the
other version checks.1 parent c4a0b1f commit 1dec4c0
19 files changed
Lines changed: 457 additions & 92 deletions
File tree
- .github/workflows
- docs/source
- contributor-guide/expression-audits
- user-guide/latest/compatibility/expressions/_category_template
- native
- core/src/execution
- proto/src/proto
- spark-expr/src
- array_funcs
- conversion_funcs
- spark/src
- main/scala/org/apache/comet
- expressions
- serde
- test
- resources/sql-tests/expressions/array
- scala/org/apache/comet
- serde
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
676 | 676 | | |
677 | 677 | | |
678 | 678 | | |
| 679 | + | |
679 | 680 | | |
680 | 681 | | |
681 | 682 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
342 | 342 | | |
343 | 343 | | |
344 | 344 | | |
| 345 | + | |
345 | 346 | | |
346 | 347 | | |
347 | 348 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
| 190 | + | |
190 | 191 | | |
191 | 192 | | |
192 | 193 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
78 | 84 | | |
79 | 85 | | |
80 | 86 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
776 | 776 | | |
777 | 777 | | |
778 | 778 | | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
779 | 785 | | |
780 | 786 | | |
781 | 787 | | |
782 | | - | |
783 | | - | |
784 | | - | |
785 | | - | |
786 | | - | |
| 788 | + | |
787 | 789 | | |
788 | 790 | | |
789 | 791 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
462 | 462 | | |
463 | 463 | | |
464 | 464 | | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
465 | 468 | | |
466 | 469 | | |
467 | 470 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
0 commit comments