Skip to content

Commit 275e4c2

Browse files
SEPURI-SAI-KRISHNAuros-b
authored andcommitted
[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_repeat and array_insert codegen
### What changes were proposed in this pull request? `array_repeat` and `array_insert` raise a different error for an array that exceeds the maximum array length depending on whether the expression runs through the interpreted path or through generated code. This PR makes the codegen paths report the same error the interpreted paths already report. **`array_repeat`** — `ArrayRepeat.eval` checks `count` against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` and raises `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, but `genCodeForNumberOfElements` performs no such check, so the generated code proceeds to the array allocation and fails with the internal `_LEGACY_ERROR_TEMP_2176`. The same bound check is added to the generated code. The interpreted path was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was not. **`array_insert`** — the codegen branch taken when `pos` is not foldable called `createArrayWithElementsExceedLimitError`, which hardcodes `"parameter" -> toSQLId("count")`. `array_insert(array, pos, value)` has no `count` parameter. It now calls `arrayFunctionWithElementsExceedLimitError`, matching all three interpreted sites and both sibling codegen sites in the same expression. Introduced by SPARK-56567 (4.2.0). `Sequence` also reports `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` naming a `count` parameter it does not have, but it does so from `sequenceLength`, which both execution paths share. There is no interpreted-vs-codegen inconsistency there, so it is left alone. ### Why are the changes needed? The same query produces a different error class depending on the execution mode, and codegen is the default path, so the wrong error is the one users normally see. `array_repeat` leaks an internal `_LEGACY_ERROR_TEMP_*` condition, which carries no SQLSTATE and is not meant to reach users. `array_insert` produces a message naming a parameter the function does not have, which is actively misleading: ``` the value of parameter(s) `count` in the function `array_insert` is invalid ``` Neither divergence was caught because the existing coverage does not reach the generated code. The existing test `"Elements exceed limit for array_repeat()"` in `QueryExecutionErrorsSuite` runs `select array_repeat(1, 2147483647)`, which is constant-folded and therefore only exercises the interpreted path. The `array_insert` branch is reachable only with a non-foldable `pos`, because a foldable positive `pos` takes a different codegen branch. ### Does this PR introduce _any_ user-facing change? Yes, in error reporting only. Both functions raise an error under exactly the same conditions as before, and neither produces different results. Before, `array_repeat` with a count above the limit, executed by codegen: ``` _LEGACY_ERROR_TEMP_2176 Cannot create array with 2147483647 elements of data due to exceeding the limit 2147483632 elements for ArrayData. array_repeat failed. ``` After, matching what the interpreted path already produced: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] Can't create array with 2147483647 elements which exceeding the array size limit 2147483632, the value of parameter(s) `count` in the function `array_repeat` is invalid. SQLSTATE: 54000 ``` Before, `array_insert` with a non-foldable `pos` above the limit, executed by codegen: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] ... the value of parameter(s) `count` in the function `array_insert` is invalid. SQLSTATE: 54000 ``` After, matching the interpreted path: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] ... unsuccessful try to create arrays in the function `array_insert`. SQLSTATE: 54000 ``` This is a change compared to released versions: the `array_repeat` divergence is present since 4.0.0 and the `array_insert` one since 4.2.0. ### How was this patch tested? Two cases added to `CollectionExpressionsSuite`, both using `checkErrorInExpression`, which runs the expression under `CODEGEN_ONLY` and `NO_CODEGEN` and so asserts the two paths agree. The `array_insert` case binds `pos` to a `BoundReference` so that the non-foldable branch is the one exercised. Both cases fail on the unpatched code. - `catalyst/testOnly *CollectionExpressionsSuite` — 63/63 pass - `sql/testOnly *QueryExecutionErrorsSuite -- -z "exceed limit"` — 3/3 pass, covering the existing `concat`, `flatten` and `array_repeat` limit tests The behaviour change was also confirmed end to end by evaluating each expression directly through both `eval` and `GenerateMutableProjection`, before and after the patch. Before, the two paths disagreed for both functions; after, both produce byte-identical messages. `sequence`, which this PR does not touch, still diverges, confirming the check was live. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #57832 from SEPURI-SAI-KRISHNA/array-size-limit-codegen-errors. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit 21c9062) Signed-off-by: Uros Bojanic <[email protected]>
1 parent e5bf1d6 commit 275e4c2

3 files changed

Lines changed: 35 additions & 1 deletion

File tree

‎docs/sql-migration-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ license: |
2828
- Since Spark 4.4, the Oracle JDBC datasource maps Oracle DATE and TIMESTAMP to TimestampNTZType unconditionally, ignoring the JDBC read option `preferTimestampNTZ` (DATE only when the driver default `oracle.jdbc.mapDateToTimestamp=true` surfaces it as TIMESTAMP; otherwise DATE stays DateType). In 4.3 and earlier these were read as TimestampType, or as TimestampNTZType only when `preferTimestampNTZ=true`. This faithfully represents these zoneless Oracle types. Unaffected: Oracle DATE read with `oracle.jdbc.mapDateToTimestamp=false` (mapped to DateType, as before), and TIMESTAMP WITH TIME ZONE / TIMESTAMP WITH LOCAL TIME ZONE. Symmetrically, a TimestampNTZType column written to Oracle is now bound zoneless via `setObject` rather than a JVM-default-zone `java.sql.Timestamp`, which can change the stored value for a wall-clock that falls in the JVM default zone's DST gap. To restore the previous read and write behavior, set `spark.sql.legacy.oracle.timestampNTZMapping.enabled` to `true`.
2929
- Since Spark 4.4, a nondeterministic common expression that a conditional branch reads more than once is evaluated once, and every read sees that one value. `BETWEEN` and `NULLIF` read their input twice, so an expression such as `CASE WHEN c THEN monotonically_increasing_id() BETWEEN 3 AND 5 END` previously gave each read its own value and could return a different result. Queries whose results depended on the earlier behaviour change, and an optimized or physical plan may now show a `with` node for such a branch. A definition that is cheap to evaluate and reports itself deterministic is still substituted at every read, as before.
3030
- Since Spark 4.4, for storage-partitioned joins, `spark.sql.requireAllClusterKeysForCoPartition` requires every join key to be covered by some partition key instead of matching the partition keys positionally. As a result, a join-key column partitioned by more than one transform no longer prevents shuffle elimination, and `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` no longer additionally requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false` when the join keys are a subset of the partition keys. As before, when the partition keys cover only part of the join keys, eliminating the shuffle still requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false`.
31+
- Since Spark 4.4, when `array_repeat` or `array_insert` is asked to build an array larger than the maximum supported array length, generated code raises the same error as interpreted evaluation. `array_repeat` now fails with `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` instead of the internal error `_LEGACY_ERROR_TEMP_2176`, and `array_insert` fails with `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` instead of `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, which named a `count` parameter that `array_insert` does not have. Both functions raise an error under exactly the same conditions as before; only the reported error condition changes.
3132

3233
## Upgrading from Spark SQL 4.2 to 4.3
3334

‎sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4235,8 +4235,14 @@ case class ArrayRepeat(left: Expression, right: Expression)
42354235

42364236
private def genCodeForNumberOfElements(ctx: CodegenContext, count: String): (String, String) = {
42374237
val numElements = ctx.freshName("numElements")
4238+
// The upper bound is checked here rather than left to the array allocation so that this path
4239+
// reports the same error as `eval`. Without it the allocation fails with an internal error.
42384240
val numElementsCode =
42394241
s"""
4242+
|if ($count > ${ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}) {
4243+
| throw QueryExecutionErrors.createArrayWithElementsExceedLimitError(
4244+
| "$prettyName", $count);
4245+
|}
42404246
|int $numElements = 0;
42414247
|if ($count > 0) {
42424248
| $numElements = $count;
@@ -5636,7 +5642,7 @@ case class ArrayInsert(
56365642
|
56375643
| $resLength = java.lang.Math.max($arr.numElements() + 1, $itemInsertionIndex + 1);
56385644
| if ($resLength > ${ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}) {
5639-
| throw QueryExecutionErrors.createArrayWithElementsExceedLimitError(
5645+
| throw QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError(
56405646
| "$prettyName", $resLength);
56415647
| }
56425648
|

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2549,6 +2549,17 @@ class CollectionExpressionsSuite
25492549
checkEvaluation(ArrayRepeat(intArray, Literal(2)), Seq(Seq(1, 2), Seq(1, 2)))
25502550
checkEvaluation(ArrayRepeat(strArray, Literal(2)), Seq(Seq("hi", "hola"), Seq("hi", "hola")))
25512551
checkEvaluation(ArrayRepeat(Literal("hi"), Literal(null, IntegerType)), null)
2552+
2553+
// A count above the max array length must raise the same error in the interpreted and the
2554+
// codegen path. The codegen path used to let the array allocation fail with an internal error.
2555+
checkErrorInExpression[SparkRuntimeException](
2556+
ArrayRepeat(Literal("hi"), Literal(Int.MaxValue)),
2557+
condition = "COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER",
2558+
parameters = Map(
2559+
"numberOfElements" -> Int.MaxValue.toString,
2560+
"functionName" -> toSQLId("array_repeat"),
2561+
"maxRoundedArrayLength" -> ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH.toString,
2562+
"parameter" -> toSQLId("count")))
25522563
}
25532564

25542565
test("Array remove") {
@@ -3471,6 +3482,22 @@ class CollectionExpressionsSuite
34713482
"functionName" -> "`array_insert`"))
34723483
}
34733484

3485+
test("SPARK-58631: Array insert with a non-foldable pos above the max array length") {
3486+
val a = Literal.create(Seq(1, 2, 4), ArrayType(IntegerType))
3487+
// A foldable positive `pos` is handled by a separate codegen branch, so `pos` has to be
3488+
// non-foldable to reach the one that used to report COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER,
3489+
// naming a `count` parameter that array_insert does not have.
3490+
checkErrorInExpression[SparkRuntimeException](
3491+
ArrayInsert(a, BoundReference(0, IntegerType, nullable = false), Literal(3),
3492+
legacyNegativeIndex = false),
3493+
InternalRow(Int.MaxValue),
3494+
condition = "COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION",
3495+
parameters = Map(
3496+
"numberOfElements" -> Int.MaxValue.toString,
3497+
"maxRoundedArrayLength" -> ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH.toString,
3498+
"functionName" -> "`array_insert`"))
3499+
}
3500+
34743501
test("SPARK-57335: element_at with index = Int.MinValue") {
34753502
val a = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType))
34763503
// Non-ANSI: an out-of-bounds index must return null. `Math.abs(Int.MinValue)`

0 commit comments

Comments
 (0)