Repository navigation
Commit 275e4c2
[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_repeat and array_insert codegen
### What changes were proposed in this pull request?
`array_repeat` and `array_insert` raise a different error for an array that exceeds the maximum array length depending on whether the expression runs through the interpreted path or through generated code. This PR makes the codegen paths report the same error the interpreted paths already report.
**`array_repeat`** — `ArrayRepeat.eval` checks `count` against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` and raises `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, but `genCodeForNumberOfElements` performs no such check, so the generated code proceeds to the array allocation and fails with the internal `_LEGACY_ERROR_TEMP_2176`. The same bound check is added to the generated code. The interpreted path was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was not.
**`array_insert`** — the codegen branch taken when `pos` is not foldable called `createArrayWithElementsExceedLimitError`, which hardcodes `"parameter" -> toSQLId("count")`. `array_insert(array, pos, value)` has no `count` parameter. It now calls `arrayFunctionWithElementsExceedLimitError`, matching all three interpreted sites and both sibling codegen sites in the same expression. Introduced by SPARK-56567 (4.2.0).
`Sequence` also reports `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` naming a `count` parameter it does not have, but it does so from `sequenceLength`, which both execution paths share. There is no interpreted-vs-codegen inconsistency there, so it is left alone.
### Why are the changes needed?
The same query produces a different error class depending on the execution mode, and codegen is the default path, so the wrong error is the one users normally see.
`array_repeat` leaks an internal `_LEGACY_ERROR_TEMP_*` condition, which carries no SQLSTATE and is not meant to reach users. `array_insert` produces a message naming a parameter the function does not have, which is actively misleading:
```
the value of parameter(s) `count` in the function `array_insert` is invalid
```
Neither divergence was caught because the existing coverage does not reach the generated code. The existing test `"Elements exceed limit for array_repeat()"` in `QueryExecutionErrorsSuite` runs `select array_repeat(1, 2147483647)`, which is constant-folded and therefore only exercises the interpreted path. The `array_insert` branch is reachable only with a non-foldable `pos`, because a foldable positive `pos` takes a different codegen branch.
### Does this PR introduce _any_ user-facing change?
Yes, in error reporting only. Both functions raise an error under exactly the same conditions as before, and neither produces different results.
Before, `array_repeat` with a count above the limit, executed by codegen:
```
_LEGACY_ERROR_TEMP_2176
Cannot create array with 2147483647 elements of data due to exceeding the limit
2147483632 elements for ArrayData. array_repeat failed.
```
After, matching what the interpreted path already produced:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] Can't create array with 2147483647 elements
which exceeding the array size limit 2147483632, the value of parameter(s) `count`
in the function `array_repeat` is invalid. SQLSTATE: 54000
```
Before, `array_insert` with a non-foldable `pos` above the limit, executed by codegen:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] ... the value of parameter(s) `count`
in the function `array_insert` is invalid. SQLSTATE: 54000
```
After, matching the interpreted path:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] ... unsuccessful try to create arrays
in the function `array_insert`. SQLSTATE: 54000
```
This is a change compared to released versions: the `array_repeat` divergence is present since 4.0.0 and the `array_insert` one since 4.2.0.
### How was this patch tested?
Two cases added to `CollectionExpressionsSuite`, both using `checkErrorInExpression`, which runs the expression under `CODEGEN_ONLY` and `NO_CODEGEN` and so asserts the two paths agree. The `array_insert` case binds `pos` to a `BoundReference` so that the non-foldable branch is the one exercised. Both cases fail on the unpatched code.
- `catalyst/testOnly *CollectionExpressionsSuite` — 63/63 pass
- `sql/testOnly *QueryExecutionErrorsSuite -- -z "exceed limit"` — 3/3 pass, covering the existing `concat`, `flatten` and `array_repeat` limit tests
The behaviour change was also confirmed end to end by evaluating each expression directly through both `eval` and `GenerateMutableProjection`, before and after the patch. Before, the two paths disagreed for both functions; after, both produce byte-identical messages. `sequence`, which this PR does not touch, still diverges, confirming the check was live.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57832 from SEPURI-SAI-KRISHNA/array-size-limit-codegen-errors.
Authored-by: Sepuri Sai Krishna <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 21c9062)
Signed-off-by: Uros Bojanic <[email protected]>1 parent e5bf1d6 commit 275e4c2
3 files changed
Lines changed: 35 additions & 1 deletion
File tree
- docs
- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4235 | 4235 | | |
4236 | 4236 | | |
4237 | 4237 | | |
| 4238 | + | |
| 4239 | + | |
4238 | 4240 | | |
4239 | 4241 | | |
| 4242 | + | |
| 4243 | + | |
| 4244 | + | |
| 4245 | + | |
4240 | 4246 | | |
4241 | 4247 | | |
4242 | 4248 | | |
| |||
5636 | 5642 | | |
5637 | 5643 | | |
5638 | 5644 | | |
5639 | | - | |
| 5645 | + | |
5640 | 5646 | | |
5641 | 5647 | | |
5642 | 5648 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2549 | 2549 | | |
2550 | 2550 | | |
2551 | 2551 | | |
| 2552 | + | |
| 2553 | + | |
| 2554 | + | |
| 2555 | + | |
| 2556 | + | |
| 2557 | + | |
| 2558 | + | |
| 2559 | + | |
| 2560 | + | |
| 2561 | + | |
| 2562 | + | |
2552 | 2563 | | |
2553 | 2564 | | |
2554 | 2565 | | |
| |||
3471 | 3482 | | |
3472 | 3483 | | |
3473 | 3484 | | |
| 3485 | + | |
| 3486 | + | |
| 3487 | + | |
| 3488 | + | |
| 3489 | + | |
| 3490 | + | |
| 3491 | + | |
| 3492 | + | |
| 3493 | + | |
| 3494 | + | |
| 3495 | + | |
| 3496 | + | |
| 3497 | + | |
| 3498 | + | |
| 3499 | + | |
| 3500 | + | |
3474 | 3501 | | |
3475 | 3502 | | |
3476 | 3503 | | |
| |||
0 commit comments