[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_repeat and array_insert codegen by SEPURI-SAI-KRISHNA · Pull Request #57832 · apache/spark

SEPURI-SAI-KRISHNA · 2026-08-07T02:07:37Z

What changes were proposed in this pull request?

array_repeat and array_insert raise a different error for an array that exceeds the maximum array length depending on whether the expression runs through the interpreted path or through generated code. This PR makes the codegen paths report the same error the interpreted paths already report.

array_repeat — ArrayRepeat.eval checks count against ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH and raises COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER, but genCodeForNumberOfElements performs no such check, so the generated code proceeds to the array allocation and fails with the internal _LEGACY_ERROR_TEMP_2176. The same bound check is added to the generated code. The interpreted path was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was not.

array_insert — the codegen branch taken when pos is not foldable called createArrayWithElementsExceedLimitError, which hardcodes "parameter" -> toSQLId("count"). array_insert(array, pos, value) has no count parameter. It now calls arrayFunctionWithElementsExceedLimitError, matching all three interpreted sites and both sibling codegen sites in the same expression. Introduced by SPARK-56567 (4.2.0).

Sequence also reports COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER naming a count parameter it does not have, but it does so from sequenceLength, which both execution paths share. There is no interpreted-vs-codegen inconsistency there, so it is left alone.

Why are the changes needed?

The same query produces a different error class depending on the execution mode, and codegen is the default path, so the wrong error is the one users normally see.

array_repeat leaks an internal _LEGACY_ERROR_TEMP_* condition, which carries no SQLSTATE and is not meant to reach users. array_insert produces a message naming a parameter the function does not have, which is actively misleading:

the value of parameter(s) `count` in the function `array_insert` is invalid

Neither divergence was caught because the existing coverage does not reach the generated code. The existing test "Elements exceed limit for array_repeat()" in QueryExecutionErrorsSuite runs select array_repeat(1, 2147483647), which is constant-folded and therefore only exercises the interpreted path. The array_insert branch is reachable only with a non-foldable pos, because a foldable positive pos takes a different codegen branch.

Does this PR introduce any user-facing change?

Yes, in error reporting only. Both functions raise an error under exactly the same conditions as before, and neither produces different results.

Before, array_repeat with a count above the limit, executed by codegen:

_LEGACY_ERROR_TEMP_2176
Cannot create array with 2147483647 elements of data due to exceeding the limit
2147483632 elements for ArrayData.  array_repeat failed.

After, matching what the interpreted path already produced:

[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] Can't create array with 2147483647 elements
which exceeding the array size limit 2147483632, the value of parameter(s) `count`
in the function `array_repeat` is invalid. SQLSTATE: 54000

Before, array_insert with a non-foldable pos above the limit, executed by codegen:

[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] ... the value of parameter(s) `count`
in the function `array_insert` is invalid. SQLSTATE: 54000

After, matching the interpreted path:

[COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] ... unsuccessful try to create arrays
in the function `array_insert`. SQLSTATE: 54000

This is a change compared to released versions: the array_repeat divergence is present since 4.0.0 and the array_insert one since 4.2.0.

How was this patch tested?

Two cases added to CollectionExpressionsSuite, both using checkErrorInExpression, which runs the expression under CODEGEN_ONLY and NO_CODEGEN and so asserts the two paths agree. The array_insert case binds pos to a BoundReference so that the non-foldable branch is the one exercised. Both cases fail on the unpatched code.

catalyst/testOnly *CollectionExpressionsSuite — 63/63 pass
sql/testOnly *QueryExecutionErrorsSuite -- -z "exceed limit" — 3/3 pass, covering the existing concat, flatten and array_repeat limit tests

The behaviour change was also confirmed end to end by evaluating each expression directly through both eval and GenerateMutableProjection, before and after the patch. Before, the two paths disagreed for both functions; after, both produce byte-identical messages. sequence, which this PR does not touch, still diverges, confirming the check was live.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…repeat and array_insert codegen

uros-b

Please update the SQL migration guide accordingly

…ion change. Co-Authored-By: Claude Opus 5 <[email protected]>

SEPURI-SAI-KRISHNA · 2026-08-07T11:20:14Z

Thanks @uros-b added an entry to the 4.3 to 4.4 section of docs/sql-migration-guide.md covering both error condition changes. Used 4.4 since the fix should land on branch-4.x as well.

Co-Authored-By: Claude Opus 5 <[email protected]>

…codegen-errors # Conflicts: # docs/sql-migration-guide.md

SEPURI-SAI-KRISHNA · 2026-09-06T09:00:50Z

@uros-b the migration guide entry is in, thanks for catching that.

Two things have changed since your review. I merged master to resolve a conflict in docs/sql-migration-guide.md: master had added new entries at the end of the 4.2 to 4.3 section, exactly where mine sat.

While resolving it I also moved the entry to the Upgrading from Spark SQL 4.3 to 4.4 section and changed it to read "Since Spark 4.4". When I first wrote it, branch-4.3 was still unreleased and 4.3 was the right target. 4.3.0-rc1 was cut on Aug 31 and branch-4.3 is now on 4.3.1-SNAPSHOT, so this change will first ship in 4.4. Note that this version move is part of the merge commit rather than its own commit, so it does not show up in the commit titles - the earlier "Move the migration guide entry to the 4.3 section" commit is superseded by it.

The wording is otherwise unchanged, and the code and test changes are untouched.

Would appreciate another look when you have a chance.

SEPURI-SAI-KRISHNA · 2026-09-09T14:12:11Z

@uros-b when you have a moment, would you mind another look at this one? It is green and still merges cleanly, and your migration guide point was addressed back in August, with the entry moved to the 4.3 to 4.4 section after 4.3.0-rc1 was cut.

No rush if you are working through the queue, I just want to make sure it has not fallen off it.

uros-b · 2026-09-09T15:13:55Z

Could you please check whether the same pattern still exists in concat and flatten? We don't need to do anything here now, but let's use the opportunity to investigate and at least open up new Jira tickets for those as well.

uros-b

LGTM, thank you @SEPURI-SAI-KRISHNA!

…repeat and array_insert codegen ### What changes were proposed in this pull request? `array_repeat` and `array_insert` raise a different error for an array that exceeds the maximum array length depending on whether the expression runs through the interpreted path or through generated code. This PR makes the codegen paths report the same error the interpreted paths already report. **`array_repeat`** — `ArrayRepeat.eval` checks `count` against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` and raises `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, but `genCodeForNumberOfElements` performs no such check, so the generated code proceeds to the array allocation and fails with the internal `_LEGACY_ERROR_TEMP_2176`. The same bound check is added to the generated code. The interpreted path was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was not. **`array_insert`** — the codegen branch taken when `pos` is not foldable called `createArrayWithElementsExceedLimitError`, which hardcodes `"parameter" -> toSQLId("count")`. `array_insert(array, pos, value)` has no `count` parameter. It now calls `arrayFunctionWithElementsExceedLimitError`, matching all three interpreted sites and both sibling codegen sites in the same expression. Introduced by SPARK-56567 (4.2.0). `Sequence` also reports `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` naming a `count` parameter it does not have, but it does so from `sequenceLength`, which both execution paths share. There is no interpreted-vs-codegen inconsistency there, so it is left alone. ### Why are the changes needed? The same query produces a different error class depending on the execution mode, and codegen is the default path, so the wrong error is the one users normally see. `array_repeat` leaks an internal `_LEGACY_ERROR_TEMP_*` condition, which carries no SQLSTATE and is not meant to reach users. `array_insert` produces a message naming a parameter the function does not have, which is actively misleading: ``` the value of parameter(s) `count` in the function `array_insert` is invalid ``` Neither divergence was caught because the existing coverage does not reach the generated code. The existing test `"Elements exceed limit for array_repeat()"` in `QueryExecutionErrorsSuite` runs `select array_repeat(1, 2147483647)`, which is constant-folded and therefore only exercises the interpreted path. The `array_insert` branch is reachable only with a non-foldable `pos`, because a foldable positive `pos` takes a different codegen branch. ### Does this PR introduce _any_ user-facing change? Yes, in error reporting only. Both functions raise an error under exactly the same conditions as before, and neither produces different results. Before, `array_repeat` with a count above the limit, executed by codegen: ``` _LEGACY_ERROR_TEMP_2176 Cannot create array with 2147483647 elements of data due to exceeding the limit 2147483632 elements for ArrayData. array_repeat failed. ``` After, matching what the interpreted path already produced: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] Can't create array with 2147483647 elements which exceeding the array size limit 2147483632, the value of parameter(s) `count` in the function `array_repeat` is invalid. SQLSTATE: 54000 ``` Before, `array_insert` with a non-foldable `pos` above the limit, executed by codegen: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] ... the value of parameter(s) `count` in the function `array_insert` is invalid. SQLSTATE: 54000 ``` After, matching the interpreted path: ``` [COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] ... unsuccessful try to create arrays in the function `array_insert`. SQLSTATE: 54000 ``` This is a change compared to released versions: the `array_repeat` divergence is present since 4.0.0 and the `array_insert` one since 4.2.0. ### How was this patch tested? Two cases added to `CollectionExpressionsSuite`, both using `checkErrorInExpression`, which runs the expression under `CODEGEN_ONLY` and `NO_CODEGEN` and so asserts the two paths agree. The `array_insert` case binds `pos` to a `BoundReference` so that the non-foldable branch is the one exercised. Both cases fail on the unpatched code. - `catalyst/testOnly *CollectionExpressionsSuite` — 63/63 pass - `sql/testOnly *QueryExecutionErrorsSuite -- -z "exceed limit"` — 3/3 pass, covering the existing `concat`, `flatten` and `array_repeat` limit tests The behaviour change was also confirmed end to end by evaluating each expression directly through both `eval` and `GenerateMutableProjection`, before and after the patch. Before, the two paths disagreed for both functions; after, both produce byte-identical messages. `sequence`, which this PR does not touch, still diverges, confirming the check was live. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #57832 from SEPURI-SAI-KRISHNA/array-size-limit-codegen-errors. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit 21c9062) Signed-off-by: Uros Bojanic <[email protected]>

uros-b · 2026-09-09T15:50:02Z

Merge Summary:

merged into master 21c9062
merged into branch-4.x 275e4c2

Posted by merge_spark_pr.py

SEPURI-SAI-KRISHNA · 2026-09-09T15:53:56Z

@uros-b checked, and the same pattern is there in both. I also swept the rest of the file rather than just those two, so we can close the question instead of leaving it open.

concat and flatten

Concat.genCodeForNumberOfElements and Flatten.genCodeForNumberOfElements both accumulate the element count into a long and hand it straight to CodeGenerator.createArrayData with no MAX_ROUNDED_ARRAY_LENGTH check, exactly like array_repeat before this PR. Their interpreted paths do check, and raise COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION. Codegen instead reaches ArrayData.allocateArrayData, whose own bound check raises the internal _LEGACY_ERROR_TEMP_2176. Same shape of divergence, same fix.

Everything else that touches the limit

expression	interpreted	codegen
`array_repeat`	checks	no check	fixed here
`array_insert`	checks	checks	fixed here
`concat`	checks	no check	gap
`flatten`	checks	no check	gap
`sequence`	checks	shared	not a gap, see below
`array_distinct`, `array_union`	checks	checks	consistent
`array_intersect`, `array_except`	none	checks	result is a subset of an input, so the limit is unreachable and the codegen check is defensive
`map_zip_with`	checks	none	`CodegenFallback`, so there is no codegen path

sequence is worth calling out because it looks like a third gap: it checks in sequenceLength and has no MAX_ROUNDED_ARRAY_LENGTH in its generated code. It is not one. Its codegen calls Sequence.sequenceLength directly, so both paths go through the same check. Anyone grepping for the pattern will land on it, so it seemed worth recording why it is fine.

I looked outside this file too. The other users of the constant are infrastructure rather than expression pairs: ByteArrayMethods, BufferHolder, WritableColumnVector, ArrayData, UTF8StringBuilder, StringConcat and the core buffers.

How much this matters

array_repeat reaches the limit cheaply, since count is just an argument. concat and flatten need the inputs to hold that many elements for real, so the executor runs out of memory long before the check is reached. The divergence looks latent rather than something a user hits today, which fits your read that nothing needs doing here now.

I also checked the branch above that check, since allocateArrayData narrows with numElements.toInt when it picks UnsafeArrayData. That branch is not reachable with an oversized count: for any element size of at least one byte, shouldUseGenericArrayData is already true by the time the length passes Int.MaxValue, so the generic branch and its bound check always win. No wrong-result risk, only the error condition.

Happy to open the tickets. Would you like one each for concat and flatten, or a single ticket covering both, given the fix is the same shape in each?

SEPURI-SAI-KRISHNA · 2026-09-09T16:22:40Z

@uros-b filed both:

SPARK-59374 for concat
SPARK-59375 for flatten

Kept them separate since they are independent call sites, and linked both to SPARK-58631. Filed as Minor rather than Major, on the reasoning above that the limit is not cheaply reachable for either function.

I plan to pick them up myself, unless you would rather they stay open for someone else.

uros-b · 2026-09-10T10:51:42Z

Thank you @SEPURI-SAI-KRISHNA! Please feel free to pick these up and continue working on them, and ping for reviews once PRs are ready!

### What changes were proposed in this pull request? `Concat.genCodeForNumberOfElements` accumulates the result element count into a `long` and hands it to `CodeGenerator.createArrayData` without checking it against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`. Interpreted evaluation does check. This PR adds the same check to the generated code so both paths raise the same error. This is the `concat` half of the pattern that [SPARK-58631](https://issues.apache.org/jira/browse/SPARK-58631) (#57832) fixed for `array_repeat`. `flatten` has the identical gap and is tracked separately as [SPARK-59375](https://issues.apache.org/jira/browse/SPARK-59375). ### Why are the changes needed? The two evaluation paths report different error conditions for the same input: - Interpreted: `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`, from `QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError`. - Codegen: no check, so execution reaches `ArrayData.allocateArrayData`, whose own bound check raises the internal `_LEGACY_ERROR_TEMP_2176` through `QueryExecutionErrors.cannotCreateArrayWithElementsExceedLimitError`. An internal error condition is not meant to be user facing, and which one a query gets should not depend on whether codegen was used. Worth noting for reviewers that this is latent rather than something a user hits today. Unlike `array_repeat`, where the element count comes straight from an argument, `concat` needs its inputs to actually hold that many elements, so an executor will normally run out of memory before the limit is reached. The existing test reaches it only by using a `ConstantColumnVector`, which reports its length without allocating. ### Does this PR introduce _any_ user-facing change? Yes, in the sense that the error condition changes for an input that already fails. `concat` with more than `MAX_ROUNDED_ARRAY_LENGTH` total elements now raises `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` under codegen, where it previously raised the internal `_LEGACY_ERROR_TEMP_2176`. That matches what interpreted evaluation has always raised. The set of inputs that fail is unchanged. Added a migration guide entry alongside the one #57832 added for `array_repeat` and `array_insert`, since this is the same kind of change. Happy to drop it if you would rather not document a case that is this hard to reach. ### How was this patch tested? Extended the existing `Elements exceed limit for concat()` test in `QueryExecutionErrorsSuite` to run the expression through `GenerateUnsafeProjection` as well as `eval`, asserting the same error from both. That test already builds a large array cheaply with `ConstantColumnVector`, which reports its length without allocating, so nothing large is allocated. It sits in `QueryExecutionErrorsSuite` rather than in `CollectionExpressionsSuite`, where #57832 put its tests, because `ConstantColumnVector` lives in `sql/core` and is not reachable from a catalyst suite. Extending the existing test also seemed better than adding a second one: the test was already there and simply never exercised codegen, which is how the gap survived. Reverting the change makes the new codegen case fail with `_LEGACY_ERROR_TEMP_2176`, which is the behaviour being fixed. ``` build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z \"Elements exceed limit\"" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #58703 from SEPURI-SAI-KRISHNA/SPARK-59374-concat-codegen-limit. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]>

### What changes were proposed in this pull request? `Concat.genCodeForNumberOfElements` accumulates the result element count into a `long` and hands it to `CodeGenerator.createArrayData` without checking it against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`. Interpreted evaluation does check. This PR adds the same check to the generated code so both paths raise the same error. This is the `concat` half of the pattern that [SPARK-58631](https://issues.apache.org/jira/browse/SPARK-58631) (#57832) fixed for `array_repeat`. `flatten` has the identical gap and is tracked separately as [SPARK-59375](https://issues.apache.org/jira/browse/SPARK-59375). ### Why are the changes needed? The two evaluation paths report different error conditions for the same input: - Interpreted: `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`, from `QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError`. - Codegen: no check, so execution reaches `ArrayData.allocateArrayData`, whose own bound check raises the internal `_LEGACY_ERROR_TEMP_2176` through `QueryExecutionErrors.cannotCreateArrayWithElementsExceedLimitError`. An internal error condition is not meant to be user facing, and which one a query gets should not depend on whether codegen was used. Worth noting for reviewers that this is latent rather than something a user hits today. Unlike `array_repeat`, where the element count comes straight from an argument, `concat` needs its inputs to actually hold that many elements, so an executor will normally run out of memory before the limit is reached. The existing test reaches it only by using a `ConstantColumnVector`, which reports its length without allocating. ### Does this PR introduce _any_ user-facing change? Yes, in the sense that the error condition changes for an input that already fails. `concat` with more than `MAX_ROUNDED_ARRAY_LENGTH` total elements now raises `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` under codegen, where it previously raised the internal `_LEGACY_ERROR_TEMP_2176`. That matches what interpreted evaluation has always raised. The set of inputs that fail is unchanged. Added a migration guide entry alongside the one #57832 added for `array_repeat` and `array_insert`, since this is the same kind of change. Happy to drop it if you would rather not document a case that is this hard to reach. ### How was this patch tested? Extended the existing `Elements exceed limit for concat()` test in `QueryExecutionErrorsSuite` to run the expression through `GenerateUnsafeProjection` as well as `eval`, asserting the same error from both. That test already builds a large array cheaply with `ConstantColumnVector`, which reports its length without allocating, so nothing large is allocated. It sits in `QueryExecutionErrorsSuite` rather than in `CollectionExpressionsSuite`, where #57832 put its tests, because `ConstantColumnVector` lives in `sql/core` and is not reachable from a catalyst suite. Extending the existing test also seemed better than adding a second one: the test was already there and simply never exercised codegen, which is how the gap survived. Reverting the change makes the new codegen case fail with `_LEGACY_ERROR_TEMP_2176`, which is the behaviour being fixed. ``` build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z \"Elements exceed limit\"" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #58703 from SEPURI-SAI-KRISHNA/SPARK-59374-concat-codegen-limit. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit fc091f5) Signed-off-by: Uros Bojanic <[email protected]>

### What changes were proposed in this pull request? `Flatten.genCodeForNumberOfElements` accumulates the result element count into a `long` and hands it to `CodeGenerator.createArrayData` without checking it against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`. Interpreted evaluation does check. This PR adds the same check to the generated code so both paths raise the same error. This is the `flatten` half of the pattern that [SPARK-58631](https://issues.apache.org/jira/browse/SPARK-58631) (#57832) fixed for `array_repeat`. `concat` has the identical gap and is fixed in #58703 ([SPARK-59374](https://issues.apache.org/jira/browse/SPARK-59374)). ### Why are the changes needed? The two evaluation paths report different error conditions for the same input: - Interpreted: `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`, from `QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError`. - Codegen: no check, so execution reaches `ArrayData.allocateArrayData`, whose own bound check raises the internal `_LEGACY_ERROR_TEMP_2176` through `QueryExecutionErrors.cannotCreateArrayWithElementsExceedLimitError`. An internal error condition is not meant to be user facing, and which one a query gets should not depend on whether codegen was used. Worth noting for reviewers that this is latent rather than something a user hits today. Unlike `array_repeat`, where the element count comes straight from an argument, `flatten` needs its inputs to actually hold that many elements, so an executor will normally run out of memory before the limit is reached. The existing test reaches it only by using a `ConstantColumnVector`, which reports its length without allocating. ### Does this PR introduce _any_ user-facing change? Yes, in the sense that the error condition changes for an input that already fails. `flatten` with more than `MAX_ROUNDED_ARRAY_LENGTH` total elements now raises `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` under codegen, where it previously raised the internal `_LEGACY_ERROR_TEMP_2176`. That matches what interpreted evaluation has always raised. The set of inputs that fail is unchanged. Added a migration guide entry alongside the one #57832 added for `array_repeat` and `array_insert`, and the one #58703 adds for `concat`, since this is the same kind of change. Happy to drop it if you would rather not document a case that is this hard to reach. ### How was this patch tested? Extended the existing `Elements exceed limit for flatten()` test in `QueryExecutionErrorsSuite` to run the expression through `GenerateUnsafeProjection` as well as `eval`, asserting the same error from both. That test already builds a large array cheaply with `ConstantColumnVector`, which reports its length without allocating, so nothing large is allocated. It sits in `QueryExecutionErrorsSuite` rather than in `CollectionExpressionsSuite`, where #57832 put its tests, because `ConstantColumnVector` lives in `sql/core` and is not reachable from a catalyst suite. Extending the existing test also seemed better than adding a second one: the test was already there and simply never exercised codegen, which is how the gap survived. Reverting the change makes the new codegen case fail with `_LEGACY_ERROR_TEMP_2176`, which is the behaviour being fixed. ``` build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z \"Elements exceed limit\"" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #58707 from SEPURI-SAI-KRISHNA/SPARK-59375-flatten-codegen-limit. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]>

### What changes were proposed in this pull request? `Flatten.genCodeForNumberOfElements` accumulates the result element count into a `long` and hands it to `CodeGenerator.createArrayData` without checking it against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`. Interpreted evaluation does check. This PR adds the same check to the generated code so both paths raise the same error. This is the `flatten` half of the pattern that [SPARK-58631](https://issues.apache.org/jira/browse/SPARK-58631) (#57832) fixed for `array_repeat`. `concat` has the identical gap and is fixed in #58703 ([SPARK-59374](https://issues.apache.org/jira/browse/SPARK-59374)). ### Why are the changes needed? The two evaluation paths report different error conditions for the same input: - Interpreted: `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`, from `QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError`. - Codegen: no check, so execution reaches `ArrayData.allocateArrayData`, whose own bound check raises the internal `_LEGACY_ERROR_TEMP_2176` through `QueryExecutionErrors.cannotCreateArrayWithElementsExceedLimitError`. An internal error condition is not meant to be user facing, and which one a query gets should not depend on whether codegen was used. Worth noting for reviewers that this is latent rather than something a user hits today. Unlike `array_repeat`, where the element count comes straight from an argument, `flatten` needs its inputs to actually hold that many elements, so an executor will normally run out of memory before the limit is reached. The existing test reaches it only by using a `ConstantColumnVector`, which reports its length without allocating. ### Does this PR introduce _any_ user-facing change? Yes, in the sense that the error condition changes for an input that already fails. `flatten` with more than `MAX_ROUNDED_ARRAY_LENGTH` total elements now raises `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` under codegen, where it previously raised the internal `_LEGACY_ERROR_TEMP_2176`. That matches what interpreted evaluation has always raised. The set of inputs that fail is unchanged. Added a migration guide entry alongside the one #57832 added for `array_repeat` and `array_insert`, and the one #58703 adds for `concat`, since this is the same kind of change. Happy to drop it if you would rather not document a case that is this hard to reach. ### How was this patch tested? Extended the existing `Elements exceed limit for flatten()` test in `QueryExecutionErrorsSuite` to run the expression through `GenerateUnsafeProjection` as well as `eval`, asserting the same error from both. That test already builds a large array cheaply with `ConstantColumnVector`, which reports its length without allocating, so nothing large is allocated. It sits in `QueryExecutionErrorsSuite` rather than in `CollectionExpressionsSuite`, where #57832 put its tests, because `ConstantColumnVector` lives in `sql/core` and is not reachable from a catalyst suite. Extending the existing test also seemed better than adding a second one: the test was already there and simply never exercised codegen, which is how the gap survived. Reverting the change makes the new codegen case fail with `_LEGACY_ERROR_TEMP_2176`, which is the behaviour being fixed. ``` build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z \"Elements exceed limit\"" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) Closes #58707 from SEPURI-SAI-KRISHNA/SPARK-59375-flatten-codegen-limit. Authored-by: Sepuri Sai Krishna <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit eaa4ade) Signed-off-by: Uros Bojanic <[email protected]>

[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_…

2f1cb53

…repeat and array_insert codegen

uros-b reviewed Aug 7, 2026

View reviewed changes

[SPARK-58631][SQL] Add SQL migration guide entry for the error condit…

3007a7a

…ion change. Co-Authored-By: Claude Opus 5 <[email protected]>

SEPURI-SAI-KRISHNA added 2 commits August 10, 2026 11:55

[SPARK-58631][SQL] Move the migration guide entry to the 4.3 section

b147998

Co-Authored-By: Claude Opus 5 <[email protected]>

Merge remote-tracking branch 'upstream/master' into array-size-limit-…

cf94ee4

…codegen-errors # Conflicts: # docs/sql-migration-guide.md

uros-b approved these changes Sep 9, 2026

View reviewed changes

uros-b closed this in 21c9062 Sep 9, 2026

This was referenced Sep 10, 2026

[SPARK-59374][SQL] Check the array size limit in concat codegen #58703

Closed

[SPARK-59375][SQL] Check the array size limit in flatten codegen #58707

Closed

SEPURI-SAI-KRISHNA mentioned this pull request Sep 21, 2026

[SPARK-59686][SQL] Clamp the rounding scale in round and bround #58939

Open

Navigation Menu

Repository navigation

[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_repeat and array_insert codegen - #57832

[SPARK-58631][SQL] Fix inconsistent array size limit errors in array_repeat and array_insert codegen#57832
SEPURI-SAI-KRISHNA wants to merge 4 commits into
apache:masterfrom
SEPURI-SAI-KRISHNA:array-size-limit-codegen-errors

SEPURI-SAI-KRISHNA commented Aug 7, 2026

Uh oh!

uros-b left a comment

Uh oh!

SEPURI-SAI-KRISHNA commented Aug 7, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 6, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

uros-b commented Sep 9, 2026

Uh oh!

uros-b left a comment

Uh oh!

uros-b commented Sep 9, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

uros-b commented Sep 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SEPURI-SAI-KRISHNA commented Aug 7, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

SEPURI-SAI-KRISHNA commented Aug 7, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 6, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

uros-b commented Sep 9, 2026

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

uros-b commented Sep 9, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

SEPURI-SAI-KRISHNA commented Sep 9, 2026

Uh oh!

uros-b commented Sep 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants