-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Minor
-
Affects Version/s: 4.2.0
-
Component/s: SQL
array_repeat and array_insert raise a different error for an array that exceeds the maximum array length depending on whether the expression runs through the interpreted path or through generated code.
array_repeat
eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER (SQLSTATE 54000) codegen -> _LEGACY_ERROR_TEMP_2176
ArrayRepeat.eval checks count against ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH and raises COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER. ArrayRepeat.genCodeForNumberOfElements performs no such check, so the generated code proceeds to the array allocation, which fails with the internal _LEGACY_ERROR_TEMP_2176. Codegen is the default execution path, so the internal error class is what users normally see. The interpreted path was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was not.
array_insert
eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION codegen -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER
The codegen branch taken when pos is not foldable calls createArrayWithElementsExceedLimitError, which hardcodes "parameter" -> toSQLId("count"). The resulting message reads "the value of parameter(s) `count` in the function `array_insert` is invalid", but array_insert(array, pos, value) has no count parameter. Every other site in the expression – all three interpreted ones and the two sibling codegen ones – uses arrayFunctionWithElementsExceedLimitError (COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION). Introduced by SPARK-56567 (4.2.0).
Why this was not caught
The existing test "Elements exceed limit for array_repeat()" in QueryExecutionErrorsSuite runs select array_repeat(1, 2147483647), which is constant-folded and therefore only exercises the interpreted path. The array_insert branch is reachable only with a non-foldable pos, because a foldable positive pos takes a different codegen branch.
Scope
Error reporting only – neither function produces wrong results, and the conditions under which each raises an error are unchanged.
Sequence also reports COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER naming a count parameter it does not have, but it does so from sequenceLength, which both execution paths share. There is no interpreted-vs-codegen inconsistency there, so it is left alone.
- is related to
-
SPARK-59686 round and bround fail for a large rounding scale instead of returning the rounded value
-
- Open
-
-
SPARK-59374 concat raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
-
- Resolved
-
-
SPARK-59375 flatten raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
-
- Resolved
-
- links to