-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Minor
-
Affects Version/s: 4.2.0, 4.3.0, 4.1.3, 4.0.4
-
Component/s: SQL
Flatten.genCodeForNumberOfElements does not check the result element count
against ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH, while the interpreted path
in Flatten.nullSafeEval does. The two paths therefore report different error
conditions for the same input: the interpreted path raises
COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION, and codegen reaches
ArrayData.allocateArrayData and raises the internal _LEGACY_ERROR_TEMP_2176.
Same pattern as SPARK-58631, which fixed it for array_repeat.
Priority is Minor because the limit is not cheaply reachable here: flatten
needs its input arrays to hold that many elements for real, so an executor will
normally run out of memory first.
- is related to
-
SPARK-59374 concat raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
-
- Resolved
-
- relates to
-
SPARK-58631 array_repeat and array_insert raise inconsistent array size limit errors in codegen and interpreted execution
-
- Resolved
-
- links to