flatten raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Minor
    • 4.4.0
    • Affects Version/s: 4.2.0, 4.3.0, 4.1.3, 4.0.4
    • Component/s: SQL

      Flatten.genCodeForNumberOfElements does not check the result element count
      against ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH, while the interpreted path
      in Flatten.nullSafeEval does. The two paths therefore report different error
      conditions for the same input: the interpreted path raises
      COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION, and codegen reaches
      ArrayData.allocateArrayData and raises the internal _LEGACY_ERROR_TEMP_2176.

      Same pattern as SPARK-58631, which fixed it for array_repeat.

      Priority is Minor because the limit is not cheaply reachable here: flatten
      needs its input arrays to hold that many elements for real, so an executor will
      normally run out of memory first.

            Assignee:
            sepuri sai krishna
            Reporter:
            sepuri sai krishna
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: