Arrow string and binary vectors only support 1 GiB

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 3.4.0
    • Component/s: SQL
    • None

      Since Arrow 10.0.0, BaseVariableWidthVector (the parent for string and binary vectors), only supports expanding up to 1 GiB through the safe interfaces, which Spark uses, instead of 2 GiB previously. This is due to https://github.com/apache/arrow/pull/13815. I added a comment in there but haven't got any responses yet, will make an issue in Arrow as well.

      Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException.

      See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above for how I recreated to show this was now the case.

            Assignee:
            Unassigned
            Reporter:
            Adam Binford
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: