[Python] Pickling a sliced array serializes all the buffers

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical
    • 11.0.0
    • Affects Version/s: None
    • Component/s: Python
    • None

      If a large array is sliced, and pickled, it seems the full buffer is serialized, this leads to excessive memory usage and data transfer when using multiprocessing or dask.

      >>> import pyarrow as pa
      >>> ar = pa.array(['foo'] * 100_000)
      >>> ar.nbytes
      700004
      >>> import pickle
      >>> len(pickle.dumps(ar.slice(10, 1)))
      700165
      
      NumPy for instance
      >>> import numpy as np
      >>> ar_np = np.array(ar)
      >>> ar_np
      array(['foo', 'foo', 'foo', ..., 'foo', 'foo', 'foo'], dtype=object)
      >>> import pickle
      >>> len(pickle.dumps(ar_np[10:11]))
      165

      I think this makes sense if you know arrow, but kind of unexpected as a user.

      Is there a workaround for this? For instance copy an arrow array to get rid of the offset, and trim the buffers?

            Assignee:
            Clark Zinzow
            Reporter:
            Maarten Breddels
            Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: