[Python] Any replacements for pa.serialize() and pa.deserialize() ?

It seems that PyArrow has deprecated the pa.serialize() and pa.deserialize() methods and suggests to use other options such a Pickle5.

Using Pickle5 doesn't seem to have the same performance as PyArrow's deprecated Serialization method. Is there ANY proper replacements for pa.serialize() and pa.deserialize()?

Here's a simplified code that compares the difference between PyArrow & Pickle when Serializing/Deserializing:

import time

import numpy as np
import pickle5
import pyarrow as pa


class Person:
    def __init__(self, Thumbnail: np.ndarray = None):
        if Thumbnail is not None:
            self.Thumbnail: np.ndarray = Thumbnail
        else:
            self.Thumbnail: np.ndarray = np.random.rand(256, 256, 3)


def serialize_Person(person):
    return {'Thumbnail': person.Thumbnail}


def deserialize_Person(person):
    return Person(person['Thumbnail'])


context = pa.SerializationContext()
context.register_type(Person, 'Person', custom_serializer=serialize_Person, custom_deserializer=deserialize_Person)

PERSONS = [Person() for i in range(100)]

"""
PyArrow
"""
t1 = time.time()
persons_serialized = pa.serialize(PERSONS, context=context).to_buffer()
persons_deserialized = pa.deserialize(persons_serialized, context=context)
t2 = time.time()
print(f'PyArrow Time => {t2 - t1}')

"""
Pickle
"""
t1 = time.time()
persons_pickled = pickle5.dumps(PERSONS, protocol=5)
persons_depickled = pickle5.loads(persons_pickled)
t2 = time.time()
print(f'Pickle Time => {t2 - t1}')

The outputs on my system are:

PyArrow Time => 0.04499983787536621
Pickle Time => 0.2220008373260498

One of the main advantages of the pickle protocol 5 is that it supports out-of-band buffers (https://docs.python.org/3/library/pickle.html#out-of-band-buffers, https://www.python.org/dev/peps/pep-0574/). This means that large buffers (i.e. the actual array data of the numpy arrays in this case) can be handled separately.

Using your example of the Persons class holding a numpy array, and adding a toy example of using out-of-band buffers:

# with pyrarow.(se)serialize
persons_serialized = pa.serialize(PERSONS, context=context).to_buffer()
%timeit pa.deserialize(persons_serialized, context=context)
...
439 µs ± 23.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# with pickle 5
persons_pickled = pickle.dumps(PERSONS, protocol=5)
%timeit pickle.loads(persons_pickled)
57.3 ms ± 2.46 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# with pickle 5 with out-of-band buffers
buffers = []
persons_pickled = pickle.dumps(PERSONS, protocol=5, buffer_callback=buffers.append)
%timeit pickle.loads(persons_pickled, buffers=buffers)
119 µs ± 5.08 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

So here you can see that pyarrow.(de)serialize and pickle with out-of-band buffers are quite similar when deserializing (pickle protocol 5 is actually faster).
Of course, this is also not a fully fair comparison. In the last case with out-of-band buffers, the ndarrays are recreated zero-copy (no copy of the actual array data is made), which is not the case with pyarrow.serialize (which will copy the data when creating the serialized buffer; you can do a similar zero-copy (de)serialization with pyarrow using pa.serialize(..).to_components() / pa.deserialize_components(..)).

But in the end, the question is how you are using this (de)serialization. In your simplified example, you are serializing/deserializing the objects in-memory in the same python process. But that might not be your actual use case (in which case your timings are not necessarily fully representative), but you might be rather saving to a temporary file, or passing through to another process in a multiprocessing context, or...? For example, in such a multiprocessing context, the out-of-band buffers can be communicated more efficiently separately.
In addition, depending on the use case you might not (need to) use pickle directly, but you might be using it through another library. For example if you do multiprocessing via dask, it will already make use such out-of-band buffers for you)

Thanks for the detailed answer. Here's my use case:

I'm serializing the Persons array (containing other attributes some of which aren't numpy array but simple data types) and sending it over some kind of network transport (such as a TCP Socket) and need to deserialize the data in another process on another computer.

If I want to use Pickle 5 for this purpose, should I send both the persons_pickled and buffers objects via the Socket? If that's the case, isn't there a cleaner way to only send one pickled object just like when I was using PyArrow?

Side question: did you check if, when sending it over the network, pyarrow is actually faster compared to plain pickle protocol 5? You might see a smaller difference in that context compared to the simplified, in-memory example above.
I didn't check it over the network, but eg when writing the serialized data to a file, I didn't see any difference.

Unfortunately benchmarking it over the network and considering the two sides would be trickier to pull.

Can you answer my question about sending the pickled data:

If I want to use Pickle 5 for this purpose, should I send both the persons_pickled and buffers objects via the Socket? If that's the case, isn't there a cleaner way to only send one pickled object just like when I was using PyArrow?

The way it goes now, I'm against removing the out-of-band Serialization/Deserialization methods of PyArrow since right now there's no other alternatives with the same performance and ease of use as PyArrow's.

Pickle Protocol 5 needs a buffer for out-of-band Serialization/Deserialization and sending two objects (the serialized object and the buffer) along the network or in a file is ugly.

Pickle Protocol 5 needs a buffer for out-of-band Serialization/Deserialization and sending two objects (the serialized object and the buffer) along the network or in a file is ugly.

It's not ugly, it's the way to do it. If you concatenate the buffers, you get memory copies.

For the record, I get the following numbers here:

pickle5 with copies

>>> %timeit persons_pickled = pickle5.dumps(PERSONS, protocol=5)
39.3 ms ± 389 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> %timeit persons_depickled = pickle5.loads(persons_pickled)
28.9 ms ± 71.8 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

pickle5 with out-of-band buffers

>>> %timeit buffers=[]; persons_pickled = pickle5.dumps(PERSONS, protocol=5, buffer_callback=buffers.append)
231 µs ± 1.13 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>> %timeit persons_depickled = pickle5.loads(persons_pickled, buffers=buffers)
121 µs ± 336 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

PyArrow serialization

>>> %timeit persons_serialized = pa.serialize(PERSONS, context=context).to_buffer()
18.6 ms ± 79.2 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> %timeit persons_deserialized = pa.deserialize(persons_serialized, context=context)
398 µs ± 282 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Summary table

	Serialization	Deserialization
pickle5 with copies	39.3 ms	28.9 ms
pickle5 with out-of-band-buffers	231 µs	121 µs
PyArrow serialization	18.6 ms	398 µs

Short analysis

By default, with pickle you pay the price of memory copies both for serialization and deserialization. PyArrow allows to avoid the price of memory copies for deserialization, but only on the read path. pickle out-of-band buffers avoid memory copies on both sides.

For a more general explanation of why we're deprecating PyArrow serialization, however, I will point out that pickle is the standard way to serialize objects in Python: you can expect many third-party types to be compatible and to serialize cleanly. PyArrow serialization would need a lot of work to support the same breadth of types; currently, it only supports a small subset.

Note that you had to write your own PyArrow serialization callbacks (serialize_Person and deserialize_Person). In that regard, I find it a bit strange that you find pickle less easy to use, while it doesn't require user-defined callbacks.

Thanks for the detailed explanation.

Does anyone have a suggestion for a valid replacement for pa.serialize going forwards which keeps many of the advantages without having to use pickle?

As far as I know Pickle is not secure:
https://docs.python.org/3/library/pickle.html
Warning The pickle module is not secure. Only unpickle data you trust.

This is not just theoretical there are tools allowing exploits to be created.
https://securityboulevard.com/2021/03/never-a-dill-moment-exploiting-machine-learning-pickle-files/

Firstly, arrow seems to perform around the same speed for my use case as Pickle protocol 5 but I am guessing does not include all the insecurity of pickle allowing for arbitrary python ops to be executed. Additionally in my example the Pickle size was orders of magnitude bigger than the arrow size which will make a huge difference on network transfer.

I have dug through the arrow code and it seems that there are some occasional references to pickle but if I understand correctly they are here:
https://github.com/apache/arrow/blob/master/python/pyarrow/serialization.py#L462

serialization_context.register_type(
    type(lambda: 0), "function",
    pickle=True)

serialization_context.register_type(type, "type", pickle=True)

Which implies that for Python classes and functions Pickle gets called.

However it seems like for nearly any other object with a backing .__dict__ it simply uses the _serialize_default_dict which decomposes the dict into keys and values.

It appears in the C++ code there is a recursive serializer which supports all the most of the Python primitives.

https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/serialize.cc

Status SerializeObject(PyObject* context, PyObject* sequence, SerializedPyObject* out) {
  PyAcquireGIL lock;
  SequenceBuilder builder;
  RETURN_NOT_OK(internal::VisitIterable(
      sequence, [&](PyObject* obj, bool* keep_going /* unused */) {
        return Append(context, obj, &builder, 0, out);
      }));
  std::shared_ptr<Array> array;
  RETURN_NOT_OK(builder.Finish(&array));
  out->batch = MakeBatch(array);
  return Status::OK();
}

I haven't read the Pickle source code but the fact that it creates Python bytecode ops is very different and surely can never be considered safe without significant work. While sending functions is cool, and there are situations where pickle is easy and probably fine, it feels like removing this highly efficient recursive serde of primitive python types from the Arrow library would remove a significant benefit for which there is no equally safe alternative.

Does anyone have a suggestion with the same convenience and performance?
Also is there any suggestion of when this will be removed. Arrow 7.0.0 has been released and pa.serialize is still there.

Which implies that for Python classes and functions Pickle gets called.

This means that an attacker can craft a serialization payload which contains a reference to a Python function whose unpickling would execute arbitrary code. So I don't think you are better with off with PyArrow serialization than with pickle itself.

If you need to accept pickles securely you need to authenticate the originating party somehow, for example using a cryptographic signature. Or you can just use TLS if this is on a network connection.

That said, if you are only interested in primitive Python types, you could take a look at msgpack or BSON, for example.

@pitrou

For the case of serializing builtin Python types, or simple classes consisting of primitive types, aside from https://github.com/msgpack/msgpack-python, another project to look at is https://github.com/jcrist/msgspec (it supports msgpack, as @pitrou mentioned, but with a supposedly nice / faster interface).

@pitrou

@pitrou thats a great point, the attacker can simply serialize something which will use the pickle deserializer internally. We currently use protobuf but are looking for something zero copy, faster and supporting much larger sizes, however I think the best thing to do here is just to re-implement our data structure to use the native Arrow datatypes that way the serialization is basically just memory copy right?

Am I correct in understanding that serializing of normal Arrow tensor / table etc types will not use the pa.serialize pathway or pickle anywhere? Is there any good documentation on implementing a custom object type made from Arrow primitives, for example multiple related np.arrays of data and some sidecar meta data possibly of different underlying numeric type and size?

Regarding the existing functionality of pa.serialize I think it would be worth adding a warning either to the function or docs that Pickle is insecure and either accelerating the deprecation or providing an optional parameter which allows the ability to ignore / raise an exception for any pickle types.

@pitrou

@pitrou I have asked a question about how to send objects that have been serialized using Pickle's out-of-band buffers over the network.

I have no idea how to send serialized objects with pickle now that each object is broken down to a serialized object and a list of buffers. Is this even possible or should we try sending them one by one?

@Neltherion

@Neltherion You could indeed send the buffers one by one or send them all at once using sendmsg.

But instead of trying to recreate it yourself I would suggest you take a look at Dask distributed. It has a lot of optimizations for transfer of Numpy- and Arrow-like data already.

@Neltherion

@Neltherion You could indeed send the buffers one by one or send them all at once using sendmsg.

@pitrou Thanks. But is this also doable with ZMQ? It seems ZMQ is only able to send bytearray data types. Since we have a byte-like object (the serialized object) and a pythonic list of buffers, how can we concatenate them together somehow to avoid memory copies and also be able to send the two in one go using ZMQ?

The code over at StackOverflow has a working example to test ideas out.

@Neltherion

@Neltherion That question probably belongs better on a ZMQ forum or bug tracker. I have no idea about which ZMQ APIs would help your use case.

@pitrou

@pitrou It's not about ZMQ: Pickle5 gives us a list of buffers and a serialized object. If we want to send them together, we have to somehow stick them together which results in memory copies, and sending them one by one would result in complexities along the way (packets not getting there, book-keeping the received data and sticking them using pickle.load(), etc...). If there is a way to stick them together in a serialized state, not ZMQ nor any other legacy code would have difficulties but as of now, I haven't found a way to keep these two objects together.

So how are we able to fully use this new feature, without somehow keeping these two different objects separate?

Well, I already suggested sendmsg if you're building your own transmission system. sendmsg doesn't make an intermediate copy of the buffers you give it. If you need a similar primitive from ZMQ, it's a ZMQ question.

Hi,

I have a general question regarding Pyarrow, and this seems to be the best place to discuss it.

Isn't the whole point of Pyarrow to be a leading data interchange format? If we're picking objects and sending them over the network, doesn't that presume the language on the other end is Python? My understanding is that Pyarrow should allow reading data into a Pyarrow object on the other side and handle the mapping for various supported data types.

Some might suggest using Pyarrow Flight RPC or protobuf, but I believe these don't fully solve the use cases because:

Flight RPC doesn't address how to serialize non-DataFrame objects, leading to compatibility issues.
Custom protobuf for every data and collection type adds complexity compared to serializing everything to bytes.
Serializing to disk can cause file locking and concurrency issues when reading or writing constantly.

My use case is that Pyarrow supports many data types, including tensors and dictionaries, which is great. However, I can't serialize them as a data interchange format unless the other end is Python.

Is there a way for common object types like numpy arrays to zero-memory serialize based on their respective Pyarrow objects, or is this not possible?

Thanks.

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.

Navigation Menu

Repository navigation

[Python] Any replacements for `pa.serialize()` and `pa.deserialize()` ? #11239

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] Any replacements for pa.serialize() and pa.deserialize() ? #11239

Description

Activity

jorisvandenbossche commented on Sep 29, 2021

Neltherion commented on Sep 29, 2021

jorisvandenbossche commented on Sep 29, 2021

Neltherion commented on Oct 2, 2021

Neltherion commented on Oct 10, 2021

pitrou commented on Oct 12, 2021

pitrou commented on Oct 12, 2021

pitrou commented on Oct 12, 2021

Neltherion commented on Oct 17, 2021

madhavajay commented on Feb 24, 2022

pitrou commented on Feb 24, 2022

pitrou commented on Feb 24, 2022

jorisvandenbossche commented on Feb 24, 2022

madhavajay commented on Mar 1, 2022

3 remaining items

Neltherion commented on Jan 23, 2023

pitrou commented on Jan 23, 2023

pitrou commented on Jan 23, 2023

Neltherion commented on Jan 23, 2023

pitrou commented on Jan 23, 2023

Neltherion commented on Jan 24, 2023

pitrou commented on Jan 24, 2023

pbower commented on May 21, 2024

github-actions commented on Nov 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Python] Any replacements for `pa.serialize()` and `pa.deserialize()` ? #11239