Skip to content

Enable serializing large objects. #412

@robertnishihara

Description

@robertnishihara

Ray currently cannot put large objects in the object store. The script below fails.

import ray
import numpy as np

ray.init()

ray.put(np.zeros(2 ** 25))

It fails with the following message.

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-1-5be56a8502e1> in <module>()
      4 ray.init()
      5 
----> 6 ray.put(np.zeros(2 ** 25))

/Users/rkn/Workspace/ray/python/ray/worker.py in put(value, worker)
   1672     object_id = worker.local_scheduler_client.compute_put_id(
   1673         worker.current_task_id, worker.put_index)
-> 1674     worker.put_object(object_id, value)
   1675     worker.put_index += 1
   1676     return object_id

/Users/rkn/Workspace/ray/python/ray/worker.py in put_object(self, objectid, value)
    505     # Serialize and put the object in the object store.
    506     try:
--> 507       ray.numbuf.store_list(objectid.id(), self.plasma_client.conn, [value])
    508     except ray.numbuf.numbuf_plasma_object_exists_error as e:
    509       # The object already exists in the object store, so there is no need to

error: NotImplemented: Sequence contains too many elements

This is related to #235.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions