Serialization failure in proxy resolve propagates as AttributeError: __wrapped__ #530
Description
Describe the problem.
If there's an error deserializing the target object when resolving a proxy, it raises a strange error.
We ran into this with a flox scaling test.
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
callback(self)
File "/home/jgpaul/workspace/flox/flox/flox/runtime/process/future_callbacks.py", line 38, in all_child_futures_finished_cbk
children_results = [fut.result() for fut in selected_children_futures]
File "/home/jgpaul/workspace/flox/flox/flox/runtime/process/future_callbacks.py", line 38, in <listcomp>
children_results = [fut.result() for fut in selected_children_futures]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed:
Traceback (most recent call last):
File "/home/jgpaul/workspace/flox/flox/flox/jobs/local_training.py", line 67, in __call__
global_model.load_state_dict(module_state_dict)
File "/home/jgpaul/workspace/flox/venv/lib/python3.10/site-packages/lazy_object_proxy/slots.py", line 210, in __getattr__
return getattr(self.__wrapped__, name)
File "/home/jgpaul/workspace/flox/venv/lib/python3.10/site-packages/lazy_object_proxy/slots.py", line 208, in __getattr__
raise AttributeError(name)
AttributeError: __wrapped__
If we modify the task code (https://github.com/nathaniel-hudson/flox/blob/prototyping/flox/jobs/local_training.py#L57-L68) to extract the key from the proxy and use Store.get(key), we get a more detailed serialization error:
Traceback (most recent call last):
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
callback(self)
File "/home/jgpaul/workspace/flox/flox/flox/runtime/process/future_callbacks.py", line 38, in all_child_futures_finished_cbk
children_results = [fut.result() for fut in selected_children_futures]
File "/home/jgpaul/workspace/flox/flox/flox/runtime/process/future_callbacks.py", line 38, in <listcomp>
children_results = [fut.result() for fut in selected_children_futures]
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
globus_compute_sdk.errors.error_types.TaskExecutionFailed:
Traceback (most recent call last):
File "/home/jgpaul/workspace/flox/flox/flox/jobs/local_training.py", line 63, in __call__
store.get(get_key(global_model))
File "/home/jgpaul/workspace/flox/venv/lib/python3.10/site-packages/proxystore/store/base.py", line 411, in get
result = self.deserializer(value)
File "/home/jgpaul/workspace/flox/venv/lib/python3.10/site-packages/proxystore/serialize.py", line 86, in deserialize
return pickle.loads(data)
AttributeError: Can't get attribute 'SmallConvModel' on <module '__mp_main__' from '/home/jgpaul/workspace/flox/venv/bin/process_worker_pool.py'>
This is an error in the script that was executing, but the fact that ProxyStore clobbered the error and propagated it in a weird way is a problem on our end.
Here's a minimal reproducer:
from proxystore.store import Store
from proxystore.connectors.local import LocalConnector
def deserialize(obj: bytes):
raise AttributeError()
with Store('test', LocalConnector()) as store:
x = store.proxy([1, 2, 3], deserializer=deserialize)
print(x)Interestingly, the strange behavior happens with AttributeError but not an Exception.
We need to:
- Catch pickle/cloudpickle errors in serialization methods and reraise as
SerializationError. - Validate serialization errors in
Store.get()get propagated through the factory.
How did you install ProxyStore?
N/AProxyStore Version
v0.6.3
Python Version
3.10
OS and Platform
x86 Linux