Skip to content

Get parsl.serialize.errors.SerializationError when using Parsl htex to run docking and moldesign #155

Closed
#163
Closed
Get parsl.serialize.errors.SerializationError when using Parsl htex to run docking and moldesign#155
#163
Assignees
gpauloski
Labels
bugSomething isn't working
@ClaudiaCumberbatch

Description

@ClaudiaCumberbatch
ClaudiaCumberbatch
opened on Sep 10, 2024

Describe the problem.

Hi. I was trying to use Parsl htex to run docking. Here is my docking.toml file and my command is python3 -m taps.run --config docking.toml.

[app]
name = "failures"
base = "docking"
failure_rate = 0
failure_type = "import"

[app.config]
smi_file_name_ligand = "/work/cse-zhousc/resilient_compute/resilience_test/taiyi/debug/data/docking/dataset_orz_original_1k.csv"
receptor = "/work/cse-zhousc/resilient_compute/resilience_test/taiyi/debug/data/docking/1iep_receptor.pdbqt"
tcl_path = "/work/cse-zhousc/resilient_compute/resilience_test/taiyi/debug/data/docking/set_element.tcl"

[engine.executor]
name = "parsl-htex"
retries = 3

[engine.executor.htex]
max_workers = 32
radio_mode = "filesystem"

[engine.executor.htex.provider]
kind = "LSFProvider"
init_blocks = 1
max_blocks = 1
min_blocks = 1
nodes_per_block = 2
queue = "debug"
scheduler_options = ""
walltime = "00:30:00"


[engine.executor.monitoring]
hub_address = "localhost"
logging_endpoint = 'sqlite:///parsl-runinfo/monitoring.db'
resource_monitoring_interval = 1
hub_port = 55055

But then I was reported errors as follows. Quite long error messages, but I guess the root error is TypeError: cannot pickle '_abc._abc_data' object. This also happened when I tried to run moldesign app.

[2024-09-10 19:41:42.019] ERROR (parsl.dataflow.dflow) > Task 20 failed after 3 retry attempts
Traceback (most recent call last):
  File "/work/cse-zhousc/parsl_resilient/parsl/executors/high_throughput/executor.py", line 646, in submit
    fn_buf = pack_apply_message(func, args, kwargs,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/facade.py", line 58, in pack_apply_message
    b_func = serialize(func, buffer_threshold=buffer_threshold)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/facade.py", line 94, in serialize
    raise result
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/facade.py", line 86, in serialize
    result = method.identifier + b'\n' + method.serialize(obj)
                                         ^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/concretes.py", line 60, in serialize
    return dill.dumps(data)
           ^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 263, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 235, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 394, in dump
    StockPickler.dump(self, obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 487, in dump
    self.save(obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 717, in save_reduce
    save(state)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 887, in save_tuple
    save(element)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 717, in save_reduce
    save(state)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 887, in save_tuple
    save(element)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 692, in save_reduce
    save(args)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 887, in save_tuple
    save(element)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1084, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1698, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 692, in save_reduce
    save(args)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 902, in save_tuple
    save(element)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
    ^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/pickle.py", line 578, in save
    rv = reduce(self.proto)
         ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle '_abc._abc_data' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 323, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 593, in _unwrap_remote_exception_wrapper
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 653, in launch_if_ready
    exec_fu = self.launch_task(task_record)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 747, in launch_task
    exec_fu = executor.submit(executable, task_record['resource_specification'], *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/executors/high_throughput/executor.py", line 649, in submit
    raise SerializationError(func.__name__)
parsl.serialize.errors.SerializationError: Failed to serialize objects for an invocation of function wrapper. Refer https://parsl.readthedocs.io/en/latest/faq.html#addressing-serializationerror

So I searched for this dill error message TypeError: cannot pickle '_abc._abc_data' object and found that it was addressed in dill 0.3.7 so I upgraded dill to this version. But then I was reported:

globus-compute-sdk 2.27.1 requires dill==0.3.6; python_version >= "3.11", but you have dill 0.3.7 which is incompatible.

If ignore this version conflict and run this app again, I got another error related to deserialization.

[2024-09-10 20:13:26.825] ERROR (parsl.dataflow.dflow) > Task 20 failed after 3 retry attempts
Traceback (most recent call last):
  File "/work/cse-zhousc/parsl_resilient/parsl/executors/high_throughput/executor.py", line 483, in _queue_management_worker
    s = deserialize(msg['exception'])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/facade.py", line 130, in deserialize
    result = deserializer.deserialize(body)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/serialize/concretes.py", line 26, in deserialize
    return pickle.loads(body)
           ^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/site-packages/tblib/pickling_support.py", line 26, in unpickle_exception
    inst = func(*args)
           ^^^^^^^^^^^
TypeError: PydanticErrorMixin.__init__() missing 1 required keyword-only argument: 'code'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 323, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/parsl_resilient/parsl/dataflow/dflow.py", line 593, in _unwrap_remote_exception_wrapper
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/work/cse-zhousc/miniconda3/envs/taps-docking2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
parsl.serialize.errors.DeserializationError: Failed to deserialize objects. Reason: Received exception, but handling also threw an exception: PydanticErrorMixin.__init__() missing 1 required keyword-only argument: 'code'

I wonder whether this dill version limit is important. If so, how can we address the _abc._abc_data object pickle problem? If not, what is this PydanticErrorMixin?

How did you install TaPS?

$ git clone [email protected]:ClaudiaCumberbatch/webs.git
# synchronized with the newest TaPS version in this repo

TaPS Version

a7001b3

Python Version

3.11

OS and Platform

x86 linux

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions