Broken serialization of ROOT objects in python3 with dill #6370
Description
Describe the bug
Serialization of ROOT objects in python using dill is broken for python3, but it works ok for python2.
( dill serialization is vital for the proper paralllel and distributive python calculations using pathos suite. The functionality of
regular paralell python pp and multiprocessing module is very limited due to limitations of pickle`` serialization, and it is a moment when dill/pathos` enters the game, significantly extending the functionality
Expected behavior
Similar behaviour for python2 and python3 is expected
b ut it works for python2 and fails for python3
To Reproduce
to reproduce see this very simple gist
https://gist.github.com/VanyaBelyaev/278074969f2f940f180885be35f04db9
It has been tested with different LCG releases from LCG_94 to LCG_97a
and it always works for pytjon2 and fails for python3
for python3 one gets an error
Traceback (most recent call last):
File "./test_dill.py", line 13, in <module>
hh = dill.loads ( dill.dumps ( h ) )
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
pik.dump(obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 637, in save_reduce
save(func)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 1064, in save_builtin_method
pickler.save_reduce(_get_attr, (module, obj.__name__), obj=obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 774, in save_tuple
save(element)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 1269, in save_module
state=_main_dict)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 859, in save_dict
self._batch_setitems(obj.items())
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 885, in _batch_setitems
save(v)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 859, in save_dict
self._batch_setitems(obj.items())
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 890, in _batch_setitems
save(v)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 819, in save_list
self._batch_appends(obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 846, in _batch_appends
save(tmp[0])
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc9-opt/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type
StockPickler.save_global(pickler, obj)
File "/cvmfs/sft.cern.ch/lcg/releases/LCG_97python3/Python/3.7.6/x86_64-centos7-gcc9-opt/lib/python3.7/pickle.py", line 960, in save_global
(obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class '_pythonization.compose_method.<locals>.composition_pythonizor'>: it's not found as _pythonization.compose_method.<locals>.composition_pythonizor
Setup
- ROOT versions from LCG_94 to LCG_97a
- centos7, lxplus7.cern.ch
- LCG-releases via cvmfs
Additional context
I've opened the issue also for dill project: uqfoundation/dill#356