My application sometimes pickles functions after the module they were imported from has been removed from sys.modules (for reasons we hopefully don't have to get into). Usually dill handles this just fine, but there seems to be a bug when the pickled function is used in a closure which also maintains a reference to the pickled function's __globals__ dictionary. The following code yields an assertion failure with dill 0.3.5.1 (and also the latest version on master; in each case I was using Python 3.9):
import dill
import sys
import other # another module; see below
def make():
func = other.blah
globs = func.__globals__
def inner():
assert globs is func.__globals__
return inner
f = make()
del sys.modules['other']
f() # assertion passes here
d = dill.dumps(f)
f2 = dill.loads(d)
f2() # assertion fails here
where other.py contains the code:
The assertion fails because in the unpickled version of inner, the free variable func is bound to the unpickled version of blah but the free variable globs is bound to a dict which is not the globals dict of func (nor is it the globals dict of other.blah, in fact).
What seems to be happening is that when blah is unpickled, dill calls _create_function with the argument fglobals set to an empty dictionary, so that the line
func = FunctionType(fcode, fglobals or dict(), fname, fdefaults, fclosure)
creates a new dict to be the globals dict of the new function. Then when the "post-processing" updates the cells of inner, it sets the cell for globs to the old dictionary instead of this newly-created one. To confirm this diagnosis, I removed the or dict() from the line above, and changed the "F1" case of save_function to always set globs = globs_copy (i.e. removing the else branch where it sets globs = {'__name__': obj.__module__}), and indeed the problem went away. I'm not sure how to properly fix the issue, though.
My application sometimes pickles functions after the module they were imported from has been removed from
sys.modules(for reasons we hopefully don't have to get into). Usuallydillhandles this just fine, but there seems to be a bug when the pickled function is used in a closure which also maintains a reference to the pickled function's__globals__dictionary. The following code yields an assertion failure withdill0.3.5.1 (and also the latest version onmaster; in each case I was using Python 3.9):where
other.pycontains the code:The assertion fails because in the unpickled version of
inner, the free variablefuncis bound to the unpickled version ofblahbut the free variableglobsis bound to a dict which is not the globals dict offunc(nor is it the globals dict ofother.blah, in fact).What seems to be happening is that when
blahis unpickled,dillcalls_create_functionwith the argumentfglobalsset to an empty dictionary, so that the linecreates a new dict to be the globals dict of the new function. Then when the "post-processing" updates the cells of
inner, it sets the cell forglobsto the old dictionary instead of this newly-created one. To confirm this diagnosis, I removed theor dict()from the line above, and changed the "F1" case ofsave_functionto always setglobs = globs_copy(i.e. removing theelsebranch where it setsglobs = {'__name__': obj.__module__}), and indeed the problem went away. I'm not sure how to properly fix the issue, though.