Recently, I upgraded to Big Sur on a Macbook Pro with a python3.9 brew install. I use a virtual environment for developing.
I have a dill-pickled item that I have repickled with a fresh vm on this machine. Prior to importing anything, I can unpickle the item:
Python 3.9.6 (default, Jun 29 2021, 06:20:32) [Clang 12.0.0 (clang-1220.127.116.11)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import dill >>> with open('item.pkl','rb') as pklstream: ... item = dill.load(pklstream) ... >>> item <Foo object at 0x10d02eac0> >>> quit() (venv) $ python3 Python 3.9.6 (default, Jun 29 2021, 06:20:32) [Clang 12.0.0 (clang-118.104.22.168)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import dill >>> from skimage.morphology import thin >>> with open('item.pkl','rb') as pklstream: ... item = dill.load(pklstream) ... Segmentation fault: 11 (venv)$
This happens for
skimage and for some unrelated local modules. The best I can tell is that there is some underlying dependency that is changing the way pickle works when it is imported in this system.
I don’t understand the degrees of freedom here — what could be changing? What is impossible to change?
Where could this error be arising?
Since everything on the Mac side is more or less a custom install via a Brew, it is hard for me to perfectly re-install and re-cache everything.
Perhaps there are some settings I could set explicitly that would override whatever implicit behavior is incurred by the imports?
dill author. What I expect is happening is that
dill is writing to the pickle dispatch table (to register new object types it understands how to pickle), and
skimage is also doing the same… and is replacing one of the dependencies of your serialized object with something incompatible. So, depending on the order either package is first imported, it will result in a different pickle registry state (it’s a dict of types mapped to serialization functions).
dill attempts to give you a workaround for this, with
dill.extend. So, you can remove
dill-registered types from the registry with
dill.extend(False), and then (re-)inject them with
I can’t be certain this will fix what you are seeing, as you don’t have a self-contained working example — but this is the only thing that makes sense to me to be behind what is happening.
It’s also possible to avoid some of these issues, for objects that you know to have a conflict in the registry… if you can use one of the
dill serialization variants before you
dump the object in question. So, setting
dill.settings['recurse'] = True, for example, may also store your object, but will not have the same serialization dependencies, and thus might not be affected by this same conflict in the registry.