Dill: After import, unpickling yields segmentation fault; what possible things could be going wrong?

Recently, I upgraded to Big Sur on a Macbook Pro with a python3.9 brew install. I use a virtual environment for developing.

I have a dill-pickled item that I have repickled with a fresh vm on this machine. Prior to importing anything, I can unpickle the item:

Python 3.9.6 (default, Jun 29 2021, 06:20:32)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> with open('item.pkl','rb') as pklstream:
...     item = dill.load(pklstream)
...
>>> item
<Foo object at 0x10d02eac0>
>>> quit()


(venv) $ python3
Python 3.9.6 (default, Jun 29 2021, 06:20:32)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> from skimage.morphology import thin
>>> with open('item.pkl','rb') as pklstream:
...     item = dill.load(pklstream)
...
Segmentation fault: 11
(venv)$

This happens for skimage and for some unrelated local modules. The best I can tell is that there is some underlying dependency that is changing the way pickle works when it is imported in this system.

I don’t understand the degrees of freedom here — what could be changing? What is impossible to change?

Where could this error be arising?


Since everything on the Mac side is more or less a custom install via a Brew, it is hard for me to perfectly re-install and re-cache everything.

Perhaps there are some settings I could set explicitly that would override whatever implicit behavior is incurred by the imports?

Answer

I’m the dill author. What I expect is happening is that dill is writing to the pickle dispatch table (to register new object types it understands how to pickle), and skimage is also doing the same… and is replacing one of the dependencies of your serialized object with something incompatible. So, depending on the order either package is first imported, it will result in a different pickle registry state (it’s a dict of types mapped to serialization functions).

dill attempts to give you a workaround for this, with dill.extend. So, you can remove dill-registered types from the registry with dill.extend(False), and then (re-)inject them with dill.extend(True).

I can’t be certain this will fix what you are seeing, as you don’t have a self-contained working example — but this is the only thing that makes sense to me to be behind what is happening.

It’s also possible to avoid some of these issues, for objects that you know to have a conflict in the registry… if you can use one of the dill serialization variants before you dump the object in question. So, setting dill.settings['recurse'] = True, for example, may also store your object, but will not have the same serialization dependencies, and thus might not be affected by this same conflict in the registry.