I am trying to understand how multiprocessing.shared_memory.SharedMemory works. I tried to run the second example from https://docs.python.org/3/library/multiprocessing.shared_memory.html – but it does not seem to work as advertised:
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32 >>> # In the first Python interactive shell >>> import numpy as np >>> a = np.array([1, 1, 2, 3, 5, 8]) # Start with an existing NumPy array >>> from multiprocessing import shared_memory >>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes) >>> # Now create a NumPy array backed by shared memory >>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf) >>> b[:] = a[:] # Copy the original data into shared memory >>> b array([1, 1, 2, 3, 5, 8]) >>> type(b) <class 'numpy.ndarray'> >>> type(a) <class 'numpy.ndarray'> >>> shm.name 'wnsm_e3abbd9a'
So far, so good. However, the problem arises when I try to access this shared array, either in the same or a new Python shell on the same machine:
>>> # In either the same shell or a new Python shell on the same machine >>> import numpy as np >>> from multiprocessing import shared_memory >>> # Attach to the existing shared memory block >>> existing_shm = shared_memory.SharedMemory(name='wnsm_e3abbd9a') >>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example >>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf) >>> c array([ 4294967297, 12884901890, 34359738373, 0, 0, 0], dtype=int64)
This clearly is not the array that was originally shared. Note that I just copy-pasted the example straight from the documentation, only changing the name of the shared memory block. Interestingly, the same thing happens even if I don’t create the array “b” or copy “a” into it before switching to the second Python shell.
Finally, changing the last element of the array in the second shell works as normal:
>>> c[-1] = 888 >>> c array([ 4294967297, 12884901890, 34359738373, 0, 0, 888], dtype=int64)
But it does not affect the original array in the first shell:
>>> # Back in the first Python interactive shell, b reflects this change >>> b array([1, 1, 2, 3, 5, 8])
Does anyone know why this is happening, or what I (along with the official documentation) am doing wrong?
Found the answer here: https://bugs.python.org/issue39655
The default dtype for ndarray in Windows is int32, not int64. Example works after changing this.
(Nice of the devs not to mention this in the documentation, despite this issue being submitted as a bug and closed.)