Repository navigation
BUG: Got Segmentation Fault when use-after-free a ndarray using SharedMemory as its buffer #23305
Description
Describe the issue:
Hi, I encountered a seg fault using ndarray with SharedMemory as its buffer, as discussed in https://discuss.python.org/t/should-sharedmemory-close-be-called-explicitly-or-we-should-just-rely-on-del/24281/1
Reproduce the code example:
import numpy as np
from multiprocessing.shared_memory import SharedMemory
class Block(object):
def __init__(self, nbytes, shape, dtype):
self._nbytes = nbytes
self._shape = shape
self._dtype = dtype
def get_ndarray(self):
shm = SharedMemory(name=self._shm_name)
return np.ndarray(self._shape, dtype=self._dtype, buffer=shm.buf)
def set_ndarray(self, arr):
shm = SharedMemory(create=True, size=self._nbytes)
shm_arr = np.ndarray(self._shape, dtype=self._dtype, buffer=shm.buf)
shm_arr[:] = arr[:]
self._shm_name = shm.name
arr = np.random.rand(1, 100)
block = Block(arr.nbytes, arr.shape, arr.dtype)
block.set_ndarray(arr)
arr_on_shm = block.get_ndarray()
print(f'arr_on_shm: {arr_on_shm}')Error message:
[1] 13006 segmentation fault python3 test_shm_numpy.pyRuntime information:
import sys, numpy; print(numpy.version); print(sys.version)
1.24.2
3.11.1 (main, Dec 23 2022, 09:39:26) [Clang 14.0.0 (clang-1400.0.29.202)]
print(numpy.show_runtime())
WARNING:threadpoolctlnot found in system! Install it bypip install threadpoolctl. Once installed, trynp.show_runtimeagain for more detailed build information
[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}}]
None
Context for the issue:
No response
Activity
knwng commentedon Mar 1, 2023
If I visit the SharedMemory.buf(which is a memoryview of mmap) after shm is released, I'll got a
ValueError: operation forbidden on released memoryview objectexception instead a seg fault. But somehow this kind of guard doesn't work in ndarray. Is it possible for ndarray to check the status of memoryview before visiting?seberg commentedon Mar 6, 2023
For what its worth
np.frombuffershould protect you here, maybe that should be expanded tonp.ndarray.knwng commentedon Mar 6, 2023
Hi @seberg , I tried to use
np.frombufferand the seg fault disappeared. But when the SharedMemory is destroyed, the__del__method raise a BufferError that can't be caught.The code is:
The message printed:
It's great to avoid the seg fault but the BufferError seems to be another problem. I'm not sure if it can be eliminated.
seberg commentedon Mar 6, 2023
Maybe you should open it as a shared memory bug. NumPy cannot do more than
memoryviewand that is what you are seeing there. So if that is an issue, it is aSharedMemoryissue, not a NumPy one.Maybe it is just unintended use, I guess, but either way it isn't NumPy. The only thing that may be plausible would be to have a numpy helper to create a shared memory array.
knwng commentedon Mar 7, 2023
I was also struggling to classify this problem so at the beginning I reported an issue to cpython. But as @sobolevn said, I can't create a reproducable code without numpy since a normally released memoryview will through a catchable ValueError. Maybe it needs some cooperations between the two projects to determine whether this should be fixed and how?
seberg commentedon Mar 7, 2023
The segfault is NumPy of course (there is historic context for this, but that doesn't matter much).
But you can reproduce the the same thing as
frombufferby simply replacing it withmemoryview.