Repository navigation
IPC tests do not exit cleanly #1074
Description
After merging #1020, I see locally that the tests generate the following output with pytest -rA:
______________________________________________________ TestIPCSharedAllocationHandleAndBufferDescriptors.test_main _______________________________________________________
-------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
Error in sys.excepthook:
Original exception was:
Error in sys.excepthook:
Original exception was:
________________________________________________________ TestIPCSharedAllocationHandleAndBufferObjects.test_main _________________________________________________________
-------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
Error in sys.excepthook:
Original exception was:
Error in sys.excepthook:
Original exception was:
__________________________________________________________________________ test_object_passing ___________________________________________________________________________
-------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
Traceback (most recent call last):
File "cuda/core/experimental/_memory.pyx", line 82, in cuda.core.experimental._memory.Buffer._shutdown_safe_close
File "cuda/core/experimental/_memory.pyx", line 877, in cuda.core.experimental._memory.DeviceMemoryResource.deallocate
File "cuda/core/experimental/_utils/cuda_utils.pyx", line 75, in cuda.core.experimental._utils.cuda_utils._check_driver_error
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Exception ignored in: 'cuda.core.experimental._memory.Buffer.__del__'
Traceback (most recent call last):
File "cuda/core/experimental/_memory.pyx", line 82, in cuda.core.experimental._memory.Buffer._shutdown_safe_close
File "cuda/core/experimental/_memory.pyx", line 877, in cuda.core.experimental._memory.DeviceMemoryResource.deallocate
File "cuda/core/experimental/_utils/cuda_utils.pyx", line 75, in cuda.core.experimental._utils.cuda_utils._check_driver_error
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Error in sys.excepthook:
Original exception was:
@Andy-Jost is this a symptom of #1063?
Activity
Andy-Jost commentedon Oct 3, 2025
I've seen this consistently when passing
-sto pytest. It seems related to the output redirection. Very difficult to debug, as many things that should work do not after the unraisable exceptions begin.leofang commentedon Oct 4, 2025
@Andy-Jost I played around with it. I think we see
CUDA_ERROR_INVALID_VALUEfrom thetest_object_passingtest because there is a user/test bug. The test was written such that we have a triple free (cudaFreeAsynccalled 3 times)cuda-python/cuda_core/tests/memory_ipc/test_serialize.py
Line 149 in b09d7ed
since all 3 buffers wrap the same memory. This is not going to work. I don't think this needs to be fixed before the 0.4.0 release because the implementation seems to be alright.
Andy-Jost commentedon Oct 7, 2025
This looks like a driver problem to me. Successful calls to
cuMemPoolImportPointercorrectly matched with calls tocuMemFree(Async)ought to work."Successful calls to
mallocmust be matched by calls tofree" is a bedrock principle, and generalizes to this scenario.leofang commentedon Oct 8, 2025
@Andy-Jost please kindly summarize what we discussed internally on the triple free problem in
test_object_passing, when you get a chance 🙂FWIW, all other stderr noises seem to be fixed by #1104 & #1105.(Update: nope, I misread)