Repository navigation
Sharing CUDA memory with OpenGL #5711
Description
In some visualization use cases, it can be helpful to share an existing CUDA buffer with OpenGL to enable fast rendering without copying. One of my colleagues outlined the steps needed for sharing memory with OpenGL in this comment ( vispy/vispy#1986 (comment) ). There's some CUDA functionality that would be nice to use that I don't think is wrapped today (though please correct me if that is wrong). At the simplest level, it would be helpful to have a way to take a CuPy array and convert that into something OpenGL could use.
Reacted by Leo Fang, Keck Jean-Baptiste and Jonathan Chemla
Reacted by Lorenzo Gaifas, Robert Haase and James D. Morris
Activity
leofang commentedon Sep 7, 2021
That's an interesting task that I too have discussed with a former colleague (who worked in Kitware for very long time) for exactly the same goal! For the needed wrappers (ex:
cudaGraphicsResourceGetMappedPointer) they are already available through CUDA Python, so I am not concerned and I'd say let's just use that and not duplicate the efforts.The main challenges, in my understanding, are:
2 might be easy to fix. Is there any progress on the CUDA side regarding 1, @jakirkham? Is there an API that we can use to hand exclusive ownership back and forth between compute and graphics?
jakirkham commentedon Sep 7, 2021
I'm not sure the answer, but maybe we can ask others.
That said, did come across this blogpost from late last year, which seems relevant.
keckj commentedon Oct 2, 2021
I checked both of your solutions, but unfortunately none of them seem to work.
cudaGraphicsGLRegisterBufferis missing,cuda_gl_interop.hfunctions are not wrapped).It would be nice to add this feature in CuPy (by using the CUDA/OpenGL interop API) to offer an alternative to PyCuda.
How to implement a locking mechanism that allows graphics and compute APIs to safely share the same memory segment?
The locking mechanism is provided by
cudaGraphicsMapResources()andcudaGraphicsUnmapResources(). It would be the user responsibility to map resources to CUDA before using the shared buffer in CuPy. Typically, OpenGL resource mapping returns a CUDA pointer that we could wrap in acupy.cuda.UnownedMemoryobject that can later be used to create acupy.ndarray. This could be feasible today if we had access tocudaGraphicsGLRegisterBufferin python.How does this interoperability work in the presence of a memory pool (say CuPy's built-in one or RMM)?
I think that shared buffers can only be allocated in OpenGL and mapped to CUDA, so we can ignore memory pools (OpenGL owns the memory).
leofang commentedon Oct 4, 2021
Hello @keckj, thanks a lot for your comments!
For No.1: I am surprised that CUDA Python didn't expose them. My bad. We are checking internally and will update here.
For No.2: Asked for clarification internally, and I don't think the nvsci* APIs are suitable for this purpose. Specifically, they are typically for Tegra platforms to interact with camera libraries etc. AFAIK they are not distributed as part of the CUDA Toolkit.
The access plan (OpenGL -> CUDA) you described is in line with my understanding (it was a nice learning process for me 🙂). We need to expose CUDA-OpenGL as well as a subset of CUDA-Graphics interop APIs to achieve this need.
leofang commentedon Oct 19, 2021
Update: a future release of CUDA Python will have the needed APIs exposed.
leofang commentedon Jan 19, 2022
Update: CUDA Python 11.6 now exposes the aforementioned OpenGL-CUDA interop APIs.
https://nvidia.github.io/cuda-python/release/11.6.0-notes.html#cuda-python-11-6-0-release-notes
keckj commentedon Jan 19, 2022
I can confirm that it works with CUDA Python 11.6. I made a demo script cupy_gl_interop.py that also provides basic wrappers for ease of use with CuPy.
leofang commentedon Mar 28, 2022
Is there anything else that CuPy should/better do to help the interop?
jakirkham commentedon Mar 28, 2022
There is some registration/unregistration and mapping/unmapping that users need to do to use this API. Ideally that could be handled by something else so that users don't need to do this themselves. Idk whether that makes sense in CuPy or CUDA Python or something else.