Repository navigation
Description
🔬 💻 🚀 Feature
Hi all,
following up earlier discussions (with @sofroniewn, @tlambert03, @GenevieveBuckley, @jni and @VolkerH ), to improve performance when processing 3D image data, I'd like to send GPU-backed 3D arrays into vispy without turning them into a numpy-array on the way.
Motivation
Vispy is based on OpenGL and thus, does the heavy computing on the GPU anyway. If we use GPU-accelerated image processing, we should not transfer image data between CPU and GPU memory while processing and viewing to get maximum performance. Thus, it might make sense to "bypass" napari and making vispy handle GPU-backed arrays. This should be relevant for users of CUDA and OpenCL based image processing libraries such as cupy, pyopencl and pyclesperanto.
Pitch
There might be at least two ways for implementing this:
-
Either we prevent lines like this one where image data is explicitly turned into a numpy array. We could then pass GPU-backed arrays through napari into vispy. Also the thumbnail generation might need to be changed then. This solution has the pre-requesite that vispy accepts CUDA/OpenCL-backed arrays. If so, memory management is on our side.
-
Alternatively, we use the memory management from vispy and get access to GPU-backed array pointers and pass them to the outside
I've no preference for one of the two alternatives though.
Furthermore, we need to make sure that OpenGL is compatible with OpenCL and CUDA. Links provided by @tlambert03 in this image.sc thread appear to me like this should be feasible. We nevertheless might want to offer some convenience layer for this stuff. If we are clever engineers, we could make all this stuff happen in a napari-plugin named e.g. "napari-gpu-backed-layers".
Additional context
To demo the current state, I implemented a small realisitic use-case script that loads 200 MByte 3D data (355x710x213 voxels, float32), segments nuclei and shows intermediate results + final labels in napari. It does some basic time measurements:
# process the image
intermediate = cle.top_hat_box(input, radius_x=5, radius_y=5, radius_z=5)
result = cle.voronoi_otsu_labeling(intermediate, spot_sigma=1, outline_sigma=2)
stopwatch("image processing")
# add it to napari
viewer.add_image(cle.pull(intermediate))
stopwatch("Showing background subtracted input in napari")
viewer.add_labels(result)
stopwatch("Showing the result in napari")
Output:
image processing takes time: 0.4350242614746094 s
Showing background subtracted input in napari takes time: 0.5765478610992432 s
Showing the result in napari takes time: 1.2743096351623535 s
As you can see, processing takes 0.4 seconds while visualizing results takes 1.8 seconds. I assume that bypassing some numpy-conversions could speedup the viewing.
Last but not least: I'm aware this project will take time. As soon as vispy and other libraries work on the same pointers, we will see napari crashing whenever we hit memory limitations. I'm happy to actively support/maintain code in this context. But I'm also clearly saying: My python/napari skill-level doesn't allow me to implement this alone. Nevertheless, I think it's worth to drive down that road and there is a 3D real-time image processing wonderland waiting for us ;-)

Activity
haesleinhuepf commentedon Feb 10, 2021
Note to self: Do more detailed identification of bottlenecks using @jni's suggestion: https://napari.org/docs/dev/guides/perfmon.html
tlambert03 commentedon Feb 10, 2021
Since ultimately this would be a vispy feature (outside of the convenience functions we might provide), you might also consider posting a quick feature request to vispy. Someone there might also have some good insights as to how feasible this is
haesleinhuepf commentedon Feb 10, 2021
There is some already:
vispy/vispy#1800
vispy/vispy#120
vispy/vispy#57
Will read more in detail and report back...
GenevieveBuckley commentedon Sep 1, 2022
There's been a short discussion on twitter about this recent work from Deepak, loading zarr arrays directly onto the GPU using kvikIO and into an xarray structure: https://xarray.dev/blog/xarray-kvikio
haesleinhuepf commentedon Sep 28, 2022
Just a follow up of a discussion with @brisvag. There appears movement on vispy side regarding this issue. See this discussion:
vispy/vispy#1986
And this code-example might be a good starting point for exploration of how to integrate this into napari:
https://gist.github.com/keckj/e37d312128eac8c5fca790ce1e7fc437
brisvag commentedon Sep 28, 2022
I tried very quickly to see how difficult things would be on the vispy side, and I don't think it would be terrible! For textures, there are only 2 places where cupy array would be either converted or mistreated, and in both cases the fix is trivial. with less than 10 lines changed I got the cupy array untouched all the way to the opengl backend. So at least we know the only real hard part is the machinery for interoperability :)
jakirkham commentedon Sep 28, 2022
@brisvag, would you have any interest in sending a (rough) PR to vispy?
brisvag commentedon Sep 29, 2022
Sure; it's very minimal now though: vispy/vispy#2391.
rixius commentedon Sep 25, 2025
Has any work on this issue been done lately?
At resolutions like 512x1024x100 @ 15 Hz my CPU can just about handle the amount of data, but pushing them to the GPU seems to be the bottleneck. In the future I will want to shift some of he data processing to the GPU anyway, and moving the data back and forth will make things even worse.
The ability to pass data as cupy arrays to napari would seem like an obvious solution ("obvious" not meaning "simple to implement").
psobolewskiPhD commentedon Sep 25, 2025
Good timing @rixius the issue from 2025 by nvidia got bumped this week. looks like a person was assigned, so maybe?
NVIDIA/cuda-python#241
@Czaki @brisvag @aganders3 Note that they are looking at OpenGL as the target, as the most universal. Maybe worth one of you more savvy to chime in? It would suck if they implemented this and then we moved to something else!
brisvag commentedon Sep 25, 2025
I have no more info than those discussions unfortunately. Would love to see this too!
psobolewskiPhD commentedon Sep 7, 2026
Another update, on the CUDA front anyways:
NVIDIA/cuda-python#1608