Skip to content

Improve viewer performance when handling GPU-backed 3D arrays #2243

Open

Description

@haesleinhuepf

🔬 💻 🚀 Feature

Hi all,

following up earlier discussions (with @sofroniewn, @tlambert03, @GenevieveBuckley, @jni and @VolkerH ), to improve performance when processing 3D image data, I'd like to send GPU-backed 3D arrays into vispy without turning them into a numpy-array on the way.

Motivation

Vispy is based on OpenGL and thus, does the heavy computing on the GPU anyway. If we use GPU-accelerated image processing, we should not transfer image data between CPU and GPU memory while processing and viewing to get maximum performance. Thus, it might make sense to "bypass" napari and making vispy handle GPU-backed arrays. This should be relevant for users of CUDA and OpenCL based image processing libraries such as cupy, pyopencl and pyclesperanto.

Pitch

There might be at least two ways for implementing this:

  • Either we prevent lines like this one where image data is explicitly turned into a numpy array. We could then pass GPU-backed arrays through napari into vispy. Also the thumbnail generation might need to be changed then. This solution has the pre-requesite that vispy accepts CUDA/OpenCL-backed arrays. If so, memory management is on our side.

  • Alternatively, we use the memory management from vispy and get access to GPU-backed array pointers and pass them to the outside

I've no preference for one of the two alternatives though.
Furthermore, we need to make sure that OpenGL is compatible with OpenCL and CUDA. Links provided by @tlambert03 in this image.sc thread appear to me like this should be feasible. We nevertheless might want to offer some convenience layer for this stuff. If we are clever engineers, we could make all this stuff happen in a napari-plugin named e.g. "napari-gpu-backed-layers".

Additional context

To demo the current state, I implemented a small realisitic use-case script that loads 200 MByte 3D data (355x710x213 voxels, float32), segments nuclei and shows intermediate results + final labels in napari. It does some basic time measurements:

# process the image
intermediate = cle.top_hat_box(input, radius_x=5, radius_y=5, radius_z=5)
result = cle.voronoi_otsu_labeling(intermediate, spot_sigma=1, outline_sigma=2)
stopwatch("image processing")

# add it to napari
viewer.add_image(cle.pull(intermediate))
stopwatch("Showing background subtracted input in napari")
viewer.add_labels(result)
stopwatch("Showing the result in napari")

Output:

image processing  takes time:  0.4350242614746094  s
Showing background subtracted input in napari  takes time:  0.5765478610992432  s
Showing the result in napari  takes time:  1.2743096351623535  s

As you can see, processing takes 0.4 seconds while visualizing results takes 1.8 seconds. I assume that bypassing some numpy-conversions could speedup the viewing.

Last but not least: I'm aware this project will take time. As soon as vispy and other libraries work on the same pointers, we will see napari crashing whenever we hit memory limitations. I'm happy to actively support/maintain code in this context. But I'm also clearly saying: My python/napari skill-level doesn't allow me to implement this alone. Nevertheless, I think it's worth to drive down that road and there is a 3D real-time image processing wonderland waiting for us ;-)

image

Activity

  1. haesleinhuepf commented on Feb 10, 2021

    @haesleinhuepf
    ContributorAuthor

    Note to self: Do more detailed identification of bottlenecks using @jni's suggestion: https://napari.org/docs/dev/guides/perfmon.html

  2. tlambert03 commented on Feb 10, 2021

    @tlambert03
    Contributor

    Since ultimately this would be a vispy feature (outside of the convenience functions we might provide), you might also consider posting a quick feature request to vispy. Someone there might also have some good insights as to how feasible this is

  3. haesleinhuepf commented on Feb 10, 2021

    @haesleinhuepf
    ContributorAuthor

    There is some already:
    vispy/vispy#1800
    vispy/vispy#120
    vispy/vispy#57

    Will read more in detail and report back...

  4. added this to the milestone on Feb 11, 2021
  5. GenevieveBuckley commented on Sep 1, 2022

    @GenevieveBuckley
    Contributor

    There's been a short discussion on twitter about this recent work from Deepak, loading zarr arrays directly onto the GPU using kvikIO and into an xarray structure: https://xarray.dev/blog/xarray-kvikio

  6. haesleinhuepf commented on Sep 28, 2022

    @haesleinhuepf
    ContributorAuthor

    Just a follow up of a discussion with @brisvag. There appears movement on vispy side regarding this issue. See this discussion:
    vispy/vispy#1986

    And this code-example might be a good starting point for exploration of how to integrate this into napari:
    https://gist.github.com/keckj/e37d312128eac8c5fca790ce1e7fc437

  7. brisvag commented on Sep 28, 2022

    @brisvag
    Contributor

    I tried very quickly to see how difficult things would be on the vispy side, and I don't think it would be terrible! For textures, there are only 2 places where cupy array would be either converted or mistreated, and in both cases the fix is trivial. with less than 10 lines changed I got the cupy array untouched all the way to the opengl backend. So at least we know the only real hard part is the machinery for interoperability :)

  8. jakirkham commented on Sep 28, 2022

    @jakirkham
    Contributor

    @brisvag, would you have any interest in sending a (rough) PR to vispy?

  9. brisvag commented on Sep 29, 2022

    @brisvag
    Contributor

    Sure; it's very minimal now though: vispy/vispy#2391.

  10. modified the milestones: , on Sep 29, 2022
  11. rixius commented on Sep 25, 2025

    @rixius

    Has any work on this issue been done lately?

    At resolutions like 512x1024x100 @ 15 Hz my CPU can just about handle the amount of data, but pushing them to the GPU seems to be the bottleneck. In the future I will want to shift some of he data processing to the GPU anyway, and moving the data back and forth will make things even worse.

    The ability to pass data as cupy arrays to napari would seem like an obvious solution ("obvious" not meaning "simple to implement").

  12. psobolewskiPhD commented on Sep 25, 2025

    @psobolewskiPhD
    Member

    Good timing @rixius the issue from 2025 by nvidia got bumped this week. looks like a person was assigned, so maybe?
    NVIDIA/cuda-python#241

    @Czaki @brisvag @aganders3 Note that they are looking at OpenGL as the target, as the most universal. Maybe worth one of you more savvy to chime in? It would suck if they implemented this and then we moved to something else!

  13. brisvag commented on Sep 25, 2025

    @brisvag
    Contributor

    I have no more info than those discussions unfortunately. Would love to see this too!

  14. removed this from the milestone on Oct 6, 2025
  15. psobolewskiPhD commented on Sep 7, 2026

    @psobolewskiPhD
    Member

    Another update, on the CUDA front anyways:
    NVIDIA/cuda-python#1608

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or requestperformanceRelates to performancetopic:vispyVispy integration

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions