Add Buffer.as_buffer_like method. #2925

Activity

1. tomwhite commented on Apr 21, 2025

   

   tomwhite

   on Apr 21, 2025

   Member

   More actions

   BTW there's a typo in the title; it should be 'Add Buffer.as_bytes_like method'.

   Also, this will be fixed in #2972.
2. kylebarron commented on Apr 21, 2025

   

   kylebarron

   on Apr 21, 2025

   Last edited by kylebarron

   ContributorAuthor

   More actions

   I called it "buffer" like to refer to the buffer protocol, hence the name in collections.abc.Buffer. FWIW I think that's a better name than "bytes like"

   👍React with 👍1Reacted by Tom White
3. carloshorn commented on Sep 23, 2026

   

   carloshorn

   on Sep 23, 2026

   More actions

   I see that the as_buffer_like method now exists. However, I was wondering why the Zarr Buffer itself does not implement the __buffer__ method to become a collections.abc.Buffer.

   Here a simple demonstration:

   class MyBuffer(zarr.core.buffer.cpu.Buffer):
       def __buffer__(self, flags: int) -> memoryview:
           return self.as_buffer_like()
   
   bfr_old = zarr.core.buffer.cpu.Buffer.from_bytes(b"abc")
   bfr_new = MyBuffer.from_bytes(b"abc")
   
   #memoryview(bfr_old)  # raises TypeError
   memoryview(bfr_new)  # works fine

   I know that this is Python >= 3.12, so the as_buffer_like is likely still relevant. In general, I would feel comfortable if the Zarr buffer were also a Python buffer.
4. d-v-b commented on Sep 23, 2026

   

   d-v-b

   on Sep 23, 2026

   Contributor

   More actions

   if the Zarr buffer were also a Python buffer.

   I believe this is impossible if we also want to support generic CUDA memory with the exact same interface, as CUDA memory generally doesn't support the buffer protocol (I think because the buffer protocol assumes immediate access, and CUDA device memory is essentially async? I'm not sure exactly. see cupy/cupy#1532). As some Zarr buffers -- those backed by numpy arrays -- do have a backend that implements the buffer protocol, we could look at a public interface to expose this.

   👍React with 👍1Reacted by Kyle Barron
5. d-v-b commented on Sep 23, 2026

   

   d-v-b

   on Sep 23, 2026

   Contributor

   More actions

   I'm assuming here that the contract of the buffer protocol is that consumers get a view of the underlying memory. copying from CUDA memory to system memory would not maintain that contract. someone who knows better should correct me if this is wrong!