Skip to content

Reading 0-d fixed-length bytes arrays (|S<n>) is ~2.5x slower than equally sized uint8 arrays #4464

Closed

Description

@janickm

Reading a scalar (0-d) fixed-length bytes array, as commonly used in zarr format v2 data to store
encoded blobs (e.g. JPEG/PNG images), is significantly slower than reading the same number of bytes from a
1-d uint8 array:

import time
import numpy as np
import zarr

g = zarr.group(zarr_format=2)
blob = g.create_array("blob", data=np.asarray(b"x" * 1_000_000), compressors=None)  # |S1000000, shape ()
u8 = g.create_array("u8", data=np.frombuffer(b"x" * 1_000_000, dtype=np.uint8), compressors=None)

def bench(name, f, n=300):
    f()
    t = time.perf_counter()
    for _ in range(n):
        f()
    print(f"{name:25s} {1e6 * (time.perf_counter() - t) / n:8.1f} us")

bench("0-d |S1000000 blob[()]", lambda: blob[()])
bench("1-d uint8 u8[...]", lambda: u8[...])

zarr-python 3.4.0, numpy 2.5.3, Python 3.12:

0-d |S1000000 blob[()]       1809.7 us
1-d uint8 u8[...]             739.8 us

(zarr-python 2.18 reads the same 0-d array in ~20 us + I/O)

Profiling shows the time is spent copying the payload multiple times:
CpuNDBuffer.__getitem__ (np.asanyarray(self._data.__getitem__(key))), CpuNDBuffer.__setitem__ into the
output buffer, and NDBuffer.as_scalar() (self.as_numpy_array()[()]). For a 0-d fixed-length bytes dtype each of
these creates a new copy of the full item.

A fast path for complete chunk reads of 0-d arrays (returning the decoded chunk directly instead of copying it into a
freshly allocated output buffer) would likely remove most of this overhead.

Activity

  1. d-v-b commented on Oct 1, 2026

    @d-v-b
    Contributor

    thanks for the report and sorry for the regression. your patch idea looks good

  2. janickm commented on Oct 2, 2026

    @janickm
    ContributorAuthor

    thanks for the report and sorry for the regression. your patch idea looks good

    ok great - let me draft a PR for this then

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions