Skip to content

Uncompressed reads incur an extra buffer copy #2345

Open

Description

@tomwhite

Problem

Uncompressed reads incur an extra buffer copy that could be saved (compared to e.g. default zarr storage or obstore).

Minimal reproducer:

# /// script
# requires-python = ">=3.11"
# dependencies = ["zarr==3.3.0", "numcodecs>=0.16.0", "icechunk", "memray", "numpy"]
# ///
import tempfile
import time
from pathlib import Path

import icechunk
import memray
import numpy as np
import zarr
from memray import FileReader

with tempfile.TemporaryDirectory() as tmp:
    profile = str(Path(tmp) / "read.bin")
    repo = icechunk.Repository.create(
        icechunk.local_filesystem_storage(str(Path(tmp) / "repo"))
    )

    session = repo.writable_session("main")
    arr = np.random.default_rng().random((5000, 5000), dtype=np.float32)
    z = zarr.create_array(
        store=session.store,
        shape=arr.shape,
        dtype=arr.dtype,
        chunks=arr.shape,
        compressors=None,
        overwrite=True,
        config={"write_empty_chunks": True},
    )
    z[:] = arr
    session.commit("write")
    del arr
    del z
    del session

    read_session = repo.readonly_session("main")
    with memray.Tracker(profile, native_traces=True):
        z = zarr.open(read_session.store, mode="r")
        out = z[:]
        print(out.shape)
        del out
        del z
        time.sleep(1)

    peak = FileReader(profile).metadata.peak_memory
    print(f"peak {peak / 1e6:.1f} MB")

This prints:

peak 300.6 MB

Showing that there are two extra copies of the 100MB array. One of these is due to zarr-developers/zarr-python#2904, but with Zarrista that goes away and there is only one extra copy of the 100MB with Icechunk (zero with obstore).

Proposed Solution

It should be possible to create a pre-allocated buffer to pass into

// TODO: we don't have a way to distinguish if we want to pass a range or not
asset_manager.fetch_chunk(&id, &byte_range).await.inject()

... but I don't really know! I'm hoping someone with in-depth knowledge of the codebase can fix this.

Additional context

The issue is put in context at https://github.com/tomwhite/memray-array#reads, where the uncompressed column shows an an additional copy for icechunk vs obstore or the default store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions