Skip to content

Avoid memory copy of the NumPy buffer when writing to an Icechunk store #2331

Closed

Description

@tomwhite

Problem

When writing a NumPy array to an Icechunk-backed Zarr store a buffer copy of the array is made, which could be avoided.

Proposed Solution

This is the same issue described for obstore in zarr-developers/zarr-python#2972, and the same fix applies here.

Additional context

The extra copy can be seen by running the following script:

Details
# /// script
# requires-python = ">=3.11"
# dependencies = ["zarr==3.3.0", "numcodecs>=0.16.0", "icechunk", "memray", "numpy"]
# ///
import tempfile
import time
from pathlib import Path

import icechunk
import memray
import numpy as np
import zarr
from memray import FileReader

with tempfile.TemporaryDirectory() as tmp:
    profile = str(Path(tmp) / "write.bin")
    repo = icechunk.Repository.create(
        icechunk.local_filesystem_storage(str(Path(tmp) / "repo"))
    )
    session = repo.writable_session("main")

    with memray.Tracker(profile, native_traces=True):
        arr = np.random.default_rng().random((5000, 5000), dtype=np.float32)
        z = zarr.create_array(
            store=session.store,
            shape=arr.shape,
            dtype=arr.dtype,
            chunks=arr.shape,
            compressors=None,
            overwrite=True,
            config={"write_empty_chunks": True},
        )
        z[:] = arr
        del arr
        del z
        time.sleep(1)

    session.commit("write")
    peak = FileReader(profile).metadata.peak_memory
    print(f"peak {peak / 1e6:.1f} MB")

When run it will print

peak 200.6 MB

because arr is copied before being stored. With the fix described above the peak should be around 100MB.

Note: I used Claude Code to write parts of the example script.

Activity

  1. added a commit that references this issue on Aug 23, 2026
    827c4f8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions