Repository navigation
Description
Problem
Uncompressed reads incur an extra buffer copy that could be saved (compared to e.g. default zarr storage or obstore).
Minimal reproducer:
# /// script
# requires-python = ">=3.11"
# dependencies = ["zarr==3.3.0", "numcodecs>=0.16.0", "icechunk", "memray", "numpy"]
# ///
import tempfile
import time
from pathlib import Path
import icechunk
import memray
import numpy as np
import zarr
from memray import FileReader
with tempfile.TemporaryDirectory() as tmp:
profile = str(Path(tmp) / "read.bin")
repo = icechunk.Repository.create(
icechunk.local_filesystem_storage(str(Path(tmp) / "repo"))
)
session = repo.writable_session("main")
arr = np.random.default_rng().random((5000, 5000), dtype=np.float32)
z = zarr.create_array(
store=session.store,
shape=arr.shape,
dtype=arr.dtype,
chunks=arr.shape,
compressors=None,
overwrite=True,
config={"write_empty_chunks": True},
)
z[:] = arr
session.commit("write")
del arr
del z
del session
read_session = repo.readonly_session("main")
with memray.Tracker(profile, native_traces=True):
z = zarr.open(read_session.store, mode="r")
out = z[:]
print(out.shape)
del out
del z
time.sleep(1)
peak = FileReader(profile).metadata.peak_memory
print(f"peak {peak / 1e6:.1f} MB")This prints:
peak 300.6 MB
Showing that there are two extra copies of the 100MB array. One of these is due to zarr-developers/zarr-python#2904, but with Zarrista that goes away and there is only one extra copy of the 100MB with Icechunk (zero with obstore).
Proposed Solution
It should be possible to create a pre-allocated buffer to pass into
icechunk/icechunk/src/session.rs
Lines 1304 to 1305 in 224519b
| // TODO: we don't have a way to distinguish if we want to pass a range or not | |
| asset_manager.fetch_chunk(&id, &byte_range).await.inject() |
... but I don't really know! I'm hoping someone with in-depth knowledge of the codebase can fix this.
Additional context
The issue is put in context at https://github.com/tomwhite/memray-array#reads, where the uncompressed column shows an an additional copy for icechunk vs obstore or the default store.
Activity