This project serves as a bridge between zarrs (Rust) and zarr (zarr-python) via PyO3. The main goal of the project is to speed up i/o (see zarr_benchmarks).
To use the project, simply install our package zarrs from PyPI (which depends on zarr-python>=3.0.0), and run:
import zarr
zarr.config.set({"codec_pipeline.path": "zarrs.ZarrsCodecPipeline"})You can then use your zarr as normal (with some caveats)!
We export a ZarrsCodecPipeline class so that zarr-python can use the class but it is not meant to be instantiated and we do not guarantee the stability of its API beyond what is required so that zarr-python can use it. Therefore, it is not documented here.
At the moment, we only support a subset of the zarr-python stores:
LocalStore(local filesystem)ObjectStore(cloud storage)HTTPFileSystemviaFsspecStore
A NotImplementedError will be raised if a store is not supported.
ZarrsCodecPipeline options are exposed through zarr.config.
Standard zarr.config options control some functionality (see the defaults in the config.py of zarr-python):
threading.max_workers: the maximum number of threads used internally by theZarrsCodecPipelineon the Rust side.- Defaults to the number of threads in the global
rayonthread pool if set toNone, which is typically the number of logical CPUs.
- Defaults to the number of threads in the global
array.write_empty_chunks: whether or not to store empty chunks.- Defaults to false if
None. Note that checking for emptiness has some overhead, see here for more info.
- Defaults to false if
The ZarrsCodecPipeline specific options are:
codec_pipeline.chunk_concurrent_maximum: the maximum number of chunks stored/retrieved concurrently.- Defaults to the number of logical CPUs if
None. It is constrained bythreading.max_workersas well.
- Defaults to the number of logical CPUs if
codec_pipeline.chunk_concurrent_minimum: the minimum number of chunks retrieved/stored concurrently when balancing chunk/codec concurrency.- Defaults to 4 if
None. See here for more info.
- Defaults to 4 if
codec_pipeline.validate_checksums: enable checksum validation (e.g. with the CRC32C codec).- Defaults to
True. See here for more info.
- Defaults to
codec_pipeline.file_handle_cache_size: the capacity of the filesystem store's file handle cache. If nonzero, files are kept open in a least-recently-used cache and reused across partial reads instead of being reopened per read, which cutsopen/stat/closeoperations when many byte ranges are read from the same files, such as partial reads of sharded arrays. This is particularly beneficial on network filesystems (e.g. Lustre, NFS), where each metadata operation is a server round trip.- Defaults to
0(disabled). Only applies to filesystem stores, and has no effect whendirect_iois enabled. - Cached handles are invalidated on writes through this pipeline, but not on modification from anywhere else — and
zarr-pythonitself is such a writer, sinceresize,delete_dirand metadata writes go through its own store. A cached handle can then still read a chunk file that has been deleted. Only enable this while nothing is modifying the array. - The cache is per
Arrayobject, not per process, so comparefile_handle_cache_sizetimes the number of open arrays againstulimit -n. See here for more info.
- Defaults to
codec_pipeline.shard_index_cache_size: the number of shard indexes kept in a least-recently-used cache. If nonzero, a partial read of a shard reuses an index decoded by an earlier read instead of fetching and decoding it again, removing one read per shard per call. Matters most on network filesystems and for repeated small reads of the same shards, such as random-access sampling.- Defaults to
0(disabled). Only sharded arrays have an index to cache. Each entry costs 16 bytes per inner chunk, so size it against the number of shards read repeatedly, not the array size. - Read-only by design: any write through this pipeline empties the cache, and modification from elsewhere is not seen — the same caveat as
file_handle_cache_size. Only enable it while nothing else is modifying the array.
- Defaults to
codec_pipeline.integer_array_indexing: serve reads with a sorted integer-array index (z[[3, 7, 8, 9], :],z[:, cols], 1-Dz.vindex[idx]) throughzarrsinstead of falling back to thezarr-pythonpipeline. A chunk read is described tozarrsas a rectangular subset, and such a selection is a stack of them, so the pipeline emits one per run of consecutive indices.- Defaults to
False. Whether it helps depends on how dense the selection is: each run decodes separately, so several runs inside one inner chunk decode it once each wherezarr-pythondecodes it once in total. That duplication is whatplan_readsremoves, so the two are normally enabled together. Measure before enabling it alone. - Reads only, and only when the indices are sorted and strictly increasing on a single axis. Unsorted or repeated indices, and two or more integer-array axes, still fall back, because
zarr-pythonreorders the output for those and a run's position in the selection is then not its position in the output.
- Defaults to
codec_pipeline.plan_reads: read the shard index directly and issue the byte ranges of exactly the inner chunks the selection needs, instead of asking the sharding partial decoder for one region at a time. Each needed inner chunk is fetched and decoded once however many requested boxes fall inside it.- Defaults to
False. Needsinteger_array_indexingto be useful, and usesshard_index_cache_sizefor the parsed indexes. - Applies only to a chunk whose codec chain is exactly the sharding codec. Any codec wrapping it means the shard bytes are not laid out as the index describes, so the array falls through to the normal path. Unsharded arrays fall through too.
- Units from every shard are issued into one pool, so reads overlap within a shard as well as across shards, and fetching and decoding run on separate pools. Sub-chunk reads are preserved when there is no compressor; a compressor makes the inner chunk atomic.
- Defaults to
codec_pipeline.plan_reads_fetch_threads: threads issuing reads forplan_reads. These block on IO, so they are their own pool rather than sharing the decode pool — a fetch waiting on the byte budget must not occupy the thread that would release it.- Defaults to
0, meaning the size of therayonpool. Raise it where per-request latency dominates. The pool is process-wide and sized on first use.
- Defaults to
codec_pipeline.plan_reads_fetch_byte_budget: ceiling on bytes fetched but not yet decoded. Without one, a read plan covering an array would fetch all of it before the first decode finished.- Defaults to
0, meaning 256 MiB. A read larger than the whole budget is admitted on its own, so a small budget throttles rather than deadlocks.
- Defaults to
codec_pipeline.plan_reads_merge_gap_bytes: merge two inner-chunk reads into one request when the gap between them is at most this many bytes.- Defaults to
0, which merges only reads that touch, so not a byte outside the request is read. That still collapses dense selections into single large reads, because a shard written in one pass lays its inner chunks down contiguously in index order. - Raising it reads unrequested bytes to save requests — a bandwidth-for-latency trade that depends on the storage, which is why it is a knob rather than a default. A merged read never spans more than one shard.
- Defaults to
codec_pipeline.direct_io: enableO_DIRECTread/write, needs support from the operating system (currently only Linux) and file system.- Defaults to
False.
- Defaults to
codec_pipeline.strict: raise exceptions for unsupported operations instead of falling back to the default codec pipeline ofzarr-python.- Defaults to
False.
- Defaults to
For example:
zarr.config.set({
"threading.max_workers": None,
"array.write_empty_chunks": False,
"codec_pipeline": {
"path": "zarrs.ZarrsCodecPipeline",
"validate_checksums": True,
"chunk_concurrent_maximum": None,
"chunk_concurrent_minimum": 4,
"file_handle_cache_size": 0,
"shard_index_cache_size": 0,
"integer_array_indexing": False,
"plan_reads": False,
"plan_reads_merge_gap_bytes": 0,
"plan_reads_fetch_threads": 0,
"plan_reads_fetch_byte_budget": 0,
"direct_io": False,
"strict": False
}
})If the ZarrsCodecPipeline is pickled, and then un-pickled, and during that time one of chunk_concurrent_minimum, chunk_concurrent_maximum, or num_threads has changed, the newly un-pickled version will pick up the new value. However, once a ZarrsCodecPipeline object has been instantiated, these values are then fixed. This may change in the future as guidance from the zarr community becomes clear.
Concurrency can be classified into two types:
- chunk (outer) concurrency: the number of chunks retrieved/stored concurrently.
- This is chosen automatically based on various factors, such as the chunk size and codecs.
- It is constrained between
codec_pipeline.chunk_concurrent_minimumandcodec_pipeline.chunk_concurrent_maximumfor operations involving multiple chunks.
- codec (inner) concurrency: the number of threads encoding/decoding a chunk.
- This is chosen automatically in combination with the chunk concurrency.
The product of the chunk and codec concurrency will approximately match threading.max_workers.
Chunk concurrency is typically favored because:
- parallel encoding/decoding can have a high overhead with some codecs, especially with small chunks, and
- it is advantageous to retrieve/store multiple chunks concurrently, especially with high latency stores.
zarrs-python will often favor codec concurrency with sharded arrays, as they are well suited to codec concurrency.
The following methods will trigger use with the old zarr-python pipeline:
-
Any
oindexorvindexintegernp.ndarrayindexing with dimensionality >=3 i.e.,arr[np.array([...]), :, np.array([...])] arr[np.array([...]), np.array([...]), np.array([...])] arr[np.array([...]), np.array([...]), np.array([...])] = ... arr.oindex[np.array([...]), np.array([...]), np.array([...])] = ...
-
Any
vindexoroindexdiscontinuous integernp.ndarrayindexing for writes in 2Darr[np.array([0, 5]), :] = ... arr.oindex[np.array([0, 5]), :] = ...
-
vindexwrites in 2D where both indexers are integernp.ndarrayindices i.e.,arr[np.array([...]), np.array([...])] = ...
-
Ellipsis indexing. We have tested some, but others fail even with
zarr-python's default codec pipeline. Thus for now we advise proceeding with caution here.arr[0:10, ..., 0:5]
Furthermore, using anything except contiguous (i.e., slices or consecutive integer) np.ndarray for numeric data will fall back to the default zarr-python implementation.
Please file an issue if you believe we have more holes in our coverage than we are aware of or you wish to contribute! For example, we have an issue in zarrs for integer-array indexing that would unblock a lot the use of the rust pipeline for that use-case (very useful for mini-batch training perhaps!).
Further, any codecs not supported by zarrs will also automatically fall back to the python implementation.