Repository navigation
Certain extension points of zarr are not implemented #3433
Description
Zarr is built with extensibility in mind with several extension points defined in the spec. However, zarr-python does not yet implement all these extension points. Notably, chunk grid and chunk key encoding are not extendable yet, and I believe storage transformers aren't even implemented altogether.
For some background: I was interested in extending the chunk key encoder to address issues with filesystems that struggle with a large number of files in a single directory. The default V3 implementation uses / as the dimension separator, but this is not helpful when you have a single large dim. Sharding with large shards provides a solution, but the current implementation requires full shard rewrites upon writing, even when appending.
I couldn't find any concrete plans to add the missing extension points. Are there any such plans?
I could help to try and implement this for chunk key encodings, but also realize that the current setup with the registry and global mutable config is not ideal. #3360
I am curious what your thoughts are.
Activity
d-v-b commentedon Sep 4, 2025
Thanks for writing this issue, It's definitely in scope for the project! The main reason we haven't implemented extensibility for chunk key encodings or chunks grids yet is basically just a lack of time.
Architecturally, I think chunk key encodings should be extensible on the writing side by implementing a new ChunkKeyEncoding, but this will fail on the reading side because we don't have a registry of chunk key encodings. We absolutely should, and it can be really simple -- just a key : value pairing where the keys are names, and the values are chunk key encoding classes. Happy to provide guidance here.
Chunk grids are more complicated. I suspect our indexing machinery currently assumes regular chunking to some extent. The good news is that this absolutely will be changed when we support irregular chunking (and that's a "when", not an "if").
remcofl commentedon Sep 4, 2025
Thank you for the quick response, and for thinking along!
I can indeed confirm that
ChunkKeyEncodingis extensible on the write side. When you read, it will parse the named config inChunkKeyEncoding.from_dict(), which will obviously fail.I have had a look at the repo and think the registry can be implemented in registry.py similar to
zarr.bufferandzarr.ndbuffer.The parsing in
_parse_chunk_key_encoding()andArrayV3Metadata.__init__()needs to be updated to read and construct classes from the registry.Is this the correct direction? Since you mention a lack of time, I am definitely willing to give it a shot if you are in support. But must admit that I am not that experienced with contributing to open-source projects.
d-v-b commentedon Sep 4, 2025
Yes that's the right direction. Add a new registry for chunk key encodings in
registry.py, ensure that the built-in chunk key encodings get added to the registry by adding required code tochunk_key_encodings.py, then add the required logic to the_parse_chunk_key_encoding()function that looks inside the registry. And feel free to ask questions in our developer chat#Zarr if you're stuck on something.d-v-b commentedon Sep 30, 2026
this was closed by #3436