Repository navigation
Description
This issues describes a concept for a Zarr v3 Storage Transformer to enable generic indirection between the Zarr keys and the name of the underlying objects in a store. It is not a new idea (see below) but this design is meant to cover a broader set of use cases.
Goals
- Enable content-addressable storage schemes (see Content-addressable storage transformer (v3 protocol extension) #82 for early proposal)
- Enable stores that reference bytes created outside Zarr (e.g. Kerchunk)
- Enable static snapshots of stores (Beyond consolidated metadata for V3: inspiration from Apache Iceberg #154)
- Enable concatenating of multiple arrays without copying any chunk keys (Refactor MultiZarrToZarr into multiple functions fsspec/kerchunk#377 (comment))
- Enable creating Zarr stores that are a mix of "reference" arrays (i.e. Kerchunk) and native Zarr arrays
Design
There has been a lot written on this subject already (see issues linked above) so I'm going to attempt to jump straight into the design. The key difference between this design and prior proposals is that the manifest will be local to the Array. The reason for this is to increase the scalability, portability, and composability of the manifest concept.
Store layout
The manifest store layout will resemble that of a regular Zarr V3 store. Consider the following directory store representation:
a/zarr.json <- group metadata
a/foo/zarr.json <- array metadata
a/foo/manifest.json <- array manifest
...
b/baz/zarr.json <- array metadata
b/baz/c/1/1 <- "regular" chunk
...
Note: array a/foo is a manifest array but array b/baz is a regular zarr array.
Array metadata
Manifest style arrays will need to declare a storage transformer configuration:
{
"node_type": "array",
...
"storage_transformers": [
{
"name": "chunk-manifest-json",
"configuration": {
"manifest": "./manifest.json"
}
}
]
}Note: the small manifests could also be inlined directly into the array metadata object.
Manifest object
In my example above, the array a/foo includes a manifest object (a/foo/manifest.json) which will store the mapping of chunk keys to keys in the store:
{
"0.0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
"0.0.1": {"path": "s3://bucket/foo.nc", "offset": 200, "length": 100},
"0.1.0": {"path": "s3://bucket/foo.nc", "offset": 300, "length": 100},
"0.1.1": {"path": "s3://bucket/foo.nc", "offset": 400, "length": 100},
}path would be the only required key, offset/length/checksum/etc could all be added keys to a) inform the store how to fetch bytes from the chunk or b) provide the store with additional metadata about the chunk.
Note 1: Kerchunk also supports inline data in place of the path. That could also be supported here.
Note 2: I'm using JSON as a manifest type here, but many other options exist, including Parquet or even Zarr arrays.
Concatenating arrays:
Edit: Feb 6 7:20p PT - After thinking about this more, I'm beginning to think serialization of concatenated arrays is a trickier problem than should be addressed in the initial iteration here. The main tricky bit is how to combine arrays with compatible dtypes/shapes/chunks but with differing codecs. Details from my original ideas below but consider this redacted from the proposal for now.
Details
One of the goals above is to enable concatenating multiple Zarr arrays. The manifest approach supports a zero-copy way to achieve this. The concept here closely resembles the approach from [Kerchunk's MultiZarrToZarr](https://fsspec.github.io/kerchunk/tutorial.html#combine-multiple-kerchunked-datasets-into-a-single-logical-aggregate-dataset), except it targeting individual arrays and could be made to work with any zarr arrray (not just Kerchunk references). The idea is that concatenating arrays can be done in Zarr, provided a set of constraints are met, by simply rewriting the keys. Implementations could provide a API for doing this concatenation like:
arr_a: zarr.Array = zarr.open(store_a, path='foo') # shape=(10, 4, 5), chunks=(2, 4, 5)
arr_b: zarr.Array = zarr.open(store_b, path='bar') # shape=(6, 4, 5), chunks=(2, 4, 5)
arr_ab: zarr.Array = zarr.concatenate([arr_a, arr_b], axis=0, store=store_c) # shape=(16, 4, 5), chunks=(2, 4, 5)In this example, zarr.concatenate would act similar numpy.concatenate, returning a new zarr.Array object after creating the new manifest in store_c. This could also be done in two steps by adding a save_manifest method to the Zarr arrays.
Possible extensions
I've tried very hard to keep the scope of this as small as possible. There are currently few v3 storage transformers to emulate so I think the best next step is to try out this simple approach before spending too much time on a spec or elaborating on future options. That said, there are some obvious ways to extend this:
- Supporting writes to manifest arrays (possible, there are many edge cases to consider)
- Enable content addressable storage by hashing keys during writes
- Support non-JSON manifests (many options)
Props
🙌 to those that have done a great job pushing this subject forward already: @martindurant, @alimanfoo, @rabernat among others.
Activity
[-]Manifest storage transformer (v3 storage transformer)[/-][+]Manifest storage transformer[/+]rabernat commentedon Feb 7, 2024
In this proposal, what type of thing is
arr_ab?jhamman commentedon Feb 7, 2024
All three are
zarr.Arrays. I'll add some clarification.rabernat commentedon Feb 7, 2024
Can we write to
zarr_ab?zarr_ab[0, 0] = 1?rabernat commentedon Feb 7, 2024
I think we should seriously consider a much lighter-weight concatenation method. What about just storing references to
store_aandstore_b, rather than duplicating the whole manifest? Basically how ncml works.The advantages of this are that
The metadata doc would somehow contain pointers to the other metadata docs. Something like
The one part I can't quite see is how to do the references to the arrays. Some sort of URL syntax? Absolute vs. relative paths?
rabernat commentedon Feb 7, 2024
Another way of putting it is that I think perhaps "chunk manifest" and "virtual concatenation of Zarr arrays" should be completely separable and orthogonal features.
martindurant commentedon Feb 7, 2024
Note that the kerchunk method and its child here already allow for content-addressable storage, e.g., IPFS. Not sure if you meant something beyond that. There has been chatter elsewhere of chunk checksums and such (stored in metadata, not the bytes of the chunk).
For the concatenation, I would want special attention paid to the multi-dimension case. Also, some consideration of groups-of-arrays which are concatenated together would be nice, but you might say that this is an xarray concern. Are you at all considering that the array chunk grid not aligning with the chunks?
Do I understand that you imagine an output metadata structure of the main "these are the arrays" and then a JSON for each of the target arrays? Or do you end up concatenating the reference lists somewhere along the way?
One important possible extension to consider along with those given - after a prototype is established - is that we now have a way to pass per-chunk information (analogous to the "context" I fought for), and so can have different behaviours for each chunk, like a different zero point in offset-scale filtering.
jhamman commentedon Feb 7, 2024
I've come around on this, but not for exactly the same reason. I've now redacted my original proposal which was not 100% thought though.
Certainly some parallels here but this could be done without IPFS. @alimanfoo's proposal in #82 is still a good read, despite using some now-outdated vernacular.
Again, I'm going to remove this from the proposal. But I'll just say that there are some parallels with @d-v-b's proposal to "fix zarr-python's slicing" (zarr-developers/zarr-python#1603, zarr-developers/zarr-python#980) - namely the creation of a lazy Zarr Array or ArrayView that wraps one or more Zarr array. If we take serialization off the table for now, we can think of this outside the spec conversation and explore how to address this at the implementation level.
I was thinking of concatenating the references but have walked this back because you have to enforce that all array metadata is equivalent (e.g. codecs) for all concatenated arrays. @rabernat is suggesting another approach with could work to resolve those concerns.
jbms commentedon Feb 7, 2024
This is very similar to the kerchunk Reference File System format but is not exactly the same JSON format:
https://fsspec.github.io/kerchunk/spec.html
There are also at least a few implementations of the kerchunk json format outside of kerchunk itself:
Would it be advantageous to use exactly the same format?
martindurant commentedon Feb 7, 2024
Can you please put references? They might be useful for inspiration.
jbms commentedon Feb 7, 2024
I updated my comment to include one other known implementation.
jbms commentedon Feb 7, 2024
@martindurant Is there a document that describes the kerchunk parquet format?
martindurant commentedon Feb 7, 2024
No, but I could make one.
jbms commentedon Feb 7, 2024
While we can all assume what
s3://means, in order for this to be fully specified, we also need to specify the meaning of the URLs. See zarr-developers/zeps#48 for one proposal regarding URLs I created, but something more limited could also suffice.jbms commentedon Feb 7, 2024
Another issue to consider is the Confused deputy problem: user A might think they are writing to "s3://someone-elses-bucket/path" but actually end up writing with user A's credentials to "s3://user-a-private-bucket/other/path". Similarly, user A may think they are exposing "s3://someone-elses-bucket/path" over an HTTP server but actually end up sharing data from "s3://user-a-private-bucket/other/path" or "file:///etc/passwd".
25 remaining items
TomNicholas commentedon Oct 20, 2024
So elephant in the room: Now that the Icechunk Spec is out (even if some of the details are still being settled), is still there much of a use case for the json-based non-versioned chunk manifests as suggested above?
elyall commentedon Nov 8, 2024
Here is an example use case for a OME-NGFF spec'd zarr. I began working through the Icechunk example workflow but VirtualiZarr needs some fixes and better test coverage for TIFFs (opened first issue here). I have limited bandwidth but I can help with that.
For Icechunk am I understanding correctly that the data would still be accessible following the OME-NGFF spec but on disk the files would exist in the Icechunk spec? I'm sure some in the OME community would push back on that, especially since raw microscopy data should never be changed invalidating the main value add of Icechunk. But I'm happy if I just have something that works.
jhamman commentedon Nov 8, 2024
As the author of this proposal and one of the creators of Icechunk, I can say that Icechunk is where I'm going to be putting my focus. The design of Icechunk was motivate by many of the same needs described above (and more!). That doesn't mean we have to abandon this as an idea, just that I don't have plans to peruse it. I know some folks have build some prototypes around this concept so I'm happy to leave this open.
@elyall - I think you've got one detail wrong. If you are using Icechunk to store virtual references (from Kerchunk or VirtualiZarr), Icechunk will just store the pointers to your data. The original files (tiff, etc) will remain unchanged.
tasansal commentedon Apr 15, 2025
If we wanted serverless access with functionality like this PR proposes, how can we do that without Icechunk?
TomNicholas commentedon Apr 15, 2025
I'm not sure what you mean exactly. If you want to read non-zarr data through the Zarr API you will be able to do that through VirtualiZarr's
ManifestStore. But that only holds virtual references in memory, it doesn't persist them. To persist them currently your only options are to write them as Kerchunk references, or write them to Icechunk. This PR proposes another serialization format to persist the references, but now that Icechunk exists there's far less reason to build that format.What exactly are you looking for that prevents you from using Icechunk?
tasansal commentedon Apr 15, 2025
Good questions. Let me explain our workflow. We are very open to suggestions, this is a blocker for us right now.
We work with a large-scale, distributed, in-house data streaming platform (up to hundreds of thousands of cores pushing PB/s of data). We are trying to integrate our Zarr-based (github.com/TGSAI/mdio-python) as a data read/write format in C++ with tensorstore. The system is Task-based, similar to Dask. Since it's streaming, we don't know our chunk grid ahead of time, and it could be formed based on incoming data. Each task is approx. equivalent to a chunk. We know the chunk local coordinates etc, but we don't know the global Zarr grid we want to write into.
Our workaround is to write into a Zarr with worker and task IDs as first two dimensions, which works fine, but we lose all the nice Zarr's indexing capabilities and have to search coordinates to retrieve anything after writing.
If this functionality existed, we could write the chunks with keys using a random hash (or whatever else), and then the mapping from
i/j/k -> hashcould be built at the end, and Zarr arrays could be used like regular Zarr arrays.E.g. we can build this map once the file is written out to read later. This is actually precisely how our legacy file format does. Write all streaming data with chunk locality information and build this map at the end to relate to master chunk grid (based on outputs of all tasks). I am assuming this map can be serialized into JSON or another optimized binary format as a part of the spec, and is not very dynamic (like version controlling chunks)
{ "0.0.0": {"path": "gs://bucket/foo.zarr/data/d465317c9c71"}, "0.0.1": {"path": "gs://bucket/foo.zarr/data/27e63d44b317"}, "0.1.0": {}, "0.1.1": {"path": "gs://bucket/foo.zarr/data/a598dab88862"} }I understand
Icechunkdoes something similar for version control; however, in our use case we are unable to depend on a service to pass PB/s data through. If we could do this serverless, from each worker, then it would be amazing. Which is what this PR proposes, no?d-v-b commentedon Apr 15, 2025
icechunk is serverless
tasansal commentedon Apr 15, 2025
Maybe I should have said "serviceless". We want to be able to do this Zarr native without running a sidecar service.
tasansal commentedon Apr 15, 2025
Re-read your message, and yes, we would like to persist them and have a native way to read it with Zarr. Will
zarr-pythonortensorstoreever be able to read a Zarr with persisted Icechunk or FSSpec References?martindurant commentedon Apr 15, 2025
Zarr already can read with kerchunk references, without kerchunk being installed. To read with icechunk or vzarr, you will need those installed.
tasansal commentedon Apr 15, 2025
Thanks @martindurant, this is exactly what I mean. Since we are reading from C++ with tensorstore, using VirtualiZarr or Icechunk is out of the question. Even the nice fsspec ecosystem doesn't exist there and
tensorstoreimplements some of the drivers.If Zarr defined this serialization (or recommended a common/existing serialization) and made it a spec extension etc; we could implement it in all languages to read the same files x-platform.
martindurant commentedon Apr 15, 2025
if it were part of zarr, and all languages implement it yes - but the implementation would need to be made nevertheless.
Interestingly, GDAL is implementing this, if you want to copy a C implementation OSGeo/gdal#12099
TomNicholas commentedon Apr 15, 2025
Well yeah, because Icechunk is an alternative to Tensorstore here.
This is misleading, because you do need to install
fsspecto read kerchunk reference files. However you do it, there are always at least 3 layers here - the on-disk format, the thing that reaches out to remote storage, and the zarr-python API. To read icechunk's format you need the icechunk client and zarr-python, to read kerchunk's format you need fsspec and zarr-python.@tasansal it's unclear to me if your use case actually requires serializing this mapping, which is what this issue is ultimately about. If you don't, and you have access to the the mapping somehow at the time that you're ready to read chunk data using the zarr API, then you could simply construct a virtualizarr
ManifestStorecontaining the mapping in-memory and read from that. If you need to persist the mapping then you need either kerchunk/icechunk or the as-yet-nonexistent zarr proposal here.TomNicholas commentedon Apr 15, 2025
This also does not make sense. There is no sidecar service that Icechunk runs.
EDIT: Note @tasansal you might be thinking of ArrayLake v1, which did rely on an earthmover-run service. Icechunk is very different.
martindurant commentedon Apr 15, 2025
well it did specifically say "with fsspec"; also, fsspec is very likely to be already installed. You are still right, but