Skip to content

Manifest storage transformer #287

Open

Description

@jhamman

This issues describes a concept for a Zarr v3 Storage Transformer to enable generic indirection between the Zarr keys and the name of the underlying objects in a store. It is not a new idea (see below) but this design is meant to cover a broader set of use cases.

Goals

Design

There has been a lot written on this subject already (see issues linked above) so I'm going to attempt to jump straight into the design. The key difference between this design and prior proposals is that the manifest will be local to the Array. The reason for this is to increase the scalability, portability, and composability of the manifest concept.

Store layout

The manifest store layout will resemble that of a regular Zarr V3 store. Consider the following directory store representation:

a/zarr.json  <- group metadata
a/foo/zarr.json  <- array metadata
a/foo/manifest.json <- array manifest 
...
b/baz/zarr.json <- array metadata
b/baz/c/1/1 <- "regular" chunk
...

Note: array a/foo is a manifest array but array b/baz is a regular zarr array.

Array metadata

Manifest style arrays will need to declare a storage transformer configuration:

{
  "node_type": "array",
  ...
  "storage_transformers": [
    {
      "name": "chunk-manifest-json",
      "configuration": {
        "manifest": "./manifest.json"
      }
    }
  ]
}

Note: the small manifests could also be inlined directly into the array metadata object.

Manifest object

In my example above, the array a/foo includes a manifest object (a/foo/manifest.json) which will store the mapping of chunk keys to keys in the store:

{
    "0.0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
    "0.0.1": {"path": "s3://bucket/foo.nc", "offset": 200, "length": 100},  
    "0.1.0": {"path": "s3://bucket/foo.nc", "offset": 300, "length": 100},  
    "0.1.1": {"path": "s3://bucket/foo.nc", "offset": 400, "length": 100}, 
}

path would be the only required key, offset/length/checksum/etc could all be added keys to a) inform the store how to fetch bytes from the chunk or b) provide the store with additional metadata about the chunk.

Note 1: Kerchunk also supports inline data in place of the path. That could also be supported here.
Note 2: I'm using JSON as a manifest type here, but many other options exist, including Parquet or even Zarr arrays.

Concatenating arrays:

Edit: Feb 6 7:20p PT - After thinking about this more, I'm beginning to think serialization of concatenated arrays is a trickier problem than should be addressed in the initial iteration here. The main tricky bit is how to combine arrays with compatible dtypes/shapes/chunks but with differing codecs. Details from my original ideas below but consider this redacted from the proposal for now.

Details

One of the goals above is to enable concatenating multiple Zarr arrays. The manifest approach supports a zero-copy way to achieve this. The concept here closely resembles the approach from [Kerchunk's MultiZarrToZarr](https://fsspec.github.io/kerchunk/tutorial.html#combine-multiple-kerchunked-datasets-into-a-single-logical-aggregate-dataset), except it targeting individual arrays and could be made to work with any zarr arrray (not just Kerchunk references). The idea is that concatenating arrays can be done in Zarr, provided a set of constraints are met, by simply rewriting the keys. Implementations could provide a API for doing this concatenation like:

arr_a: zarr.Array = zarr.open(store_a, path='foo')  # shape=(10, 4, 5), chunks=(2, 4, 5)
arr_b: zarr.Array = zarr.open(store_b, path='bar')  # shape=(6, 4, 5), chunks=(2, 4, 5)
arr_ab: zarr.Array = zarr.concatenate([arr_a, arr_b], axis=0, store=store_c)  # shape=(16, 4, 5), chunks=(2, 4, 5)

In this example, zarr.concatenate would act similar numpy.concatenate, returning a new zarr.Array object after creating the new manifest in store_c. This could also be done in two steps by adding a save_manifest method to the Zarr arrays.

Possible extensions

I've tried very hard to keep the scope of this as small as possible. There are currently few v3 storage transformers to emulate so I think the best next step is to try out this simple approach before spending too much time on a spec or elaborating on future options. That said, there are some obvious ways to extend this:

  1. Supporting writes to manifest arrays (possible, there are many edge cases to consider)
  2. Enable content addressable storage by hashing keys during writes
  3. Support non-JSON manifests (many options)

Props

🙌 to those that have done a great job pushing this subject forward already: @martindurant, @alimanfoo, @rabernat among others.

Activity

  1. changed the title [-]Manifest storage transformer (v3 storage transformer)[/-] [+]Manifest storage transformer[/+] on Feb 7, 2024
  2. rabernat commented on Feb 7, 2024

    @rabernat
    Contributor

    In this proposal, what type of thing is arr_ab?

  3. jhamman commented on Feb 7, 2024

    @jhamman
    MemberAuthor

    In this proposal, what type of thing is arr_ab?

    All three are zarr.Arrays. I'll add some clarification.

  4. rabernat commented on Feb 7, 2024

    @rabernat
    Contributor

    Can we write to zarr_ab? zarr_ab[0, 0] = 1?

  5. rabernat commented on Feb 7, 2024

    @rabernat
    Contributor

    after creating the new manifest in store_c

    I think we should seriously consider a much lighter-weight concatenation method. What about just storing references to store_a and store_b, rather than duplicating the whole manifest? Basically how ncml works.

    The advantages of this are that

    • It doesn't require a chunk manifest. It works with vanilla Zarr arrays.
    • It allows concatenation of arrays with different codecs and chunk sizes
    • For arrays with manifests, it doesn't require duplicating all of the references

    The metadata doc would somehow contain pointers to the other metadata docs. Something like

    "concatenation": {
        "axis": 0,
        "arrays": ["../foo", "../bar"]
    }

    The one part I can't quite see is how to do the references to the arrays. Some sort of URL syntax? Absolute vs. relative paths?

  6. rabernat commented on Feb 7, 2024

    @rabernat
    Contributor

    Another way of putting it is that I think perhaps "chunk manifest" and "virtual concatenation of Zarr arrays" should be completely separable and orthogonal features.

  7. martindurant commented on Feb 7, 2024

    @martindurant
    Member

    Note that the kerchunk method and its child here already allow for content-addressable storage, e.g., IPFS. Not sure if you meant something beyond that. There has been chatter elsewhere of chunk checksums and such (stored in metadata, not the bytes of the chunk).

    For the concatenation, I would want special attention paid to the multi-dimension case. Also, some consideration of groups-of-arrays which are concatenated together would be nice, but you might say that this is an xarray concern. Are you at all considering that the array chunk grid not aligning with the chunks?

    Do I understand that you imagine an output metadata structure of the main "these are the arrays" and then a JSON for each of the target arrays? Or do you end up concatenating the reference lists somewhere along the way?

    One important possible extension to consider along with those given - after a prototype is established - is that we now have a way to pass per-chunk information (analogous to the "context" I fought for), and so can have different behaviours for each chunk, like a different zero point in offset-scale filtering.

  8. jhamman commented on Feb 7, 2024

    @jhamman
    MemberAuthor

    Another way of putting it is that I think perhaps "chunk manifest" and "virtual concatenation of Zarr arrays" should be completely separable and orthogonal features.

    I've come around on this, but not for exactly the same reason. I've now redacted my original proposal which was not 100% thought though.


    Note that the kerchunk method and its child here already allow for content-addressable storage, e.g., IPFS. Not sure if you meant something beyond that.

    Certainly some parallels here but this could be done without IPFS. @alimanfoo's proposal in #82 is still a good read, despite using some now-outdated vernacular.

    For the concatenation, I would want special attention paid to the multi-dimension case. Also, some consideration of groups-of-arrays which are concatenated together would be nice, but you might say that this is an xarray concern. Are you at all considering that the array chunk grid not aligning with the chunks?

    Again, I'm going to remove this from the proposal. But I'll just say that there are some parallels with @d-v-b's proposal to "fix zarr-python's slicing" (zarr-developers/zarr-python#1603, zarr-developers/zarr-python#980) - namely the creation of a lazy Zarr Array or ArrayView that wraps one or more Zarr array. If we take serialization off the table for now, we can think of this outside the spec conversation and explore how to address this at the implementation level.

    Do I understand that you imagine an output metadata structure of the main "these are the arrays" and then a JSON for each of the target arrays? Or do you end up concatenating the reference lists somewhere along the way?

    I was thinking of concatenating the references but have walked this back because you have to enforce that all array metadata is equivalent (e.g. codecs) for all concatenated arrays. @rabernat is suggesting another approach with could work to resolve those concerns.

  9. jbms commented on Feb 7, 2024

    @jbms
    Contributor

    This is very similar to the kerchunk Reference File System format but is not exactly the same JSON format:
    https://fsspec.github.io/kerchunk/spec.html

    There are also at least a few implementations of the kerchunk json format outside of kerchunk itself:

    Would it be advantageous to use exactly the same format?

  10. martindurant commented on Feb 7, 2024

    @martindurant
    Member

    a few implementations of the kerchunk json format outside of kerchunk

    Can you please put references? They might be useful for inspiration.

  11. jbms commented on Feb 7, 2024

    @jbms
    Contributor

    a few implementations of the kerchunk json format outside of kerchunk

    Can you please put references? They might be useful for inspiration.

    I updated my comment to include one other known implementation.

  12. jbms commented on Feb 7, 2024

    @jbms
    Contributor

    @martindurant Is there a document that describes the kerchunk parquet format?

  13. martindurant commented on Feb 7, 2024

    @martindurant
    Member

    No, but I could make one.

  14. jbms commented on Feb 7, 2024

    @jbms
    Contributor

    While we can all assume what s3:// means, in order for this to be fully specified, we also need to specify the meaning of the URLs. See zarr-developers/zeps#48 for one proposal regarding URLs I created, but something more limited could also suffice.

  15. jbms commented on Feb 7, 2024

    @jbms
    Contributor

    Another issue to consider is the Confused deputy problem: user A might think they are writing to "s3://someone-elses-bucket/path" but actually end up writing with user A's credentials to "s3://user-a-private-bucket/other/path". Similarly, user A may think they are exposing "s3://someone-elses-bucket/path" over an HTTP server but actually end up sharing data from "s3://user-a-private-bucket/other/path" or "file:///etc/passwd".

  16. 25 remaining items

  17. TomNicholas commented on Oct 20, 2024

    @TomNicholas
    Member

    So elephant in the room: Now that the Icechunk Spec is out (even if some of the details are still being settled), is still there much of a use case for the json-based non-versioned chunk manifests as suggested above?

  18. elyall commented on Nov 8, 2024

    @elyall

    Here is an example use case for a OME-NGFF spec'd zarr. I began working through the Icechunk example workflow but VirtualiZarr needs some fixes and better test coverage for TIFFs (opened first issue here). I have limited bandwidth but I can help with that.

    For Icechunk am I understanding correctly that the data would still be accessible following the OME-NGFF spec but on disk the files would exist in the Icechunk spec? I'm sure some in the OME community would push back on that, especially since raw microscopy data should never be changed invalidating the main value add of Icechunk. But I'm happy if I just have something that works.

  19. jhamman commented on Nov 8, 2024

    @jhamman
    MemberAuthor

    So elephant in the room: Now that the Icechunk Spec is out (even if some of the details are still being settled), is still there much of a use case for the json-based non-versioned chunk manifests as suggested above?

    As the author of this proposal and one of the creators of Icechunk, I can say that Icechunk is where I'm going to be putting my focus. The design of Icechunk was motivate by many of the same needs described above (and more!). That doesn't mean we have to abandon this as an idea, just that I don't have plans to peruse it. I know some folks have build some prototypes around this concept so I'm happy to leave this open.


    For Icechunk am I understanding correctly that the data would still be accessible following the OME-NGFF spec but on disk the files would exist in the Icechunk spec? I'm sure some in the OME community would push back on that, especially since raw microscopy data should never be changed invalidating the main value add of Icechunk. But I'm happy if I just have something that works.

    @elyall - I think you've got one detail wrong. If you are using Icechunk to store virtual references (from Kerchunk or VirtualiZarr), Icechunk will just store the pointers to your data. The original files (tiff, etc) will remain unchanged.

  20. tasansal commented on Apr 15, 2025

    @tasansal

    So elephant in the room: Now that the Icechunk Spec is out (even if some of the details are still being settled), is still there much of a use case for the json-based non-versioned chunk manifests as suggested above?

    As the author of this proposal and one of the creators of Icechunk, I can say that Icechunk is where I'm going to be putting my focus. The design of Icechunk was motivate by many of the same needs described above (and more!). That doesn't mean we have to abandon this as an idea, just that I don't have plans to peruse it. I know some folks have build some prototypes around this concept so I'm happy to leave this open.

    For Icechunk am I understanding correctly that the data would still be accessible following the OME-NGFF spec but on disk the files would exist in the Icechunk spec? I'm sure some in the OME community would push back on that, especially since raw microscopy data should never be changed invalidating the main value add of Icechunk. But I'm happy if I just have something that works.

    @elyall - I think you've got one detail wrong. If you are using Icechunk to store virtual references (from Kerchunk or VirtualiZarr), Icechunk will just store the pointers to your data. The original files (tiff, etc) will remain unchanged.

    If we wanted serverless access with functionality like this PR proposes, how can we do that without Icechunk?

  21. TomNicholas commented on Apr 15, 2025

    @TomNicholas
    Member

    If we wanted serverless access with functionality like this PR proposes, how can we do that without Icechunk?

    I'm not sure what you mean exactly. If you want to read non-zarr data through the Zarr API you will be able to do that through VirtualiZarr's ManifestStore. But that only holds virtual references in memory, it doesn't persist them. To persist them currently your only options are to write them as Kerchunk references, or write them to Icechunk. This PR proposes another serialization format to persist the references, but now that Icechunk exists there's far less reason to build that format.

    What exactly are you looking for that prevents you from using Icechunk?

  22. tasansal commented on Apr 15, 2025

    @tasansal

    If we wanted serverless access with functionality like this PR proposes, how can we do that without Icechunk?

    I'm not sure what you mean exactly. If you want to read non-zarr data through the Zarr API you will be able to do that through VirtualiZarr's ManifestStore. But that only holds virtual references in memory, it doesn't persist them. To persist them currently your only options are to write them as Kerchunk references, or write them to Icechunk. This PR proposes another serialization format to persist the references, but now that Icechunk exists there's far less reason to build that format.

    What exactly are you looking for that prevents you from using Icechunk?

    Good questions. Let me explain our workflow. We are very open to suggestions, this is a blocker for us right now.

    We work with a large-scale, distributed, in-house data streaming platform (up to hundreds of thousands of cores pushing PB/s of data). We are trying to integrate our Zarr-based (github.com/TGSAI/mdio-python) as a data read/write format in C++ with tensorstore. The system is Task-based, similar to Dask. Since it's streaming, we don't know our chunk grid ahead of time, and it could be formed based on incoming data. Each task is approx. equivalent to a chunk. We know the chunk local coordinates etc, but we don't know the global Zarr grid we want to write into.

    Our workaround is to write into a Zarr with worker and task IDs as first two dimensions, which works fine, but we lose all the nice Zarr's indexing capabilities and have to search coordinates to retrieve anything after writing.

    If this functionality existed, we could write the chunks with keys using a random hash (or whatever else), and then the mapping from i/j/k -> hash could be built at the end, and Zarr arrays could be used like regular Zarr arrays.

    E.g. we can build this map once the file is written out to read later. This is actually precisely how our legacy file format does. Write all streaming data with chunk locality information and build this map at the end to relate to master chunk grid (based on outputs of all tasks). I am assuming this map can be serialized into JSON or another optimized binary format as a part of the spec, and is not very dynamic (like version controlling chunks)

    {
        "0.0.0":  {"path": "gs://bucket/foo.zarr/data/d465317c9c71"},
        "0.0.1":  {"path": "gs://bucket/foo.zarr/data/27e63d44b317"},
        "0.1.0":  {}, 
        "0.1.1":  {"path": "gs://bucket/foo.zarr/data/a598dab88862"}
    }

    I understand Icechunk does something similar for version control; however, in our use case we are unable to depend on a service to pass PB/s data through. If we could do this serverless, from each worker, then it would be amazing. Which is what this PR proposes, no?

  23. d-v-b commented on Apr 15, 2025

    @d-v-b
    Contributor

    icechunk is serverless

  24. tasansal commented on Apr 15, 2025

    @tasansal

    icechunk is serverless

    Maybe I should have said "serviceless". We want to be able to do this Zarr native without running a sidecar service.

  25. tasansal commented on Apr 15, 2025

    @tasansal

    If we wanted serverless access with functionality like this PR proposes, how can we do that without Icechunk?

    I'm not sure what you mean exactly. If you want to read non-zarr data through the Zarr API you will be able to do that through VirtualiZarr's ManifestStore. But that only holds virtual references in memory, it doesn't persist them. To persist them currently your only options are to write them as Kerchunk references, or write them to Icechunk. This PR proposes another serialization format to persist the references, but now that Icechunk exists there's far less reason to build that format.

    What exactly are you looking for that prevents you from using Icechunk?

    Re-read your message, and yes, we would like to persist them and have a native way to read it with Zarr. Will zarr-python or tensorstore ever be able to read a Zarr with persisted Icechunk or FSSpec References?

  26. martindurant commented on Apr 15, 2025

    @martindurant
    Member

    Zarr already can read with kerchunk references, without kerchunk being installed. To read with icechunk or vzarr, you will need those installed.

  27. tasansal commented on Apr 15, 2025

    @tasansal

    Zarr already can read with kerchunk references, without kerchunk being installed. To read with icechunk or vzarr, you will need those installed.

    Thanks @martindurant, this is exactly what I mean. Since we are reading from C++ with tensorstore, using VirtualiZarr or Icechunk is out of the question. Even the nice fsspec ecosystem doesn't exist there and tensorstore implements some of the drivers.

    If Zarr defined this serialization (or recommended a common/existing serialization) and made it a spec extension etc; we could implement it in all languages to read the same files x-platform.

  28. martindurant commented on Apr 15, 2025

    @martindurant
    Member

    we could implement it in all languages to read the same files

    if it were part of zarr, and all languages implement it yes - but the implementation would need to be made nevertheless.

    Interestingly, GDAL is implementing this, if you want to copy a C implementation OSGeo/gdal#12099

  29. TomNicholas commented on Apr 15, 2025

    @TomNicholas
    Member

    Since we are reading from C++ with tensorstore, using VirtualiZarr or Icechunk is out of the question.

    Well yeah, because Icechunk is an alternative to Tensorstore here.

    Zarr already can read with kerchunk references, without kerchunk being installed. To read with icechunk or vzarr, you will need those installed.

    This is misleading, because you do need to install fsspec to read kerchunk reference files. However you do it, there are always at least 3 layers here - the on-disk format, the thing that reaches out to remote storage, and the zarr-python API. To read icechunk's format you need the icechunk client and zarr-python, to read kerchunk's format you need fsspec and zarr-python.

    the mapping from i/j/k -> hash could be built at the end, and Zarr arrays could be used like regular Zarr arrays.

    @tasansal it's unclear to me if your use case actually requires serializing this mapping, which is what this issue is ultimately about. If you don't, and you have access to the the mapping somehow at the time that you're ready to read chunk data using the zarr API, then you could simply construct a virtualizarr ManifestStore containing the mapping in-memory and read from that. If you need to persist the mapping then you need either kerchunk/icechunk or the as-yet-nonexistent zarr proposal here.

  30. TomNicholas commented on Apr 15, 2025

    @TomNicholas
    Member

    icechunk is serverless

    Maybe I should have said "serviceless". We want to be able to do this Zarr native without running a sidecar service.

    This also does not make sense. There is no sidecar service that Icechunk runs.

    EDIT: Note @tasansal you might be thinking of ArrayLake v1, which did rely on an earthmover-run service. Icechunk is very different.

  31. martindurant commented on Apr 15, 2025

    @martindurant
    Member

    This is misleading, because you do need to install fsspec

    well it did specifically say "with fsspec"; also, fsspec is very likely to be already installed. You are still right, but

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions