Repository navigation
URL syntax for icechunk #576
Description
I'm in the process of implementing support for ZEP 8-style URL syntax in Neuroglancer, and in conjunction with that am planning to also add support for icechunk format.
Here are some examples of existing URLs that are supported or will be supported:
gs://bucket/path/to/array/|zarr3:
gs://bucket/path/to/group/|zarr3:path/to/array/
gs://bucket/path/to/file.zip|zip:path/to/array/|zarr3:
gs://bucket/path/to/file.zip|zip:path/to/nested.zip|zip:path/to/array/|zarr3:
gs://bucket/path/to/ocdbt/|ocdbt:path/to/array/|zarr3:
Note that the URL consists of a "pipeline" of |-separated components, where the first component must be a base kvstore protocol (e.g. gs, s3, http), followed by zero or more kvstore adapter schemes, like zip, followed by a data format scheme, e.g. zarr2, zarr3, precomputed, n5, etc. There is also format auto-detection, which can add necessary kvstore adapter schemes and the final data format scheme automatically. For example, if you type just gs://bucket/path/to/file.zip then it will first get completed to gs://bucket/path/to/file.zip|zip: (based on the content, not the filename) and then if there is a zarr array or group at the root within the zip file it will get further completed to gs://bucket/path/to/file.zip|zip:|zarr3:.
Note that whether some part of the path goes before or after the |zarr: currently doesn't matter, but if a group storage transformer were used, it would matter. I'm planning to normalize urls so that the outer kvstore url points to the topmost valid zarr v3 group. E.g. if we normalize to gs://bucket/path/to/group/|zarr3:path/to/array/ then that means:
- gs://bucket/path/to/zarr.json does NOT exist, but
- gs://bucket/path/to/group/zarr.json does exist, and
- gs://bucket/path/to/group/path/zarr.json does exist and
- gs://bucket/path/to/group/path/to/zarr.json does exist and
- gs://bucket/path/to/group/path/to/array/zarr.json does exist
With this background out of the way, for icechunk, there are two questions:
- How to encode the branch / tag / snapshot in the URL
- Whether to treat icechunk as a key-value store adapter, or as a final data format in place of zarr3.
Some possible options:
gs://bucket/path/to/icechunk_repo/|icechunk:branch.main/path/to/array/|
gs://bucket/path/to/icechunk_repo/|icechunk:branch.main@path/to/array/|
gs://bucket/path/to/icechunk_repo/|icechunk:branch.main|zarr3:path/to/array/|
gs://bucket/path/to/icechunk_repo/refs/branch.main/|icechunk:path/to/array/|
gs://bucket/path/to/icechunk_repo/refs/branch.main/|icechunk:|zarr3:path/to/array/|
Neuroglancer needs a URL syntax to support it at all, unlike e.g. zarr-python, but it would be nice to standardize on a syntax that will also be supported by other tools in the future.
Choosing a URL syntax that includes a final |zarr3: component most closely corresponds to the current zarr-python integration where icechunk just behaves as a key-value store and translates the metadata and chunks back to the standard zarr v3 metadata encoding and key encoding. I think the right choice, though, depends on how you expect to evolve icechunk in the future.
Activity
rabernat commentedon Jan 27, 2025
Hi Jeremy--sorry it took so long to reply to this very helpful and collaborative issue. We really appreciate you reaching out and for working to implement IC in Neuroglancer! 🙌
We are standing by to help however we can.
We don't have a strong preference here. Are there any constraints on the character space you want to use? If not, we'd suggest something like:
icechunk@branch=main,icechunk@tag=v1,icechunk@snapshot=XXX.I think it makes sense to treat IC as the key-value adaptor. So a full URL like
jbms commentedon Jan 27, 2025
Currently the intention is to use URL-like syntax for each
|-separated component of the "URL pipeline", so normally:scheme:whateverorscheme:whatever?query#fragmentbut as a special case, also justschemeas a shorthand forscheme:.Allowing
icechunk@branch=mainas the scheme would be a departure from that syntax.I suppose in principle that we could extend the syntax to allow a scheme-specific parameter after
@, in order to allowicechunk@xxx:--- but I'd be inclined to stick with something closer to traditional URL syntax to avoid confusion if we can find something reasonable.icechunk:@branch=main/path...oricechunk:@branch.main/path...would be closer to traditional URL syntax.In any case, it would probably make sense to allow plain
icechunk:as a shorthand for the main branch and to select that by default in the format auto-detection --- e.g. if someone types ins3://bucket/path/to/repoand it contains a zarr v3 array at the root --- then it could get automatically completed tos3://bucket/path/to/repo/|icechunk:|zarr3:.One other advantage of
icechunk:@branch=main/overicechunk@branch=main/is that after the user types:s3://bucket/path/to/repo/, URL completion can suggest a completion of|icechunk:which the user can then accept with TAB. Then the user immediately will see completions for paths within the main branch (and given that it will be a zarr hierarchy, will be suggested|zarr3:), but will also be able to type@branch...to get a different branch, and the completion list could include@branch,@tag, etc. to let the user know. On the other hand, if theicechunk@branchsyntax is used, then the user would have to type backspace to remove the final colon in order to being typing the branch name, and it would be less discoverable.I am still deciding on the syntax for OCDBT but it would make sense to use a similar syntax.
With OCDBT there are no branches or tags but the version can either be "HEAD" or specified by generation number (an integer) or commit time (timestamp).
What seems most promising at the moment is:
ocdbt:path/for HEADocdbt:@v123/path/for generation 123ocdbt:@2024-01-01T01:23:45.678Z/path/for an exact timestamp (i.e. https://en.wikipedia.org/wiki/ISO_8601 UTC date time syntax).For icechunk we might potentially also want a syntax for indicating a particular version of a branch, either by number or timestamp.
Note that to avoid ambiguity with
@being a valid path character, we can require that@in the path is percent encoded as%40.rabernat commentedon Jan 27, 2025
Sure, definitely fine with
icechunk:with a colon. Really no strong feelings aboutbranch=mainvsbranch.main. Maybebranch/mainis the most consistent with URL style?Thanks for thinking this stuff through! It will be very useful to be able to use URLs this way.
jbms commentedon Feb 11, 2025
I have landed the icechunk support in Neuroglancer:
https://neuroglancer-docs.web.app/datasource/icechunk/index.html
I ultimately went with:
icechunk:@branch.main/pathicechunk:@tag.v123/pathicechunk:@XXXXXXXXXXXXXXXXXXXX/pathHowever, given that this support has just been added, if there are strong objections to this syntax I could still change it.