Hierarchical chunk storage

@joshmoore

Currently, the ome zarr specification demands that the chunks are stored in a single directory.
This causes issues on the file system for large images / volumes with many chunks.

N5 stores chunks in a hierarchy natively and zarr also supports this with the NestedDirectoryStore (although this is currently not working according to @joshmoore).

Here's what I've been working on:

n5 supports nested (of course) but zarr+s3 is missing: N5S3ZarrReader: combine N5ZarrReader logic into N5AmazonS3Reader subclass saalfeldlab/n5-zarr#5
jzarr doesn't support nested: Array nesting: Add the ability to use N5-style nested layout bcdev/jzarr#17
NestedDirectoryStore supports zarr+nesting but not s3. The newer FSStore which does support s3 had a bug in the nesting: FSStore: key_separator fix zarr-developers/zarr-python#699

Conversations starting https://gitter.im/zarr-developers/community?at=601d471fc83ec358be27944f and during the 2021-02-10 community call suggest that even before the V3 spec (which defaults to nested storage) it might be possible for us to transition to using it more heavily with the V2 spec. A requirement would be a heuristic or metadata entry to determine if chunks are nested or flat. zarr-python is waiting on a PR from me.

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

@SabineEmbacher

Status update:

bioformats2raw is moving towards 0.3.0. There will already be some breaking layout changes.
zarr-python has the FSStore fixe merged to the mainline. The 2.7.0 release will contain it.
The jzarr work will be taken over by @SabineEmbacher but is moving forward.

With all of this in mind, I'd propose that we go ahead and bump to OME-Zarr version to 0.2.0 stating in the spec that nested storage is a MUST. A client can then detect whether the fileset is 0.1.0 or 0.2.0 from the metadata and make a choice as to the chunk index separator.

cc: @manzt (since I've failed to engage on the JS front)

0.2.0: switch to nested layout

see: zarr-developers/zarr-python#715

Feels like most implementations have now implemented this new strategy. Likely a solid next step would be to add v0.1 and v0.2 implementations to https://github.com/ome/ome_zarr_test_suite

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/open-an-image-from-omero-in-imagej-js/47747/17

Navigation Menu

Repository navigation

Hierarchical chunk storage #29

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hierarchical chunk storage #29

Description

Activity

joshmoore commented on Feb 23, 2021

imagesc-bot commented on Feb 24, 2021

joshmoore commented on Mar 12, 2021

joshmoore commented on Apr 8, 2021

joshmoore commented on Apr 24, 2021

imagesc-bot commented on May 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions