Skip to content
This repository was archived by the owner on Aug 17, 2026. It is now read-only.
This repository was archived by the owner on Aug 17, 2026. It is now read-only.

dimension_separator and n5 key reversal #12

Open

Description

@d-v-b

N5 requires that the dimension ordering of chunk keys be reversed. I.e., for a chunk key foo/bar/0.1.2, invert_chunk_coords will return foo/bar/2.1.0.

Pre-2.81, chunk keys used two delimiters: "/" delimits the hierarchy, and "." delimits the chunk indices. After 2.8.1, chunk keys can take the form foo/bar/0/1/2, and suddenly there is no distinction between hierarchy and chunk index delimiters. "/"-separated chunk keys can arise if the array's chunk store has a key_separator property: Array._chunk_key(). N5Store has a dimension_separator property but not a key_separator property, so for N5Store chunk keys will use the default "." separator, and invert_chunk_coords will work fine.

But FSStore uses the key_separator property for determining how to write keys to storage. Arrays created using FSStore will read the key_separator property of the store adapt chunk keys accordingly. Thus, attempts to extend FSStore to work for N5 (where key_separator is "/") will run into issues where chunk keys look like foo/bar/0/1/2 and it becomes impossible for invert_chunk_coords to parse this into (array_prefix, chunk_coordinate)

Zooming out, it seems like a leaky abstraction if the format of chunk keys changes as a function of the implementation details of the storage layer. Chunk keys should have a consistent, parseable format, and it is the job of the store to convert chunk keys into the names of objects in storage.

Activity

  1. d-v-b commented on Apr 29, 2021

    @d-v-b
    CollaboratorAuthor

    actually it looks like the key change was in this PR: zarr-developers/zarr-python#718

  2. joshmoore commented on Apr 29, 2021

    @joshmoore
    Member

    My best guess at the moment is that normalization is being called multiple times. I've certainly noticed the need to carefully set normalize_keys on the FSStore contructor elsewhere, but hadn't yet tracked down what was going on.

    cc: @martindurant

  3. joshmoore commented on Sep 22, 2021

    @joshmoore
    Member

    @d-v-b : is this still an issue?

  4. d-v-b commented on Oct 18, 2024

    @d-v-b
    CollaboratorAuthor

    I transferred this issue from zarr-python to n5py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions