Skip to content

Enable deterministic and formatted metadata output #3281

New issue
New issue
Closed
Closed
Enable deterministic and formatted metadata output#3281
Labels
enhancementNew features or improvements

Description

@lkluft
lkluft
opened on Jul 22, 2025

Currently, the order of entries in Zarr metadata is non-deterministic. This was previously discussed in issue #3254, where it was concluded that the Zarr specification does not prescribe a specific order—so downstream applications should not rely on one.

However, it would still be beneficial if writing the same dataset consistently produced identical metadata. This would, for example, facilitate computing dataset checksums.

Additionally, Zarr v3 now writes unformatted JSON to the .zmetadata file of v2 stores. In Zarr v2, human-readable (i.e., pretty-printed) metadata was preferred (see #704), a practice that v3 also adopted (see #1952). I assume that this is simply a regression. Ideally, metadata formatting would be consistent across versions and backends.

It would be great to consider enabling deterministic and formatted metadata output. This would improve reproducibility, readability, and tooling interoperability without conflicting with the specification.

/cc @d70-t

Activity

  1. d-v-b commented on Jul 22, 2025

    @d-v-b
    d-v-b
    on Jul 22, 2025
    Contributor

    this makes sense, I see two actionable items here:

    • we ensure that the keys of the metadata dict are written in a consistent order (ideally the order in which the keys are defined in the zarr spec)
    • We ensure that writing .zmetadata is sensitive to the global JSON formatting configuration.

    evidently we were not testing either of these behaviors

  2. d-v-b commented on Jul 22, 2025

    @d-v-b
    d-v-b
    on Jul 22, 2025
    Contributor

    @lkluft can you link to any test failures caused by these regressions, or point me to the code that's sensitive to key order?

  3. dstansby
    added
    enhancementNew features or improvements
    on Jul 22, 2025

  4. lkluft commented on Jul 22, 2025

    @lkluft
    lkluft
    on Jul 22, 2025
    ContributorAuthor

    The following code snippet is sufficient to reproduce both issues:

    import xarray as xr


ds = xr.Dataset({"a": range(10), "b": range(10)})

# Creates a "non-formatted" store_v2.zarr/.zmetadata
ds.to_zarr("store_v2.zarr", zarr_format=2, mode="w")

# Creates a pretty-formatted store_v3.zarr/zarr.json
ds.to_zarr("store_v3.zarr", zarr_format=3, mode="w")

    This directly shows that the v2 metadata is not formatted, whereas the v3 metadata is. (In the v2 library, the v2 metadata was formatted.)

    The order of the keys is more subtle. While all Zarr-specific (top-level) keys appear to be deterministic, the order of the variables in the consolidated metadata is not. Here, "a" and "b" may appear in any order if the snippet is run repeatedly. It seems that the consolidation of the metadata occurs after the variables have been written. Consequently, a non-deterministic listing of the filesystem could cause this behaviour. However, sorting the consolidated metadata before attaching it should solve this.

  5. d-v-b commented on Jul 22, 2025

    @d-v-b
    d-v-b
    on Jul 22, 2025
    Contributor

    ok maybe I misunderstood the scope of the term "metadata". I thought this issue was about array metadata documents, but it looks like this is pretty specific to consolidated metadata, is that right?

  6. lkluft commented on Jul 22, 2025

    @lkluft
    lkluft
    on Jul 22, 2025
    ContributorAuthor

    ok maybe I misunderstood the scope of the term "metadata". I thought this issue was about array metadata documents, but it looks like this is pretty specific to consolidated metadata, is that right?

    Yes, I wasn't very specific on that. So far, this seems to only affect the consolidated metadata. However, this still implies the above mentioned problems of reproducibility (in terms of checksums).

  7. d-v-b commented on Jul 22, 2025

    @d-v-b
    d-v-b
    on Jul 22, 2025
    Contributor

    if you're interested in contributing a patch, I think we could add a step that sorts the contents of consolidated metadata. The sorting algorithm should probably sort by ascending path length, where a path is a sequence of strings joined by the "/", falling back to lexicographic sorting for keys with the same path length.

  8. lkluft commented on Jul 23, 2025

    @lkluft
    lkluft
    on Jul 23, 2025
    ContributorAuthor

    Sure! I opened #3288 with a straight-forward implementation. I'm not sure if this approach is too simplistic, but I'm happy to discuss other options.

  9. d-v-b commented on Jul 23, 2025

    @d-v-b
    d-v-b
    on Jul 23, 2025
    Contributor

    thank you! I will look at it tomorrow, unless another dev gets there first :)

  10. TomAugspurger commented on Nov 2, 2026

    @TomAugspurger
    TomAugspurger
    on Nov 2, 2026
    Contributor

    I think this was closed by #3288

  11. TomAugspurger
    closed this as completedon Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew features or improvements

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions