Consolidated Metadata Not Passed Down #3954

Activity

1. ilan-gold

   added

   bugPotential issues with the zarr-python libraryPotential issues with the zarr-python library

   on May 7, 2026
2. d-v-b commented on May 10, 2026

   

   d-v-b

   on May 10, 2026

   Contributor

   More actions

   With nested groups, the consolidated metadata is available on the children, recursively

   I think this is meant to describe zarr-python's behavior when writing consolidated metadata. What you are observing is data where consolidated metadata was not written to the subgroups.
3. ilan-gold commented on May 11, 2026

   

   ilan-gold

   on May 11, 2026

   ContributorAuthor

   More actions

   Could you elaborate a bit? In the following example, I don't see any consolidated metadata written to disk at the subgroups but it is present on the in-memory data when read back in, which is what I would expect given the description of the behavior:

   # /// script
   # requires-python = ">=3.12"
   # dependencies = [
   #   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
   # ]
   # ///
   #
   # This script automatically imports the development branch of zarr to check for issues
   
   from __future__ import annotations
   
   import json
   
   import numpy as np
   import zarr
   
   z = zarr.open("foo.zarr", mode="w")
   z.create_group("bar")
   z["bar"].create_group("fuzz")
   z["bar"]["fuzz"].create_group("bazz")
   z["bar"]["fuzz"]["bazz"] = np.arange(5)
   zarr.consolidate_metadata(z.store)
   
   z = zarr.open("foo.zarr", mode="r")
   assert z["bar"]["fuzz"].metadata.consolidated_metadata is not None  # works
   with open("foo.zarr/bar/zarr.json") as f:
       print(json.loads(f.read())) # does not contain metadata i.e., at the next group down

   I think it's possible this is a v2 problem since the original store is v2 only!
4. d-v-b commented on May 11, 2026

   

   d-v-b

   on May 11, 2026

   Contributor

   More actions

   the v2 thing seems explanatory, but I'm not sure that it makes sense. Evidently I don't know the consolidated metadata semantics very well! maybe @TomAugspurger can explain better how things should work.
5. TomAugspurger commented on May 11, 2026

   

   TomAugspurger

   on May 11, 2026

   Contributor

   More actions

   IIUC, this is behaving as intended, at least as I understood it. Consolidating metadata takes all the metadata at and under a point in the tree and consolidates it to a single metadata document, which is written to disk. But you need to access it from that consolidation point to use it.

   I don't think we would want to write out multiple consolidated metadata documents, one at each node in the tree... Though I see how the asymmetry between zarr.open_consolidate(store)["path/to/group"] and zarr.open(store, path="path/to/group") is jarring.
6. TomAugspurger commented on May 11, 2026

   

   TomAugspurger

   on May 11, 2026

   Contributor

   More actions

   With respect to

   With nested groups, the consolidated metadata is available on the children, recursively

   I'm not sure if I wrote that or not, but in my reading that's making a statement about the in-memory metadata (.metadata.consolidated_metadata) rather than the representation on disk.

   👍React with 👍1Reacted by Ilan Gold
7. ilan-gold commented on May 11, 2026

   

   ilan-gold

   on May 11, 2026

   Last edited by ilan-gold

   ContributorAuthor

   More actions

   I'm not sure if I wrote that or not, but in my reading that's making a statement about the in-memory metadata (.metadata.consolidated_metadata) rather than the representation on disk.

   This was my impression as well (same with the on-disk, doesn't make sense to write it all out)
8. TomAugspurger commented on May 11, 2026

   

   TomAugspurger

   on May 11, 2026

   Contributor

   More actions

   Sorry, I'm just now looking at the original post more closely, and this definitely looks like a bug

   (Pdb) pp z["raw"]["varm"].metadata.consolidated_metadata

   I'll take a closer look today.

   ❤️React with ❤️1Reacted by Ilan Gold
9. ilan-gold commented on May 11, 2026

   

   ilan-gold

   on May 11, 2026

   ContributorAuthor

   More actions

   Thanks!
10. TomAugspurger

    added a commit that references this issue on May 12, 2026

    BUG: Fix empty Group nodes lacking a "consolidated_metadata" field

    ...

    5d49b92
11. TomAugspurger commented on May 12, 2026

    

    TomAugspurger

    on May 12, 2026

    Contributor

    More actions

    See #3967
12. TomAugspurger

    closed this as completedin #3967on May 12, 2026
13. TomAugspurger

    added a commit that references this issue on May 12, 2026

    BUG: Fix empty Group nodes lacking a "consolidated_metadata" field (#…

    ...

    1020ca5