Skip to content

Missmatch of Array.store_path.store.list_prefix() and Array._iter_chunk_keys() in chunks_initialized() for an array in a group #2861

Closed

Description

@relativityhd

Zarr version

v3.0.2

Numcodecs version

v0.15.1

Python Version

3.11

Operating System

Linux

Installation

uv add zarr

Description

Currently, it is not possible to view how many chunks are initialized in an array inside a group.

I was able to track it down to a mismatch of the outputs of the functions z.store_path.store.list_prefix(prefix=z.store_path.path) and z._iter_chunk_keys().

Steps to reproduce

import zarr

# Create a group with two arrays
root = zarr.group("reproduction.zarr")
root.create("x", shape=(1000, 1000), chunks=(100, 100), dtype="f4")
root.create("y", shape=(1000, 1000), chunks=(100, 100), dtype="uint8")
root.tree()
from zarr.core.array import chunks_initialized

# Now check which chunks are initialized (should be an empty tuple)
await chunks_initialized(root["x"])
> ()
# Now fill two chunks
root["x"][0:100, 0:100] = 1.0
root["x"][100:200, 0:100] = 2.0

# Now check which chunks are initialized (should be a tuple with two chunks)
await chunks_initialized(root["x"])
> ()  # <-- This shouldn't be empty
# Further investigate what happens in "chunks_initialized()"
store_contents = [x async for x in root["x"].store_path.store.list_prefix(prefix=root["x"].store_path.path)]
store_contents
> ['x/zarr.json', 'x/c/1/0', 'x/c/0/0']
tuple(chunk_key for chunk_key in root["x"]._iter_chunk_keys())

> ('c/0/0',
 'c/0/1',
 'c/0/2',
 'c/0/3',
 'c/0/4',
 'c/0/5',
 'c/0/6',
 'c/0/7',
 'c/0/8',
 'c/0/9',
 'c/1/0',
 'c/1/1',
 'c/1/2',
 'c/1/3',
 'c/1/4',
...
 'c/9/5',
 'c/9/6',
 'c/9/7',
 'c/9/8',
 'c/9/9')

Additional output

No response

Activity

  1. added
    bugPotential issues with the zarr-python library
    on Feb 25, 2025
  2. d-v-b commented on Feb 25, 2025

    @d-v-b
    Contributor

    good report, and sorry for the bug. seems like we need to relativize the output of list_prefix, or absolutifiy the values in iter_chunk_keys. Since this is so obviously wrong that I'm wondering how our tests did not catch it.

  3. relativityhd commented on Feb 25, 2025

    @relativityhd
    Author

    No problem, I am glad I could help. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugPotential issues with the zarr-python library

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions