Divergent behavior of memory store and localstore list prefix

Zarr version

main

Numcodecs version

main

Python Version

3.11

Operating System

Mac

Installation

uv

Description

Memory Store and Fsspec file system store return different results for list_prefix I think that the localstore matches what I would expect

Steps to reproduce

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "zarr@git+https://github.com/zarr-developers/zarr-python.git@main",
# ]
# ///
#

import asyncio

import zarr

zarr.print_debug_info()


async def list_prefix(store, prefix):
    return sorted([k async for k in store.list_prefix(prefix)])


def run(store):
    root = zarr.open_group(store, mode="w")
    root.create_group("0")
    root.create_array("0_c", shape=(1,), dtype="i4")
    print(asyncio.run(list_prefix(store, "0")))


print("LocalStore:")
run(zarr.storage.LocalStore(root="/tmp/zarr_repro"))

print("MemoryStore:")
run(zarr.storage.MemoryStore())

Additional output

LocalStore:
['0/zarr.json']
MemoryStore:
['0/zarr.json', '0_c/zarr.json']

platform: macOS-15.7.4-arm64-arm-64bit-Mach-O
python: 3.13.9
zarr: 3.1.6.dev62+ga02d99616

**Required dependencies:**
packaging: 26.0
numpy: 2.4.3
numcodecs: 0.16.5
typing_extensions: 4.15.0
donfig: 0.8.1.post1

**Optional dependencies:**
numcodecs: 0.16.5

**Not Installed:**
botocore
cupy-cuda12x
fsspec
s3fs
gcsfs
universal-pathlib
rich
obstore

Thanks for looking at this. https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface states that the behavior is undefined for non-slash-terminated prefixes, so this isn't a bug. Although, I understand that it's not user friendly.

I think the reason for this is that different storage implementations may want to treat "/" as special, or not. In the filesystem, "/" is special because it's used for directory listings. In memory storage and object storage, "/" is not special. Therefore, I think a user should need to explicitly include "/" in order for it to be considered as part of a prefix. I don't think we should append a "/" on memory or object stores.

I'd prefer better documentation in the listing methods as a way to address this.

IMO we should use object storage semantics consistently, which means a prefix is a plain string, and so I think in this case both stores should list all keys that start with the string "0", namely ['0/zarr.json', '0_c/zarr.json']

But the spec also says:

The store interface also defines some operations involving prefixes. In the context of this interface, a prefix is a string containing only characters that are valid for use in keys and ending with a trailing / character.

and

Note: the behaviour of list_prefix is undefined if prefix does not end with a trailing slash / and the store can assume there is at least one key that starts with prefix.

the store interface part of the spec should not be considered normative

the store interface part of the spec should not be considered normative

the store implementations section states that it is not normative - https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#store-implementations. By contrast, regardless of whether that decision was wise, the abstract store interface - https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#abstract-store-interface - is not excluded from the normative default.

But the spec also says:

The store interface also defines some operations involving prefixes. In the context of this interface, a prefix is a string containing only characters that are valid for use in keys and ending with a trailing / character.

and

Note: the behaviour of list_prefix is undefined if prefix does not end with a trailing slash / and the store can assume there is at least one key that starts with prefix.

I think the second statement makes it clear that the behavior when a trailing slash is not included is undefined.

the abstract store interface is just a description of an abstract store interface. It's not telling implementations what to do. there's no normative language. If zarr-python's store does not match the abstract store interface, but we still implement the normative content of the spec, then we are fine.

the main purpose of prefix listing is to discover the contents of groups. that's a list prefix operation on "<group name>/", which we have covered. How we support list prefix with strings not ending in a "/" is up to us. I would do whatever is most normal for object storage.

also, even if the spec did say "your implementation MUST implement the store using such and such interface" it would be wrong and we would 100% ignore that, because our store implementation is internal to zarr python and what matters for spec compliance is the data we create.

the abstract store interface is just a description of an abstract store interface. It's not telling implementations what to do. there's no normative language. If zarr-python's store does not match the abstract store interface, but we still implement the normative content of the spec, then we are fine.

the main purpose of prefix listing is to discover the contents of groups. that's a list prefix operation on "<group name>/", which we have covered. How we support list prefix with strings not ending in a "/" is up to us. I would do whatever is most normal for object storage.

I'd feel more confident in your assessment if zarr-developers/zarr-specs#307 had the response of anyone from the ZSC. Maybe this can get cleared up with the new spec commitee.

it really does not matter what the ZSC or the spec says about the design of our internal APIs. I mean that in a very literal sense. If zarr implements the store "off spec", it literally does not matter in any detectable way. By contrast, if we violate the spec rules about stored representations of data, it does matter in a detectable way: we would create data that other implementations could not read.

But the design of our internal APIs is internal to zarr python, is has no effect on other implementation, and so even if the spec says anything about it, that part of the spec would be totally ignorable.

I think a reasonable person could be confused by the lack of an explicit non-normative disclaimer in that section of the spec, but I can agree that's a separate problem. We can put it aside.

To the actual implementation questions, as I see them:

should we normalize all prefixes to include a trailing slash?

I'm a strong no on this, because it's breaking for object storage interfaces.

should we remove normalization on prefixes to never add a trailing slash?

I'm mixed on this. fsspec is a file-system centered abstraction, so I'm not sure about adjusting it to behave like object storage.

conceptually the simplest thing is for list_prefix(prefx) to return all keys in the store starting with the string prefx. That should be our starting point. At this time I don't see the value in adding special logic for prefixes that end in special characters, or applying any transformations to prefixes, because callers should be able to handle this.

a store could raise an exception if it can't handle a certain prefix, e.g. one that doesn't end with "/"

fix zarr-developers#3773: Fix divergent behavior between MemoryStore …

We're dealing with another path handling bug in #3926, which, similar to this issue, results from the lack of a contract for how paths and prefixes are interpreted.

We could consider at least some documentation about expectations:

# src/zarr/abc/store.py

@abstractmethod
def list_prefix(self, prefix: str) -> AsyncIterator[str]:
    """
    Yield every stored key K that lies under ``prefix`` in the path tree.

    A key K lies under ``prefix`` if and only if one of the following holds:

    * ``prefix == ""`` (root: every key is under it), or
    * ``K == prefix.rstrip("/")`` (the prefix itself is a stored key), or
    * ``K`` starts with ``prefix.rstrip("/") + "/"`` (K is in a sub-path).

    Importantly, this is **path-component containment**, not string
    containment. Given stored keys ``"0/zarr.json"`` and ``"0_c/zarr.json"``,
    ``list_prefix("0")`` MUST yield only ``"0/zarr.json"`` — never
    ``"0_c/zarr.json"`` — because ``0_c`` and ``0`` are distinct components.

    Trailing slashes on ``prefix`` are not significant: ``list_prefix("a")``
    and ``list_prefix("a/")`` MUST yield the same set.

    Keys are returned relative to the root of the store (i.e. with no
    additional prefix prepended), in unspecified order, with no duplicates.
    """

@abstractmethod
def list_dir(self, prefix: str) -> AsyncIterator[str]:
    """
    Yield the immediate children of ``prefix`` in the path tree.

    Defined in terms of ``list_prefix``: for every key K yielded by
    ``list_prefix(prefix)`` whose remainder ``R = K[len(prefix.rstrip("/"))+1:]``
    is non-empty, this method MUST yield ``R.split("/", 1)[0]``, deduplicated.
    Both file children (e.g. ``"zarr.json"``) and directory children (e.g.
    ``"c"`` for a key ``"c/0/0"``) are yielded as bare names, with no slashes.

    Trailing slashes on ``prefix`` are not significant.

    If ``prefix`` itself is not a path-component prefix of any stored key,
    this method MUST yield nothing (not raise).
    """

Having defined expectations would allow us to design more comprehensive hypothesis tests that catch for issues like this and #3926.

Or, we could have a layer in between the external Store APIs and the Group/Array APIs that apply these path interpretations.

We're dealing with another path handling bug in #3926, which, similar to this issue, results from the lack of a contract for how paths and prefixes are interpreted.

IMO the root cause of #3296 is the fact that the FsspecStore has a path attribute, but the store abc doesn't define any semantics for stores with a path attribute. As the path attribute is effectively a bare prefix that all path operations are relative to, I caused the bug in #3296 by mistakenly using a routine designed for normalizing zarr node paths (and thereby stripping leading "/" characters) on that plain prefix.

Definitely agree that clearer semantics would have helped here, but I'm not sure this touches on the skew in list_prefix implementations. IMO our base prefix listing routine should work like normal object storage prefix listing. stores that can't invoke normal object storage prefix listing for a given prefix (e.g., local storage) should signal this with an exception.

Navigation Menu

Repository navigation

Divergent behavior of memory store and localstore list prefix #3773

Zarr version

Numcodecs version

Python Version

Operating System

Installation

Description

Steps to reproduce

Additional output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Divergent behavior of memory store and localstore list prefix #3773

Description

Zarr version

Numcodecs version

Python Version

Operating System

Installation

Description

Steps to reproduce

Additional output

Activity

maxrjones commented on Mar 13, 2026

d-v-b commented on Mar 13, 2026

ianhi commented on Mar 13, 2026

d-v-b commented on Mar 13, 2026

maxrjones commented on Mar 13, 2026

d-v-b commented on Mar 13, 2026

d-v-b commented on Mar 13, 2026

maxrjones commented on Mar 13, 2026

d-v-b commented on Mar 13, 2026

maxrjones commented on Mar 13, 2026

d-v-b commented on Mar 14, 2026

d-v-b commented on Mar 14, 2026

maxrjones commented on Apr 28, 2026

d-v-b commented on Apr 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions