Repository navigation
keys() returning an empty list on a FsspecStore #3185
Description
Zarr version
3.0.8
Numcodecs version
0.16.1
Python Version
3.11.7
Operating System
Mac
Installation
using pip into a virtual environment
Description
When calling the keys() function on the root group of a zarr file hosted in a S3 bucket I get an empty list, while accessing directly the key (of a group which I know exists in the file) works.
I am not sure whether it is a bug or there is something wrong in the way I stored the zarr file in the S3 bucket or opened it through the zarr library.
Steps to reproduce
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "zarr >= 3.0",
# "zarr[remote]>=3"
# "aiohttp"
# ]
# ///
#
import zarr
filename = r'https://s3.embl.de/bls-app-test/test.bls.zarr'
store = zarr.storage.FsspecStore.from_url(url=filename, read_only=True)
print(f"listing supported: {store.supports_listing}")
root = zarr.open_group(store=store, mode='r')
root["Brillouin_data"] # the key 'Brillouin_data' can e found
print(f"keys: {list(root.keys())}") # but when listing the keys in the root an empty list is returnedAdditional output
listing supported: True
keys: []
Activity
d-v-b commentedon Jun 30, 2025
It's possible that your bucket is not configured to allow listing. can you list the objects in the bucket using the s3 api (or fsspec)?
bevilacqc commentedon Jul 3, 2025
@d-v-b thanks for your input. I checked and listing was already enabled.
I managed to make it work by creating a FsspecStore directly from a fsspec filesystem explicitly set to S3:
d-v-b commentedon Jul 3, 2025
ok, I think I see what the problem was:
https://s3.embl.de/bls-app-test/test.bls.zarris not an s3 url (i.e, it doesn't start withs3://, so zarr-python can't infer that it should use the s3 backend, which results in the httpfilesystem backend to be used instead. I'm not sure there's anything zarr python can do here -- I think if you have an s3-compatible http service, you will need to explicitly request an s3 filesystem backend, as you did here.bevilacqc commentedon Jul 3, 2025
Maybe one could add a sentence in the documentation of the Remote store along the line of
Something similar could also be added to the documentation of
FsspecStore.from_urld-v-b commentedon Jul 4, 2025
@bevilacqc that's a great idea for a docs addition. would you like to open a PR with those changes? if so, let us know if you run into any issues getting the development environment set up
bevilacqc commentedon Jul 8, 2025
@d-v-b I opened the PR #3212 which now passed all the checks
d-v-b commentedon Oct 27, 2026
closed via #3212