Skip to content

Commit e023851

Browse files
committed
fix: accept universal-pathlib UPath as a StoreLike value
A remote UPath now creates an FsspecStore from the filesystem and storage options the UPath already carries, and a local UPath creates a LocalStore so that UPath("/data") and Path("/data") agree. This previously worked only by accident. In universal-pathlib < 0.3 every UPath subclassed pathlib.Path and implemented __fspath__, so a remote path was either converted to a URI string by the caller (xarray does this) or wrapped in a LocalStore that happened to dispatch through fsspec. Since universal-pathlib 0.3 remote paths do neither, and passing one raised TypeError: Unsupported type for store_like. The UPath branch is checked before the Path branch so that routing is identical on both universal-pathlib 0.2 and 0.3. FsspecStore.from_upath now converts the UPath's filesystem to async mode instead of raising TypeError for synchronous filesystems and warning for sync-mode instances of async ones. The memory:// routing test wraps a synchronous MemoryFileSystem, which needs fsspec's AsyncFileSystemWrapper, so it is skipped below fsspec 2024.12.0 as test_wrap_sync_filesystem already does. Closes zarr-developers#4244 Assisted-by: ClaudeCode:claude-fable-5
1 parent 2569d1f commit e023851

7 files changed

Lines changed: 127 additions & 4 deletions

File tree

‎changes/4245.bugfix.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Accept [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath` objects wherever
2+
zarr accepts a `StoreLike` value. A remote `UPath` now creates an `FsspecStore` using the
3+
filesystem and storage options the `UPath` already carries, and a local `UPath` creates a
4+
`LocalStore`, so that `UPath('/data')` and `Path('/data')` behave the same.
5+
6+
Previously this worked only by accident: in universal-pathlib < 0.3 every `UPath` subclassed
7+
`pathlib.Path` and implemented `__fspath__`, so remote paths were either converted to a URI string
8+
by the caller or wrapped in a `LocalStore` that happened to dispatch through fsspec. Since
9+
universal-pathlib 0.3 remote paths do neither, and passing one raised
10+
`TypeError: Unsupported type for store_like`.
11+
12+
`FsspecStore.from_upath` also now converts the `UPath`'s filesystem to async mode, instead of
13+
raising `TypeError` for synchronous filesystems and warning for sync-mode instances of async ones.

‎docs/user-guide/storage.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,15 @@ print(group)
9090
- an FSSpec [FSMap object](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.FSMap),
9191
which will create an [FsspecStore](#remote-store).
9292

93+
- a [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath`, which will create an
94+
[FsspecStore](#remote-store), or a [local store](#local-store) if the `UPath` is local. The
95+
`UPath` carries its own storage options, so passing `storage_options` alongside one is an error:
96+
97+
```python exec="false" reason="requires universal-pathlib, which is not in the docs environment"
98+
from upath import UPath
99+
group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r')
100+
```
101+
93102
- a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
94103
see explicit store creation below.
95104

‎src/zarr/storage/_common.py‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
from zarr.errors import ContainsArrayAndGroupError, ContainsArrayError, ContainsGroupError
2525
from zarr.storage._local import LocalStore
2626
from zarr.storage._memory import ManagedMemoryStore, MemoryStore
27-
from zarr.storage._utils import _join_paths, normalize_path, parse_store_url
27+
from zarr.storage._utils import UPath, _join_paths, normalize_path, parse_store_url
2828

2929
_has_fsspec = importlib.util.find_spec("fsspec")
3030
if _has_fsspec:
@@ -302,7 +302,7 @@ def __eq__(self, other: object) -> bool:
302302
return False
303303

304304

305-
type StoreLike = Store | StorePath | FSMap | Path | str | dict[str, Buffer]
305+
type StoreLike = Store | StorePath | FSMap | Path | UPath | str | dict[str, Buffer]
306306

307307

308308
async def make_store(
@@ -322,6 +322,7 @@ async def make_store(
322322
- `dict[str, Buffer]` = `MemoryStore` object.
323323
- `None` = `MemoryStore` object.
324324
- `FSMap` = `FsspecStore` object.
325+
- `UPath` = `FsspecStore` object, or `LocalStore` for a local `UPath`.
325326
326327
Parameters
327328
----------
@@ -382,6 +383,17 @@ async def make_store(
382383
# Create a new in-memory store
383384
return await make_store({}, mode=mode, storage_options=storage_options)
384385

386+
elif isinstance(store_like, UPath):
387+
# This must be checked before Path: in universal-pathlib < 0.3 every UPath, including
388+
# remote ones like S3Path, subclasses pathlib.Path, and would otherwise be misrouted to a
389+
# LocalStore. Local UPaths get a LocalStore so that UPath("/data") and Path("/data") agree,
390+
# mirroring how the equivalent strings are routed below.
391+
if store_like.protocol in ("", "file"):
392+
return await make_store(
393+
Path(store_like.path), mode=mode, storage_options=storage_options
394+
)
395+
return FsspecStore.from_upath(store_like, read_only=_read_only)
396+
385397
elif isinstance(store_like, Path):
386398
# Create a new LocalStore
387399
return await LocalStore.open(root=store_like, mode=mode, read_only=_read_only)

‎src/zarr/storage/_fsspec.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,8 +171,12 @@ def from_upath(
171171
-------
172172
FsspecStore
173173
"""
174+
# A UPath hands back a filesystem in whatever mode it was constructed with, which is
175+
# synchronous unless the caller passed asynchronous=True. Route it through _make_async so
176+
# that sync-mode instances of async filesystems are re-created in async mode, and
177+
# genuinely synchronous filesystems are wrapped.
174178
return cls(
175-
fs=upath.fs,
179+
fs=_make_async(upath.fs),
176180
path=upath.path.rstrip("/"),
177181
read_only=read_only,
178182
allowed_exceptions=allowed_exceptions,

‎src/zarr/storage/_utils.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
from urllib.parse import urlparse
77

88
if importlib.util.find_spec("upath"):
9-
from upath.core import UPath
9+
# Re-exported for zarr.storage._common, which needs it to recognize UPath store_like values.
10+
from upath.core import UPath as UPath
1011
else:
1112

1213
class UPath: # type: ignore[no-redef]

‎tests/test_store/test_core.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,19 @@
44
from typing import Any, Literal
55

66
import pytest
7+
from packaging.version import parse as parse_version
78

89
import zarr
910
from zarr import Group
11+
from zarr.abc.store import Store
1012
from zarr.core.buffer import cpu
1113
from zarr.core.common import ZARR_JSON, AccessModeLiteral, ZarrFormat
1214
from zarr.storage import FsspecStore, LocalStore, MemoryStore, StoreLike, StorePath, ZipStore
1315
from zarr.storage._common import (
1416
_contains_node_v3,
1517
contains_array,
1618
contains_group,
19+
make_store,
1720
make_store_path,
1821
)
1922
from zarr.storage._utils import (
@@ -248,6 +251,45 @@ async def test_make_store_path_storage_options_raises(store_like: StoreLike) ->
248251
await make_store_path(store_like, storage_options={"foo": "bar"})
249252

250253

254+
# universal-pathlib 0.2.x emits this from its own subclass registry when a local UPath is built.
255+
@pytest.mark.filterwarnings(
256+
"ignore:Detected a customized `__new__` method in subclass:DeprecationWarning"
257+
)
258+
@pytest.mark.parametrize(
259+
("url", "expected"),
260+
[
261+
("memory://bucket/foo.zarr", FsspecStore),
262+
("s3://bucket/foo.zarr", FsspecStore),
263+
("file://{tmp}/foo.zarr", LocalStore),
264+
("{tmp}/foo.zarr", LocalStore),
265+
],
266+
)
267+
async def test_make_store_upath(url: str, expected: type[Store], tmp_path: Path) -> None:
268+
"""
269+
A remote UPath becomes an FsspecStore, and a local one becomes a LocalStore, so that
270+
UPath("/data") and Path("/data") agree. See https://github.com/zarr-developers/zarr-python/issues/4244.
271+
"""
272+
upath = pytest.importorskip("upath")
273+
fsspec = pytest.importorskip("fsspec")
274+
if url.startswith("s3://"):
275+
pytest.importorskip("s3fs")
276+
if url.startswith("memory://") and parse_version(fsspec.__version__) < parse_version(
277+
"2024.12.0"
278+
):
279+
# MemoryFileSystem is synchronous, so it can only be used once fsspec is new enough to
280+
# supply AsyncFileSystemWrapper.
281+
pytest.skip("No AsyncFileSystemWrapper")
282+
store = await make_store(upath.UPath(url.format(tmp=tmp_path)))
283+
assert isinstance(store, expected)
284+
285+
286+
async def test_make_store_upath_storage_options_raises() -> None:
287+
"""A UPath carries its own storage options, so a separate mapping is ambiguous."""
288+
upath = pytest.importorskip("upath")
289+
with pytest.raises(TypeError, match="storage_options"):
290+
await make_store(upath.UPath("memory://bucket/foo.zarr"), storage_options={"foo": "bar"})
291+
292+
251293
async def test_unsupported() -> None:
252294
with pytest.raises(TypeError, match="Unsupported type for store_like: 'int'"):
253295
await make_store_path(1)

‎tests/test_store/test_fsspec.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import json
44
import re
5+
import warnings
56
from typing import TYPE_CHECKING, Any
67

78
import numpy as np
@@ -249,6 +250,47 @@ def test_from_upath(self, endpoint_url: str) -> None:
249250
assert result.fs.asynchronous
250251
assert result.path == f"{test_bucket_name}/foo/bar"
251252

253+
@pytest.mark.skipif(
254+
parse_version(fsspec.__version__) < parse_version("2024.03.01"),
255+
reason="Prior bug in from_upath",
256+
)
257+
def test_from_upath_sync_filesystem(self, endpoint_url: str) -> None:
258+
"""
259+
A UPath built without ``asynchronous=True`` -- the common case -- yields an async-mode
260+
filesystem that keeps the original storage options.
261+
"""
262+
upath = pytest.importorskip("upath")
263+
path = upath.UPath(
264+
f"s3://{test_bucket_name}/foo/bar/",
265+
endpoint_url=endpoint_url,
266+
anon=False,
267+
)
268+
assert not path.fs.asynchronous
269+
with warnings.catch_warnings():
270+
warnings.simplefilter("error", ZarrUserWarning)
271+
result = FsspecStore.from_upath(path)
272+
assert result.fs.asynchronous
273+
assert result.fs.endpoint_url == endpoint_url
274+
assert result.path == f"{test_bucket_name}/foo/bar"
275+
276+
async def test_open_group_from_upath(self, endpoint_url: str) -> None:
277+
"""
278+
Passing a remote UPath to the top-level API works.
279+
280+
Regression test for https://github.com/zarr-developers/zarr-python/issues/4244.
281+
"""
282+
upath = pytest.importorskip("upath")
283+
path = upath.UPath(
284+
f"s3://{test_bucket_name}/upath-group",
285+
endpoint_url=endpoint_url,
286+
anon=False,
287+
)
288+
group = await zarr.api.asynchronous.open_group(path, mode="w", attributes={"key": "value"})
289+
assert isinstance(group.store_path.store, FsspecStore)
290+
291+
reopened = await zarr.api.asynchronous.open_group(path, mode="r")
292+
assert dict(reopened.attrs) == {"key": "value"}
293+
252294
def test_init_warns_if_fs_asynchronous_is_false(self, endpoint_url: str) -> None:
253295
try:
254296
from fsspec import url_to_fs

0 commit comments

Comments
 (0)