Skip to content

Commit ad34ae8

Browse files
d-v-bclaude
andcommitted
fix(array): read only None as an absent chunks argument in legacy zarr.create
The Zarr format 2 path of AsyncArray._create used the truth value of `chunks` to decide whether it was given, so False, 0, [] and np.array([0]) were silently auto-chunked while Zarr format 3 normalized or rejected them. Both formats now share the None-only `_raw_chunks` and agree on every input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent a1e6620 commit ad34ae8

3 files changed

Lines changed: 64 additions & 43 deletions

File tree

‎changes/4376.bugfix.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of a numpy array given as `chunks`, so numpy arrays work there. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array. The `chunks` and `chunk_shape` parameters of `zarr.create` and the `ShardsLike` type now declare every chunk specification the normalizer accepts, including numpy integers and arrays, explicit edge lists and stored chunk grid metadata.
1+
Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of `chunks`: only `None` means not given, as for Zarr format 3. `chunks=False` is now one chunk instead of an automatic chunk shape, numpy arrays work, and `0`, `[]` and `np.array([0])` raise instead of chunking automatically. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array. The `chunks` and `chunk_shape` parameters of `zarr.create` and the `ShardsLike` type now declare every chunk specification the normalizer accepts, including numpy integers and arrays, explicit edge lists and stored chunk grid metadata.

‎src/zarr/core/array.py‎

Lines changed: 2 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -351,18 +351,6 @@ def _array_metadata_dict_v3(zarr_json_bytes: Buffer) -> dict[str, JSON]:
351351
return metadata_dict
352352

353353

354-
def _v2_chunks_given(chunks: object) -> bool:
355-
"""Whether the legacy Zarr format 2 `chunks` argument is given.
356-
357-
A falsy `chunks` (such as `None`, 0, `[]` or `False`) is read as not given. A numpy
358-
array with more than one element has no truth value and is always given; a shorter
359-
one is given when its element is nonzero (an empty array is not given).
360-
"""
361-
if isinstance(chunks, np.ndarray):
362-
return chunks.size > 1 or bool(chunks.any())
363-
return bool(chunks)
364-
365-
366354
@dataclass(frozen=True)
367355
class AsyncArray[T_ArrayMetadata: (ArrayV2Metadata, ArrayV3Metadata)]:
368356
"""
@@ -532,11 +520,10 @@ async def _create(
532520
item_size = 1
533521
if isinstance(dtype_parsed, HasItemSize):
534522
item_size = dtype_parsed.item_size
535-
_raw_v2 = chunks if _v2_chunks_given(chunks) else chunk_shape
536-
if _raw_v2 is None:
523+
if _raw_chunks is None:
537524
outer_chunks = guess_chunks(shape, item_size)
538525
else:
539-
outer_chunks = normalize_chunks_nd(_raw_v2, shape)
526+
outer_chunks = normalize_chunks_nd(_raw_chunks, shape)
540527
if not outer_chunks.is_regular:
541528
raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
542529
_chunks = outer_chunks.chunk_shape

‎tests/test_unified_chunk_grid.py‎

Lines changed: 61 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141
from collections.abc import Callable, Generator
4242
from pathlib import Path
4343

44-
from zarr.core.common import ChunksLike
44+
from zarr.core.common import ChunksLike, ZarrFormat
4545

4646

4747
@pytest.fixture(autouse=True)
@@ -1246,46 +1246,80 @@ def test_rectilinear_chunks_gates(
12461246
zarr.create(store=MemoryStore(), shape=shape, chunks=chunks, zarr_format=2, dtype="uint8")
12471247

12481248

1249+
_LEGACY_SHAPE = (2**12, 2**12)
1250+
1251+
1252+
@pytest.mark.parametrize("zarr_format", [2, 3])
12491253
@pytest.mark.parametrize(
12501254
("chunks", "expected"),
12511255
[
1252-
(None, None),
1253-
(0, None),
1254-
((), None),
1255-
([], None),
1256-
(False, None),
1257-
(np.int64(0), None),
1258-
(np.False_, None),
1259-
(np.array(0), None),
1260-
(np.array(False), None),
1261-
(np.array([0]), None),
1262-
(np.array([]), None),
1256+
(None, guess_chunks(_LEGACY_SHAPE, 4).chunk_shape),
1257+
(False, _LEGACY_SHAPE),
12631258
(np.array(7), (7, 7)),
12641259
(np.array([5, 3]), (5, 3)),
12651260
],
12661261
ids=repr,
12671262
)
1268-
def test_legacy_create_v2_chunks(
1269-
chunks: ChunksLike | None, expected: tuple[int, ...] | None
1263+
def test_legacy_create_chunks(
1264+
chunks: ChunksLike | None, expected: tuple[int, ...], zarr_format: ZarrFormat
12701265
) -> None:
1271-
"""The legacy `zarr.create` Zarr format 2 path reads a falsy `chunks` as not given
1272-
and chunks automatically (`expected` is `None`). It never tests the truth value of a
1273-
numpy array with more than one element, which has none: that array is the chunk
1274-
shape."""
1275-
shape = (2**12, 2**12)
1276-
arr = zarr.create(store=MemoryStore(), shape=shape, chunks=chunks, dtype="int32", zarr_format=2)
1277-
assert arr.chunks == (expected or guess_chunks(shape, 4).chunk_shape)
1266+
"""The legacy `zarr.create` reads `chunks` the same way for both Zarr formats: only
1267+
`None` means not given and chunks automatically; `False` is one chunk, and a numpy
1268+
array is the chunk shape."""
1269+
arr = zarr.create(
1270+
store=MemoryStore(),
1271+
shape=_LEGACY_SHAPE,
1272+
chunks=chunks,
1273+
dtype="int32",
1274+
zarr_format=zarr_format,
1275+
)
1276+
assert arr.chunks == expected
12781277
assert all(type(c) is int for c in arr.chunks)
12791278

12801279

1281-
def test_legacy_create_v2_chunks_all_zero_array_raises() -> None:
1282-
"""A numpy `chunks` array with more than one element is given even when every element
1283-
is 0, so the legacy Zarr format 2 path rejects its 0 sizes instead of chunking
1284-
automatically."""
1285-
chunks = np.array([0, 0])
1280+
@pytest.mark.parametrize("zarr_format", [2, 3])
1281+
@pytest.mark.parametrize("chunks", [0, np.int64(0), np.array(0), np.array([0, 0])], ids=repr)
1282+
def test_legacy_create_chunks_rejects_zero(chunks: ChunksLike, zarr_format: ZarrFormat) -> None:
1283+
"""A chunk size of 0 is rejected, not read as "not given"."""
12861284
with pytest.raises(ValueError, match="Chunk size must be positive or -1, got 0"):
12871285
zarr.create(
1288-
store=MemoryStore(), shape=(2**12, 2**12), chunks=chunks, dtype="int32", zarr_format=2
1286+
store=MemoryStore(),
1287+
shape=_LEGACY_SHAPE,
1288+
chunks=chunks,
1289+
dtype="int32",
1290+
zarr_format=zarr_format,
1291+
)
1292+
1293+
1294+
@pytest.mark.parametrize("zarr_format", [2, 3])
1295+
@pytest.mark.parametrize("chunks", [(), [], np.array([]), np.array([0])], ids=repr)
1296+
def test_legacy_create_chunks_rejects_wrong_ndim(
1297+
chunks: ChunksLike, zarr_format: ZarrFormat
1298+
) -> None:
1299+
"""An empty or too-short chunk shape is rejected, not read as "not given"."""
1300+
with pytest.raises(ValueError, match="dimensions but shape has 2 dimensions"):
1301+
zarr.create(
1302+
store=MemoryStore(),
1303+
shape=_LEGACY_SHAPE,
1304+
chunks=chunks,
1305+
dtype="int32",
1306+
zarr_format=zarr_format,
1307+
)
1308+
1309+
1310+
@pytest.mark.parametrize("zarr_format", [2, 3])
1311+
@pytest.mark.parametrize("chunks", [np.False_, np.array(False)], ids=repr)
1312+
def test_legacy_create_chunks_rejects_numpy_bool(
1313+
chunks: ChunksLike, zarr_format: ZarrFormat
1314+
) -> None:
1315+
"""A numpy boolean is not a chunk size, and is not read as "not given"."""
1316+
with pytest.raises(TypeError, match="Chunk specification must be an integer"):
1317+
zarr.create(
1318+
store=MemoryStore(),
1319+
shape=_LEGACY_SHAPE,
1320+
chunks=chunks,
1321+
dtype="int32",
1322+
zarr_format=zarr_format,
12891323
)
12901324

12911325

0 commit comments

Comments
 (0)