Skip to content

Commit 47587bc

Browse files
d-v-bclaude
andcommitted
fix(metadata): parse the Zarr format 2 chunk shape as a regular chunk shape
ArrayV2Metadata read `chunks` with parse_shapelike, which accepts 0, while RegularChunkGridMetadata rejects it. It now uses parse_regular_chunk_shape, as ShardingCodec does, so a chunk size below 1 raises at construction. Stored documents are unaffected: from_dict repairs a stored 0 before the constructor sees it. The shared chunk-shape tests cover both sites again. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent baddb31 commit 47587bc

4 files changed

Lines changed: 41 additions & 100 deletions

File tree

‎changes/4376.bugfix.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of `chunks`: only `None` means not given, as for Zarr format 3. `chunks=False` is now one chunk instead of an automatic chunk shape, numpy arrays work, and `0`, `[]` and `np.array([0])` raise instead of chunking automatically. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array. The `chunks` and `chunk_shape` parameters of `zarr.create` and the `ShardsLike` type now declare every chunk specification the normalizer accepts, including numpy integers and arrays, explicit edge lists and stored chunk grid metadata. The inner chunk shape of `ShardingCodec` is parsed with the chunk normalizer's integer test as a regular chunk shape: every size must be at least 1, so a stored or given inner chunk size of 0 raises a `ValueError` instead of a `ZeroDivisionError` on read, and an explicit list of chunk edges is rejected with a `TypeError`.
1+
Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of `chunks`: only `None` means not given, as for Zarr format 3. `chunks=False` is now one chunk instead of an automatic chunk shape, numpy arrays work, and `0`, `[]` and `np.array([0])` raise instead of chunking automatically. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array. The `chunks` and `chunk_shape` parameters of `zarr.create` and the `ShardsLike` type now declare every chunk specification the normalizer accepts, including numpy integers and arrays, explicit edge lists and stored chunk grid metadata. The chunk shape of `ArrayV2Metadata` and the inner chunk shape of `ShardingCodec` are parsed with the chunk normalizer's integer test as a regular chunk shape: every size must be at least 1, so `ArrayV2Metadata(chunks=(0,))` raises a `ValueError` as the Zarr format 3 regular chunk grid does (a stored Zarr format 2 chunk size of 0 is still repaired on read), a stored inner chunk size of 0 raises a `ValueError` instead of a `ZeroDivisionError` on read, and an explicit list of chunk edges is rejected with a `TypeError`.

‎src/zarr/core/metadata/v2.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333

3434
from zarr.core._json import json_to_buffer
3535
from zarr.core.array_spec import ArrayConfig, ArraySpec
36+
from zarr.core.chunk_grids import parse_regular_chunk_shape
3637
from zarr.core.chunk_key_encodings import parse_separator
3738
from zarr.core.common import (
3839
JSON,
@@ -331,9 +332,10 @@ def parse_compressor(data: object) -> Numcodec | None:
331332

332333

333334
def parse_chunks(chunks: ShapeLike, shape: tuple[int, ...]) -> tuple[int, ...]:
334-
"""Check a chunk shape: one non-negative integer per array axis (see
335-
`parse_shapelike`). Stored chunk sizes of 0 are read by `zarr.core.metadata.repair`."""
336-
chunks_parsed = parse_shapelike(chunks)
335+
"""Check a chunk shape: one integer of at least 1 per array axis, as a regular chunk
336+
shape (see `parse_regular_chunk_shape`). Stored chunk sizes of 0 are read by
337+
`zarr.core.metadata.repair` before they get here."""
338+
chunks_parsed = parse_regular_chunk_shape(chunks)
337339
if len(chunks_parsed) != len(shape):
338340
raise ValueError(
339341
f"The `shape` and `chunks` attributes must have the same length. "

‎tests/test_codecs/test_sharding.py‎

Lines changed: 1 addition & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
import math
33
import pickle
44
import warnings
5-
from typing import TYPE_CHECKING, Any, cast, get_args
5+
from typing import Any, cast, get_args
66
from unittest.mock import AsyncMock
77

88
import numpy as np
@@ -40,9 +40,6 @@
4040
from ..conftest import ArrayRequest
4141
from .test_codecs import _AsyncArrayProxy, order_from_dim
4242

43-
if TYPE_CHECKING:
44-
from zarr.core.common import JSON
45-
4643

4744
def _reads_are_sync(store_mock: AsyncMock) -> bool:
4845
"""True when the partial-shard read for this store+pipeline goes through the
@@ -1396,51 +1393,3 @@ def test_sharding_orthogonal_set_multiple_array_dims(
13961393
expected[ix] = value.reshape(expected[ix].shape)
13971394
assert np.array_equal(a[:], expected)
13981395
assert np.array_equal(a.oindex[selection], value)
1399-
1400-
1401-
@pytest.mark.parametrize(
1402-
("chunk_shape", "expected"),
1403-
[
1404-
((4,), (4,)),
1405-
([4, 2], (4, 2)),
1406-
(4, (4,)),
1407-
(np.int64(4), (4,)),
1408-
((np.int64(4),), (4,)),
1409-
(np.array([4]), (4,)),
1410-
((True,), (1,)),
1411-
(range(4, 5), (4,)),
1412-
((), ()),
1413-
],
1414-
ids=repr,
1415-
)
1416-
def test_sharding_codec_chunk_shape(chunk_shape: Any, expected: tuple[int, ...]) -> None:
1417-
"""The inner chunk shape of a shard is parsed by the chunk normalizer's integer test as
1418-
a regular chunk shape: one integer of at least 1 per dimension, written back as
1419-
given."""
1420-
codec = ShardingCodec(chunk_shape=chunk_shape)
1421-
assert codec.chunk_shape == expected
1422-
assert all(type(size) is int for size in codec.chunk_shape)
1423-
configuration = cast("dict[str, JSON]", codec.to_dict()["configuration"])
1424-
assert configuration["chunk_shape"] == expected
1425-
assert ShardingCodec.from_dict(codec.to_dict()).chunk_shape == expected
1426-
1427-
1428-
@pytest.mark.parametrize("chunk_shape", [(0,), (False,), (-1,), 0, [8, 0]], ids=repr)
1429-
def test_sharding_codec_chunk_shape_rejects_non_positive(chunk_shape: Any) -> None:
1430-
"""An inner chunk size below 1 is rejected, including `-1` and `False`, which have no
1431-
axis length to be resolved against."""
1432-
with pytest.raises(ValueError, match="All chunk sizes must be positive"):
1433-
ShardingCodec(chunk_shape=chunk_shape)
1434-
1435-
1436-
@pytest.mark.parametrize("chunk_shape", [[[4, 4]], [8, [4, 4]], (4.0,), "4"], ids=repr)
1437-
def test_sharding_codec_chunk_shape_rejects_non_integer(chunk_shape: Any) -> None:
1438-
"""An inner chunk shape is regular: an explicit list of chunk edges, which declares a
1439-
rectilinear dimension, is rejected like any other non-integer chunk size."""
1440-
with pytest.raises(TypeError, match="Each chunk size of a regular chunk shape must be"):
1441-
ShardingCodec(chunk_shape=chunk_shape)
1442-
1443-
1444-
def test_sharding_codec_chunk_shape_rejects_non_iterable() -> None:
1445-
with pytest.raises(TypeError, match="must be an integer or an iterable of integers"):
1446-
ShardingCodec(chunk_shape=None) # type: ignore[arg-type]

‎tests/test_metadata/test_repair.py‎

Lines changed: 34 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -254,7 +254,7 @@ def _open_strictly(path: Path, mode: Literal["r", "a", "r+"] = "r") -> AnyArray:
254254

255255
def test_stored_negative_chunk_size_rejected() -> None:
256256
"""No known writer stored a negative chunk size: it is rejected, not repaired."""
257-
with pytest.raises(ValueError, match="^Expected all values to be non-negative"):
257+
with pytest.raises(ValueError, match=r"^All chunk sizes must be positive, got \[-1\]$"):
258258
_read_strictly(_v2_doc([4], [-1]))
259259

260260

@@ -333,7 +333,7 @@ def test_read_invalid_edges_in_rectilinear_grid(
333333
_v3_doc([8], [4], inner=[2.0]),
334334
"got non-integer element(s) (2.0,) at indices (0,)",
335335
),
336-
(_v2_doc([20], [10.0]), "Expected an iterable of integers. Got [10.0] instead."),
336+
(_v2_doc([20], [10.0]), "got non-integer element(s) (10.0,) at indices (0,)"),
337337
(
338338
_rectilinear_doc([8], [[[4, 2.0]]]),
339339
"Dimension 0: RLE repeat count must be an int, got 2.0",
@@ -494,12 +494,19 @@ def test_regular_chunk_grid_rejects_chunk_shape_not_a_sequence(chunk_shape: Any)
494494
RegularChunkGridMetadata(chunk_shape=chunk_shape)
495495

496496

497+
def _sharding_chunk_shape(chunks: Any) -> tuple[tuple[int, ...], object]:
498+
codec = ShardingCodec(chunk_shape=chunks)
499+
configuration = cast("dict[str, JSON]", codec.to_dict()["configuration"])
500+
return codec.chunk_shape, configuration["chunk_shape"]
501+
502+
497503
CHUNK_SHAPE_SITES: dict[str, Callable[[Any], tuple[tuple[int, ...], object]]] = {
498504
"v2": lambda chunks: ((md := _v2_metadata(chunks)).chunks, md.to_dict()["chunks"]),
505+
"sharding-inner": _sharding_chunk_shape,
499506
}
500-
"""`ArrayV2Metadata` reads a chunk shape as an array shape, returning the chunk shape
501-
and the value `to_dict` writes for it. The inner chunk shape of `ShardingCodec` is a
502-
regular chunk shape instead (see `test_sharding_codec_chunk_shape`)."""
507+
"""`ArrayV2Metadata` and `ShardingCodec` read a regular chunk shape (see
508+
`parse_regular_chunk_shape`), returning the chunk shape and the value `to_dict` writes
509+
for it."""
503510

504511

505512
@pytest.mark.parametrize("site", CHUNK_SHAPE_SITES)
@@ -513,37 +520,44 @@ def test_regular_chunk_grid_rejects_chunk_shape_not_a_sequence(chunk_shape: Any)
513520
((np.int64(4),), (4,)),
514521
(np.array([4]), (4,)),
515522
((True,), (1,)),
516-
((0,), (0,)),
517-
((False,), (0,)),
518523
(range(4, 5), (4,)),
519524
],
520525
)
521-
def test_chunk_shape_read_as_array_shape(
522-
site: str, chunks: object, expected: tuple[int, ...]
523-
) -> None:
524-
"""`ArrayV2Metadata` reads its chunk shape as `parse_shapelike`
525-
reads an array shape: an integer or an iterable of non-negative integers, including
526-
NumPy integers and bools. A chunk size of 0 is written back as given; reading a
527-
stored 0 is `zarr.core.metadata.repair`' business."""
526+
def test_regular_chunk_shape(site: str, chunks: object, expected: tuple[int, ...]) -> None:
527+
"""`ArrayV2Metadata` and `ShardingCodec` read their chunk shape with the chunk
528+
normalizer's integer test: an integer or an iterable of integers of at least 1,
529+
including NumPy integers and bools, written back as given. Reading a stored 0 is
530+
`zarr.core.metadata.repair`' business, before the constructors see it."""
528531
parsed, written = CHUNK_SHAPE_SITES[site](chunks)
529532
assert parsed == expected
530533
assert all(type(size) is int for size in parsed)
531534
assert written == expected
532535

533536

534537
@pytest.mark.parametrize("site", CHUNK_SHAPE_SITES)
535-
def test_chunk_shape_read_as_array_shape_rejects_negative(site: str) -> None:
536-
with pytest.raises(ValueError, match="Expected all values to be non-negative"):
537-
CHUNK_SHAPE_SITES[site]((-1,))
538+
@pytest.mark.parametrize("chunks", [(0,), (False,), (-1,), 0])
539+
def test_regular_chunk_shape_rejects_non_positive(site: str, chunks: object) -> None:
540+
"""A chunk size below 1 is rejected, including `-1` and `False`, which have no axis
541+
length to be resolved against here."""
542+
with pytest.raises(ValueError, match="All chunk sizes must be positive"):
543+
CHUNK_SHAPE_SITES[site](chunks)
538544

539545

540546
@pytest.mark.parametrize("site", CHUNK_SHAPE_SITES)
541-
@pytest.mark.parametrize("chunks", [(4.0,), "4", None])
542-
def test_chunk_shape_read_as_array_shape_rejects_non_integer(site: str, chunks: object) -> None:
543-
with pytest.raises(TypeError, match="Expected an"):
547+
@pytest.mark.parametrize("chunks", [(4.0,), "4", [[2, 2]]])
548+
def test_regular_chunk_shape_rejects_non_integer(site: str, chunks: object) -> None:
549+
"""A non-integer chunk size is rejected, and so is an explicit list of chunk edges,
550+
which declares a rectilinear dimension that a regular chunk shape cannot have."""
551+
with pytest.raises(TypeError, match="Each chunk size of a regular chunk shape must be"):
544552
CHUNK_SHAPE_SITES[site](chunks)
545553

546554

555+
@pytest.mark.parametrize("site", CHUNK_SHAPE_SITES)
556+
def test_regular_chunk_shape_rejects_non_iterable(site: str) -> None:
557+
with pytest.raises(TypeError, match="must be an integer or an iterable of integers"):
558+
CHUNK_SHAPE_SITES[site](None)
559+
560+
547561
def _rewrite_doc(path: Path, zarr_format: Literal[2, 3], edit: Any) -> None:
548562
doc_path = path / (".zarray" if zarr_format == 2 else "zarr.json")
549563
doc = json.loads(doc_path.read_text())
@@ -971,30 +985,6 @@ def test_from_dict_keeps_values_that_are_not_json(codec: Any) -> None:
971985
assert isinstance(metadata.codecs[0], ShardingCodec)
972986

973987

974-
@pytest.mark.parametrize(("shape", "expected"), [((0,), (1,)), ((3,), (1,))])
975-
def test_array_from_metadata_with_chunk_size_zero(shape: tuple[int], expected: tuple[int]) -> None:
976-
"""`ArrayV2Metadata` accepts a chunk size of 0, as a stored document may hold it. An
977-
array built from such metadata reads it as the repairs read that document, silently
978-
(no data was read or written under it): `create_hierarchy` stores the metadata as
979-
given and yields such an array, which stores the repair before its first write."""
980-
metadata = ArrayV2Metadata(shape=shape, chunks=(0,), dtype=Int16(), fill_value=0, order="C")
981-
store = MemoryStore()
982-
with warnings.catch_warnings():
983-
warnings.simplefilter("error", ZarrUserWarning)
984-
nodes = dict(zarr.create_hierarchy(store=store, nodes={"a": metadata}))
985-
array = nodes["a"]
986-
assert isinstance(array, zarr.Array)
987-
assert array.chunks == expected
988-
assert json.loads(store._store_dict["a/.zarray"].to_bytes())["chunks"] == [0]
989-
990-
array[...] = 1
991-
992-
# Writing an empty selection stores no chunks, so it stores no metadata either.
993-
resaved = list(expected) if array.size else [0]
994-
assert json.loads(store._store_dict["a/.zarray"].to_bytes())["chunks"] == resaved
995-
np.testing.assert_array_equal(zarr.open_array(store, path="a")[...], np.ones(shape))
996-
997-
998988
def test_array_from_metadata_with_numpy_scalar_codec_configuration() -> None:
999989
"""An array is built from metadata whose codec configuration holds NumPy scalars
1000990
(which are not JSON values), as zarr always built one."""

0 commit comments

Comments
 (0)