Skip to content

Commit 25d390b

Browse files
d-v-bclaude
andauthored
fix(chunk-grids): judge every chunk specification by the chunk normalizer alone (#4376)
* fix: make the chunk normalizer the one judge of a chunk specification Chunk specifications were parsed by `normalize_chunks_nd`, but a separate duck-typed classifier, `_is_rectilinear_chunks`, ran on the raw input first at three sites to decide whether the spec was rectilinear. Two opinions on the same input is the shape of the bug in #4374, and they could disagree: a 0-d numpy array counted as rectilinear because it has `__iter__`. The classifier is gone. Each site normalizes first and asks the resulting `ChunkGrid` (`is_regular`); stored rectilinear metadata passed as `chunks=` counts as rectilinear even when its edges are uniform. The shard resolver sends regular and rectilinear shard specs through the same normalizer. Also fixed on the way: - The legacy v2 branch of `AsyncArray.create` tested `chunks or chunk_shape`, so `zarr.create(chunks=np.array([...]), zarr_format=2)` failed with "truth value of an array is ambiguous". It now uses the `is not None` form the v3 branch already had. - 0-d numpy arrays unwrap to their scalar in both normalizers instead of failing with "len() of unsized object". - A non-integer scalar spec (`2.0`, `np.float64`) raises the normalizer's own TypeError instead of "object has no len()". Assisted-by: ClaudeCode:claude-fable-5-1 * docs: add changelog fragment for #4376 Assisted-by: ClaudeCode:claude-fable-5-1 * refactor(chunk_grids): one integer test for chunk input `_chunk_int` is the normalizer's single integer rule: anything Python's integer protocol accepts (int, numpy integer scalars, 0-d integer arrays), except bool. It replaces the repeated numbers.Integral checks and the two 0-d ndarray unwraps in normalize_chunks_1d / normalize_chunks_nd. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test: keep numpy chunk tests that cover real bugs; pin legacy v2 rectilinear rejection The numpy cases that passed before this PR are dropped; the 0-d array and float cases move into the normalizer's table and error tests. The legacy zarr.create Zarr format 2 path gets a truthiness test and its rectilinear rejection is now asserted with a message match. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: note legacy v2 chunks=0/[] and bool chunk sizes in the #4376 changelog Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(chunk_grids): explicit edge lists stay rectilinear in ChunkGrid.from_sizes `ChunkGrid.from_sizes` collapsed uniform edge lists to `FixedDimension` while `normalize_chunks_1d` keeps them as `VaryingDimension`, so `init_array` needed an `isinstance(chunks, RectilinearChunkGridMetadata)` guard to keep uniform stored rectilinear grids under the Zarr format 2 and sharding restrictions. Both now agree that a list declares a rectilinear dimension, and the guard is gone: the normalized grid is the one judge. Also: `normalize_chunks_nd` and the shard spec are typed with `ChunksLike`; a bool chunk size is reported as not a chunk size; an integral float (`10.0`) is pinned as rejected; `None` reaching the normalizer gets the generic non-integer `TypeError`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(chunk_grids): reject every spelling of a boolean chunk size with one error A bool, np.bool_, or boolean array anywhere in a chunk specification now raises the same TypeError from the normalizer's one integer test, instead of three different errors depending on spelling. Document that a RectilinearChunkGridMetadata of bare integers is read as a regular grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(chunk_grids): keep zarr 3.4.0's reading of bool chunk sizes and falsy v2 chunks (patch release) Nothing in a patch release may reject a chunk specification that zarr 3.4.0 accepted. The normalizer's integer test reads a Python `bool` as the `int` it is again, so `chunks=(True, 5)` and a `True` edge are a chunk size of 1; `chunks=True` and `chunks=None` raise 3.4.0's `ValueError` again. Numpy booleans stay rejected, as they were. The legacy `zarr.create(..., zarr_format=2)` again reads a falsy `chunks` (0, [], False) as not given and chunks automatically; a numpy array is always taken as given, so its truth value is never tested. The `TypeError` for every boolean spelling returns in the next minor release. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs: describe the 4376 changes for a patch release Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(array): read one-element and empty numpy chunk arrays in legacy v2 create as zarr 3.4.0 did The legacy `zarr.create(..., zarr_format=2)` took every numpy array as a given chunk specification, so `np.array(0)`, `np.array(False)` and `np.array([0])` raised instead of auto-chunking as in zarr 3.4.0. A numpy array with more than one element has no truth value and is always given; a shorter one is read by `.any()`, its truth value, which is False when empty, so `np.array([])` auto-chunks like `[]` (as 3.4.0 did with numpy 2.1). The two legacy v2 tests become one table. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(chunk_grids): let operator.index be the only integer test in _chunk_int `operator.index` raises `TypeError` for everything the `SupportsIndex` check rejected, so the check was redundant (identical results over bool, numpy bools and integers, 0-d and 1-d arrays, float, str, bytes, None, list and a custom `__index__` class). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(chunk_grids): read bool chunk sizes as rows of the normalizer table Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(array): name the legacy v2 chunks-given test as _v2_chunks_given Replace the inline conditional expression in `AsyncArray._create` with a small named predicate. Results are identical on every probed input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(array): pin 0-d and all-zero numpy chunks in legacy v2 create Add `np.array(7)` -> `(7, 7)` to `test_legacy_create_v2_chunks` and an error test for `np.array([0, 0])`, killing the mutants that drop either half of `_v2_chunks_given`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(chunk-grids): accept sequences that only define __getitem__ and __len__ The chunk normalizer checked `isinstance(chunks, Iterable)` before calling `list(chunks)`. `Iterable` does not recognize a sequence that implements only `__getitem__` and `__len__`, which `list` accepts and which zarr 3.4.0 took as a chunk specification. The normalizer now calls `list` and turns its `TypeError` into the chunk specification error. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(chunk-grids): reject NumPy booleans as chunk sizes on NumPy before 2.3 NumPy before 2.3 takes a NumPy boolean as an index with a DeprecationWarning, so `operator.index(np.True_)` returned 1 there, and the normalizer read `chunks=np.True_` as a chunk size of 1 instead of rejecting it as zarr 3.4.0 did. The min-deps CI job failed on this. The integer test now rejects a value whose dtype is NumPy bool before calling `operator.index`. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(chunk-grids): declare every accepted chunk form in ShardsLike and zarr.create ShardsLike is now ChunksLike plus the sharding configuration and "auto", so numpy integers and arrays are declared for shards as they are for chunks. zarr.create and AsyncArray._create take ChunksLike for chunks and chunk_shape, matching what the normalizer reads. Tests drop the Any annotations and the type: ignore that worked around the narrower types. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(array): read only None as an absent chunks argument in legacy zarr.create The Zarr format 2 path of AsyncArray._create used the truth value of `chunks` to decide whether it was given, so False, 0, [] and np.array([0]) were silently auto-chunked while Zarr format 3 normalized or rejected them. Both formats now share the None-only `_raw_chunks` and agree on every input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(sharding): parse the inner chunk shape with the chunk normalizer as a regular grid ShardingCodec read its inner chunk_shape with parse_shapelike, which allows 0, so an inner size of 0 was accepted at construction and failed later with ZeroDivisionError. parse_regular_chunk_shape shares the normalizer's integer test (_chunk_int/_chunk_list) and requires every size to be at least 1; with no axis length to resolve against, -1 and False are rejected, and an explicit edge list (a rectilinear dimension) is rejected because the inner grid of a shard is regular. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(metadata): parse the Zarr format 2 chunk shape as a regular chunk shape ArrayV2Metadata read `chunks` with parse_shapelike, which accepts 0, while RegularChunkGridMetadata rejects it. It now uses parse_regular_chunk_shape, as ShardingCodec does, so a chunk size below 1 raises at construction. Stored documents are unaffected: from_dict repairs a stored 0 before the constructor sees it. The shared chunk-shape tests cover both sites again. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(api): describe what zarr.create does with chunks=None, False and True The docstring said True guesses the chunk shape and is the default; None is the default and True raises. The auto-chunking error now tells create_array callers to pass "auto" rather than the None they just passed. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * revert: keep the strict chunk-size changes in #4431, out of this patch PR Reverts 47587bc, baddb31 and ad34ae8, and the error-message part of e547401. Rejecting a chunk size of 0 in ArrayV2Metadata and ShardingCodec, and no longer reading a falsy `chunks` as absent in the legacy Zarr format 2 zarr.create, reject inputs zarr 3.4.0 accepted; this PR targets the 3.4.1 patch release, which must not. #4431 makes the same changes, more strictly, for the next minor release. The widened chunk and shard annotations and the zarr.create docstring fix stay. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent f078bd3 commit 25d390b

9 files changed

Lines changed: 256 additions & 193 deletions

File tree

‎changes/4376.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Chunk specifications are judged only by the chunk normalizer: the separate duck-typed check for rectilinear input is gone, so the Zarr format 2 and sharding restrictions apply to what the normalized grid says was declared. A chunk size is an integer by Python's integer protocol: numpy integer scalars and 0-d integer arrays are accepted wherever a chunk or shard shape is. The legacy `zarr.create(..., zarr_format=2)` no longer tests the truth value of a numpy array given as `chunks`, so numpy arrays work there. A non-integer scalar chunk specification raises a `TypeError` naming the problem. An explicit list of chunk edges declares a rectilinear dimension everywhere, including a stored rectilinear chunk grid passed as `chunks=` whose edges happen to be uniform, so the Zarr format 2 and sharding restrictions treat it the same whether it is given as lists or as metadata, and `zarr.create` stores it as a rectilinear chunk grid, as `create_array` does. A `RectilinearChunkGridMetadata` made only of bare integers declares no explicit edge list, so it is read as the regular grid it describes: it is accepted for Zarr format 2 and as the chunk shape of a sharded array. The `chunks` and `chunk_shape` parameters of `zarr.create` and the `ShardsLike` type now declare every chunk specification the normalizer accepts, including numpy integers and arrays, explicit edge lists and stored chunk grid metadata.

‎src/zarr/api/asynchronous.py‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
from zarr.core.common import (
2525
JSON,
2626
AccessModeLiteral,
27+
ChunksLike,
2728
DimensionNamesLike,
2829
MemoryOrder,
2930
ZarrFormat,
@@ -917,7 +918,7 @@ async def open_group(
917918
async def create(
918919
shape: tuple[int, ...] | int,
919920
*, # Note: this is a change from v2
920-
chunks: tuple[int, ...] | int | bool | None = None,
921+
chunks: ChunksLike | None = None,
921922
dtype: ZDTypeLike | None = None,
922923
compressor: CompressorLike = "auto",
923924
fill_value: Any | None = DEFAULT_FILL_VALUE,
@@ -938,7 +939,7 @@ async def create(
938939
meta_array: Any | None = None, # TODO: need type
939940
attributes: dict[str, JSON] | None = None,
940941
# v3 only
941-
chunk_shape: tuple[int, ...] | int | None = None,
942+
chunk_shape: ChunksLike | None = None,
942943
chunk_key_encoding: (
943944
ChunkKeyEncoding
944945
| tuple[Literal["default"], Literal[".", "/"]]
@@ -958,11 +959,11 @@ async def create(
958959
----------
959960
shape : int or tuple of ints
960961
Array shape.
961-
chunks : int or tuple of ints, optional
962-
Chunk shape. If True, will be guessed from `shape` and `dtype`. If
963-
False, will be set to `shape`, i.e., single chunk for the whole array.
964-
If an int, the chunk size in each dimension will be given by the value
965-
of `chunks`. Default is True.
962+
chunks : ChunksLike, optional
963+
Chunk shape. If None (the default), it is guessed from `shape` and `dtype`. If
964+
False, it is set to `shape`, i.e., a single chunk for the whole array. If an
965+
int, the chunk size in each dimension is given by the value of `chunks`.
966+
`True` is not a chunk shape and raises a `ValueError`.
966967
dtype : str or dtype, optional
967968
NumPy dtype.
968969
compressor : Codec, optional
@@ -1045,7 +1046,7 @@ async def create(
10451046
Not implemented.
10461047
attributes : dict[str, JSON], optional
10471048
A dictionary of user attributes to store with the array.
1048-
chunk_shape : int or tuple of ints, optional
1049+
chunk_shape : ChunksLike, optional
10491050
The shape of the Array's chunks (default is None).
10501051
Zarr format 3 only. Zarr format 2 arrays should use `chunks` instead.
10511052
chunk_key_encoding : ChunkKeyEncoding, optional

‎src/zarr/api/synchronous.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -635,7 +635,7 @@ def create_group(
635635
def create(
636636
shape: tuple[int, ...] | int,
637637
*, # Note: this is a change from v2
638-
chunks: tuple[int, ...] | int | bool | None = None,
638+
chunks: ChunksLike | None = None,
639639
dtype: ZDTypeLike | None = None,
640640
compressor: CompressorLike = "auto",
641641
fill_value: Any | None = DEFAULT_FILL_VALUE, # TODO: need type
@@ -656,7 +656,7 @@ def create(
656656
meta_array: Any | None = None, # TODO: need type
657657
attributes: dict[str, JSON] | None = None,
658658
# v3 only
659-
chunk_shape: tuple[int, ...] | int | None = None,
659+
chunk_shape: ChunksLike | None = None,
660660
chunk_key_encoding: (
661661
ChunkKeyEncoding
662662
| tuple[Literal["default"], Literal[".", "/"]]
@@ -676,11 +676,11 @@ def create(
676676
----------
677677
shape : int or tuple of ints
678678
Array shape.
679-
chunks : int or tuple of ints, optional
680-
Chunk shape. If True, will be guessed from `shape` and `dtype`. If
681-
False, will be set to `shape`, i.e., single chunk for the whole array.
682-
If an int, the chunk size in each dimension will be given by the value
683-
of `chunks`. Default is True.
679+
chunks : ChunksLike, optional
680+
Chunk shape. If None (the default), it is guessed from `shape` and `dtype`. If
681+
False, it is set to `shape`, i.e., a single chunk for the whole array. If an
682+
int, the chunk size in each dimension is given by the value of `chunks`.
683+
`True` is not a chunk shape and raises a `ValueError`.
684684
dtype : str or dtype, optional
685685
NumPy dtype.
686686
compressor : Codec, optional
@@ -763,7 +763,7 @@ def create(
763763
Not implemented.
764764
attributes : dict[str, JSON], optional
765765
A dictionary of user attributes to store with the array.
766-
chunk_shape : int or tuple of ints, optional
766+
chunk_shape : ChunksLike, optional
767767
The shape of the Array's chunks (default is None).
768768
Zarr format 3 only. Zarr format 2 arrays should use `chunks` instead.
769769
chunk_key_encoding : ChunkKeyEncoding, optional

‎src/zarr/core/array.py‎

Lines changed: 35 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,6 @@
4545
ChunkGrid,
4646
_is_auto,
4747
_is_keep,
48-
_is_rectilinear_chunks,
4948
guess_chunks,
5049
normalize_chunks_nd,
5150
resolve_outer_and_inner_chunks,
@@ -352,6 +351,18 @@ def _array_metadata_dict_v3(zarr_json_bytes: Buffer) -> dict[str, JSON]:
352351
return metadata_dict
353352

354353

354+
def _v2_chunks_given(chunks: object) -> bool:
355+
"""Whether the legacy Zarr format 2 `chunks` argument is given.
356+
357+
A falsy `chunks` (such as `None`, 0, `[]` or `False`) is read as not given. A numpy
358+
array with more than one element has no truth value and is always given; a shorter
359+
one is given when its element is nonzero (an empty array is not given).
360+
"""
361+
if isinstance(chunks, np.ndarray):
362+
return chunks.size > 1 or bool(chunks.any())
363+
return bool(chunks)
364+
365+
355366
@dataclass(frozen=True)
356367
class AsyncArray[T_ArrayMetadata: (ArrayV2Metadata, ArrayV3Metadata)]:
357368
"""
@@ -431,7 +442,7 @@ async def _create(
431442
fill_value: Any | None = DEFAULT_FILL_VALUE,
432443
attributes: dict[str, JSON] | None = None,
433444
# v3 only
434-
chunk_shape: ShapeLike | None = None,
445+
chunk_shape: ChunksLike | None = None,
435446
chunk_key_encoding: (
436447
ChunkKeyEncodingLike
437448
| tuple[Literal["default"], Literal[".", "/"]]
@@ -441,7 +452,7 @@ async def _create(
441452
codecs: Iterable[Codec | dict[str, JSON]] | None = None,
442453
dimension_names: DimensionNamesLike = None,
443454
# v2 only
444-
chunks: ShapeLike | None = None,
455+
chunks: ChunksLike | None = None,
445456
dimension_separator: Literal[".", "/"] | None = None,
446457
order: MemoryOrder | None = None,
447458
filters: Iterable[dict[str, JSON] | Numcodec] | None = None,
@@ -518,17 +529,16 @@ async def _create(
518529
)
519530
if dimension_names is not None:
520531
raise ValueError("dimension_names cannot be used for arrays with zarr_format 2.")
521-
if _is_rectilinear_chunks(_raw_chunks):
522-
raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
523-
524532
item_size = 1
525533
if isinstance(dtype_parsed, HasItemSize):
526534
item_size = dtype_parsed.item_size
527-
_raw = chunks or chunk_shape
528-
if _raw is None:
535+
_raw_v2 = chunks if _v2_chunks_given(chunks) else chunk_shape
536+
if _raw_v2 is None:
529537
outer_chunks = guess_chunks(shape, item_size)
530538
else:
531-
outer_chunks = normalize_chunks_nd(_raw, shape)
539+
outer_chunks = normalize_chunks_nd(_raw_v2, shape)
540+
if not outer_chunks.is_regular:
541+
raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
532542
_chunks = outer_chunks.chunk_shape
533543

534544
if order is None:
@@ -4163,13 +4173,8 @@ class ShardsConfigParam(TypedDict):
41634173
index_location: IndexLocation | None
41644174

41654175

4166-
type ShardsLike = (
4167-
tuple[int, ...]
4168-
| Sequence[int | Sequence[int]]
4169-
| ChunkGridMetadata
4170-
| ShardsConfigParam
4171-
| Literal["auto"]
4172-
)
4176+
# A shard shape is given in any form a chunk shape is, or as a sharding configuration.
4177+
type ShardsLike = ChunksLike | ShardsConfigParam | Literal["auto"]
41734178

41744179

41754180
async def from_array(
@@ -4570,8 +4575,17 @@ async def init_array(
45704575
chunk_key_encoding, zarr_format=zarr_format
45714576
)
45724577

4573-
# Validate rectilinear chunks constraints
4574-
if _is_rectilinear_chunks(chunks):
4578+
# Normalize the user's chunks into a canonical ChunkGrid
4579+
4580+
if _is_auto(chunks):
4581+
max_bytes = None if shards is None else SHARDED_INNER_CHUNK_MAX_BYTES
4582+
chunks_normalized = guess_chunks(shape_parsed, item_size, max_bytes=max_bytes)
4583+
else:
4584+
chunks_normalized = normalize_chunks_nd(chunks, shape_parsed)
4585+
4586+
# Validate rectilinear chunks constraints. The normalized grid is the one
4587+
# judge of what the user declared.
4588+
if not chunks_normalized.is_regular:
45754589
if zarr_format == 2:
45764590
raise ValueError("Zarr format 2 does not support rectilinear chunk grids.")
45774591
if shards is not None:
@@ -4581,14 +4595,6 @@ async def init_array(
45814595
"chunks=(inner_size, ...), shards=[[shard_sizes], ...]"
45824596
)
45834597

4584-
# Normalize the user's chunks into a canonical ChunkGrid
4585-
4586-
if _is_auto(chunks):
4587-
max_bytes = None if shards is None else SHARDED_INNER_CHUNK_MAX_BYTES
4588-
chunks_normalized = guess_chunks(shape_parsed, item_size, max_bytes=max_bytes)
4589-
else:
4590-
chunks_normalized = normalize_chunks_nd(chunks, shape_parsed)
4591-
45924598
# Resolve chunks + shards into outer_chunks (grid metadata) and
45934599
# inner (sub-chunk structure for ShardingCodec, None if no sharding)
45944600
outer_chunks, inner = resolve_outer_and_inner_chunks(
@@ -4910,10 +4916,9 @@ def _stored_rectilinear_grid_or_none(
49104916
"""The *stored* rectilinear chunk grid, or None if the stored grid is regular
49114917
(in which case `.chunks` and `.shards` are defined).
49124918
4913-
Dispatches on the stored metadata, not the runtime ``ChunkGrid``: the
4914-
runtime grid collapses a rectilinear dimension whose edges happen to be
4915-
uniform to a ``FixedDimension`` as an optimization, so it can report regular
4916-
for an array whose stored metadata — and therefore `.chunks` — is
4919+
Dispatches on the stored metadata, not the runtime `ChunkGrid`: a
4920+
rectilinear grid whose dimensions are all bare-int step sizes is regular at
4921+
runtime, while its stored metadata — and therefore `.chunks` — is
49174922
rectilinear. Zarr format 2 grids are always regular.
49184923
"""
49194924
if isinstance(metadata, ArrayV3Metadata) and isinstance(

0 commit comments

Comments
 (0)