Skip to content

Commit 8f1320b

Browse files
committed
Merge branch 'up/4334' into up/4377
2 parents d43470f + 9e07aaf commit 8f1320b

5 files changed

Lines changed: 241 additions & 115 deletions

File tree

‎changes/4334.bugfix.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ A chunk edge length is now always at least 1, while an array extent may be 0. Ev
44

55
Stored metadata with a regular chunk size of 0 or JSON `false` (Zarr format 2 `chunks`, Zarr format 3 `regular` `chunk_shape`) is now read as chunk size 1 (the inner chunk size for sharded arrays, which previously failed to open), however long the axis is: that is the chunk size `chunks=-1` gives on a zero-length axis, and it does not depend on how far the axis has grown since. zarr-python wrote such sizes for arrays created with a zero-length axis until 3.4, and, from 2.18.7 to 3.2.1, for an explicit chunk size of 0 or `False` on an axis of any length, as in `chunks=(0,)` (those arrays could store no data). On a zero-length axis this opens silently, as before, and appending to it stores chunks of size 1. On an axis of positive length it opens with a `ZarrUserWarning`, which says that the axis holds only the fill value and how to store valid metadata: `array.update_attributes({})`, then `zarr.consolidate_metadata` if the metadata is consolidated. JSON `true`, as zarr-python 3.0 and 3.2 wrote for a chunk size of `True`, is read as 1, silently, in the chunk grid (regular, and the explicit edges and run-length encoded sizes of a rectilinear one) and in the inner chunk shape of every sharding codec, nested or not. A stored rectilinear chunk grid whose edge lengths are integral JSON floats (`[[4.0, 2]]`), as zarr-python 3.2 wrote for float edges, is read with those edges as integers, silently; a float anywhere no release wrote one (a regular chunk shape, the inner chunk shape of a sharding codec, a run-length repeat count, an edge below 1) is rejected.
66

7-
Writing data to an array read from such metadata (other than an empty selection) first stores the upgrade of the metadata the store then holds, unless another writer has stored valid metadata since, so that other readers find the chunks written; in a `ZipStore` this adds a second entry for the metadata document, as every metadata update does. If the metadata the store then holds lays out chunks differently (another writer stored a different chunk size since, say), the write raises a `ValueError` asking to reopen the array, and stores nothing. Apart from the array's own metadata writes (`update_attributes`, `resize`), which store the upgrade as before, this is the only operation that stores it: a group's consolidated metadata keeps such an array's metadata as it was stored (`zarr.consolidate_metadata` copies it as the array's own document holds it) until the array stores its upgrade through the group's handle, and changing a group reads and writes no metadata of its members.
7+
Writing data to an array whose stored metadata holds a regular chunk size of 0 or `false` (other than an empty selection) first stores the upgrade of the metadata the store then holds, unless another writer has stored valid metadata since, so that other readers find the chunks written (a `true` chunk size or an integral float edge is read as the value it equals, so a write stores no metadata for it); in a `ZipStore` this adds a second entry for the metadata document, as every metadata update does. If the metadata the store then holds lays out chunks differently (another writer stored a different chunk size since, say), the write raises a `ValueError` asking to reopen the array, and stores nothing. Apart from the array's own metadata writes (`update_attributes`, `resize`), which store the upgrade as before, this is the only operation that stores it: a group's consolidated metadata keeps such an array's metadata as it was stored (`zarr.consolidate_metadata` copies it as the array's own document holds it) until the array stores its upgrade through the group's handle, and changing a group reads and writes no metadata of its members.

‎src/zarr/core/array.py‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1642,15 +1642,19 @@ async def _store_upgraded_document(self) -> None:
16421642
"""Store the upgrade of this array's current stored document, if it needs one,
16431643
before chunks are written under this handle's metadata.
16441644
1645-
Only for metadata read from a document that had to be upgraded (see
1645+
Only for metadata read from a document whose upgrade moves chunks (see
16461646
`zarr.core.metadata.upgrades`). The document is read again, because the store may
16471647
hold a newer one than this handle's metadata. If that one lays out chunks
1648-
differently (the array was resized since by software that kept the invalid chunk
1649-
size), this handle would write chunks no reader finds, so it raises and stores
1650-
nothing. If it needs no upgrade (the array was re-saved since, possibly by another
1651-
implementation), it is left as written; if there is none, there is nothing to
1652-
upgrade. Storing the same upgrade twice is harmless, so concurrent callers need
1653-
no coordination.
1648+
differently (another writer stored a different chunk size since), this handle
1649+
would write chunks no reader finds, so it raises and stores nothing. If it needs
1650+
no upgrade (the array was re-saved since, possibly by another implementation), it
1651+
is left as written; if there is none, there is nothing to upgrade.
1652+
1653+
Storing the same upgrade twice is harmless, so handles that write chunks
1654+
concurrently need no coordination. The read and the store are not one atomic
1655+
step, though: a metadata write by another handle between them (a resize, an
1656+
attribute update) is replaced by the upgrade of the document read before it, as
1657+
with any two metadata writes that race.
16541658
"""
16551659
if self.metadata._stored_document is None:
16561660
return

‎src/zarr/core/metadata/io.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ def parse_stored_array(documents: Mapping[str, Buffer], zarr_format: ZarrFormat)
132132
else:
133133
raise ArrayNotFoundError(f"No Zarr format {zarr_format} array metadata document.")
134134
upgraded, readings = upgrade_array_document(stored, zarr_format)
135-
return mark_upgraded(parse_array_metadata(dict(upgraded)), stored, [None] * len(readings), None)
135+
return mark_upgraded(parse_array_metadata(dict(upgraded)), stored, readings, None, warn=False)
136136

137137

138138
def _build_parents(store_path: StorePath, zarr_format: ZarrFormat) -> dict[str, GroupMetadata]:

0 commit comments

Comments
 (0)