You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e75bcf9
Browse filesBrowse the repository at this point in the historyBrowse files
Read-only support for Zarr V3 rectilinear (variable-size) chunks (#11592)
* Support reading Zarr V3 rectilinear (variable-sized) chunks
Extracts the read-only half of #11279 (by Max Jones) into its own
change. The read path just needs to catch the NotImplementedError
that zarr_array.chunks raises for a rectilinear grid and fall back to
write_chunk_sizes for the per-dimension chunk tuples.
Write support (#11279) still has open correctness issues flagged in
review -- align_chunks/safe_chunks validation reuses uniform-grid
logic that doesn't handle partial chunks or region-start alignment --
so it's left for a separate follow-up PR rather than gating this on
resolving those.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Use read_chunk_sizes instead of write_chunk_sizes for rectilinear reads
Addresses review feedback on #11279 (same line of code, carried over
in the split): write_chunk_sizes gives outer/shard sizes, but .chunks
(used in the regular-grid branch just above) gives the inner chunk
shape when sharding is enabled. read_chunk_sizes keeps the two
branches consistent for sharded rectilinear arrays.
#11279 (comment)#11279 (comment)
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Add tests for mixed regular/rectilinear dims and dask-backed reads
Mirrors coverage from #11279 that was missing here: a dimension mix
of regular and variable-length chunks, and reading into dask arrays
(checking the dask array's own .chunks, not just Dataset.chunks).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Organize rectilinear chunk read tests into a parametrized class
Groups the four tests into TestZarrRectilinearChunksRead, sharing the
zarr-array-creation helper and parametrizing the two shape/chunk
cases (pure rectilinear, mixed regular+rectilinear) across a plain
read test and a dask-backed read test.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Fix rectilinear test expectations for zarr's actual regular-grid check
CI failed on test-py314-no-dask: my local zarr (3.2.1) let .chunks
return a compact per-dim value (2, (5, 10, 5)) for a mixed
regular/rectilinear array without raising, but zarr 3.3.0 (what CI
installs) correctly raises NotImplementedError there per the
docstring, so xarray falls back to read_chunk_sizes and reports the
fully-expanded form (2, 2, 2), (5, 10, 5)) instead.
Verified against zarr 3.3.0 locally and updated the expected values
to match the documented/correct behavior rather than the older
version's quirk. Also collapsed the parametrize down to a single
expected_chunks field, since it's the same fully-expanded tuple in
both the encoding check and the dask-chunks check.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Handle rectilinear shard grids when reading (not just rectilinear chunks)
zarr_array.shards raises NotImplementedError for a rectilinear shard
grid the same way zarr_array.chunks does for a rectilinear chunk
grid, but only the latter was handled. Regular inner chunks with
rectilinear shards (e.g. chunks=(1,), shards=((1, 2),)) crashed
xr.open_zarr with an uncaught NotImplementedError.
Falls back to write_chunk_sizes (the outer/storage-level sizes,
matching what "shards" means) the same way the chunks fallback uses
read_chunk_sizes (the inner/read-level sizes).
Reported at #11592 (comment)
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Address review feedback: PR reference, error message, region-write docs
1. whats-new.rst cited #11279 (the write-side PR) instead of this
PR (#11592); fixed to reference both.
2/3. Trying to write a Dataset read from a rectilinear-chunked store
(via round-trip to_zarr, or a region write) hit a generic "must be
an int or a tuple of ints" TypeError that gave no indication this
was about rectilinear chunks or how to work around it. Both cases
go through the same _determine_zarr_chunks code path, so a single
fix covers both: the error now names rectilinear chunks and states
the two-step workaround (clear the chunk encoding *and* rechunk to
a uniform size -- verified both are actually needed). Documented
the region-write limitation in the rectilinear chunks docs note.
4. CI coverage: confirmed this is an upstream constraint, not
something to fix here -- zarr>=3.2 requires Python>=3.12
(pypi.org/pypi/zarr/3.2.0/json), so xarray's Python-3.11-pinned
environments can never resolve a zarr new enough to exercise this
feature and correctly skip via requires_zarr_rectilinear_chunks.
#11592 (comment) (Ian Hunt-Isaak)
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Validate encoding['shards'] on write; it bypassed the chunks guard
Investigated the claim that reading a rectilinear-shard store and
writing it back "succeeds cleanly" with valid rectilinear output.
Traced zarr-python's ArrayV3Metadata.chunks/.shards properties: both
gate on the identical `isinstance(self.chunk_grid,
RegularChunkGridMetadata)` check, so for a store actually read through
our open_store_variable fallback, encoding["chunks"] and
encoding["shards"] always agree -- if the shard grid is genuinely
rectilinear, both raise NotImplementedError together, and our
existing encoding["chunks"] guard already catches it (verified: a
round-trip from a real rectilinear-sharded store still raises
TypeError, not a silent write).
The part that *is* real: encoding["shards"] is passed straight
through to zarr_group.create() in _create_new_array with no
validation at all, unlike encoding["chunks"] (_determine_zarr_chunks).
Confirmed this is pre-existing on main, unrelated to anything in this
PR: hand-setting ds['var'].encoding = {"chunks": (10,), "shards":
((20, 10, 30),)} and calling to_zarr() writes a genuine rectilinear
shard grid on main today, with no rectilinear reading involved at all.
Added the same style of validation for encoding["shards"] that
encoding["chunks"] already has, plus a regression test reproducing
the hand-set-encoding case.
#11592 (comment)
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Fix regression: int encoding['shards'] crashed the new write guard
any(not isinstance(x, int) for x in shards) iterated shards without
checking it was iterable first. A bare int shards value (applies to
every dimension, same convention as encoding['chunks']) crashed with
TypeError: 'int' object is not iterable instead of writing normally.
Guard it the same way _determine_zarr_chunks already guards chunks:
skip validation when shards is an int.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
* Validate encoding before resizing an existing array on append
extract_zarr_variable_encoding ran after zarr_array.resize() in
set_variables, so if it raised (as it now does for rectilinear chunk
encodings, which xarray can't write yet) the store was left with a grown
array whose new region was never written. Move the encoding extraction
ahead of the resize; it only depends on the variable, not on the region
or the existing array.
Co-authored-by: Claude <[email protected]>
* Only flag encoding['shards'] as rectilinear when it nests sequences
The new shards guard iterated whatever it was given and rejected any
non-int element. zarr also accepts shards="auto" (which worked before
this branch) and a config dict, both of which iterate to non-int values
and so hit a misleading "looks like a rectilinear shard grid" error.
Restrict the check to a list/tuple containing lists/tuples, mirroring the
chunks guard, and cover "auto" in the regression test.
Co-authored-by: Claude <[email protected]>
* Normalise rectilinear chunk encodings across zarr-python versions
What zarr-python reports for an array whose chunk grid is regular along
only some dimensions, or for regular inner chunks inside rectilinear
shards, differs by version: 3.2 returns a mixed tuple like
(2, (5, 10, 5)) from Array.chunks, 3.3 raises and read_chunk_sizes
expands the regular dimension to (2, 2, 2), and 3.4 returns the compact
inner chunk shape for sharded arrays where 3.3 raises. The tests were
pinned to 3.3 and failed on 3.2.x (the locally installed version) and
3.4.0 (what unpinned CI environments resolve to).
Compact zarr's per-chunk size listings back to a single int along every
dimension that describes a regular grid, so encoding['chunks'] and
encoding['shards'] come out the same regardless of zarr version: an int
per regular dimension, a tuple of sizes per rectilinear one. Test
expectations updated to that representation, with the fully expanded
form kept for the dask-chunks assertion.
zarr-python 3.2.x also crashes on a bare-int shards spec inside its own
metadata parsing, so that test case is skipped below 3.3.
Co-authored-by: Claude <[email protected]>
* Clarify rectilinear write errors for region and append writes
The messages told users to clear the variable's chunk/shard encoding, but
for writes into an existing array (region= or append_dim=) the store's
own encoding is re-applied to the variable, so that advice cannot work.
Say explicitly that such writes are unsupported and that the workaround
only applies when writing to a new array.
Co-authored-by: Claude <[email protected]>
* Expand a bare int encoding['shards'] to a per-dimension tuple
zarr-python 3.2.x crashes on an int shard spec ("'int' object is not
iterable" from its metadata parsing), so expand it to one size per
dimension in extract_zarr_variable_encoding, as zarr itself does for
chunks. This keeps the documented zarr >= 3.2 minimum working without a
version skip in the tests. As a side effect the dask/shard alignment
checks in set_variables, which only run for a tuple spec, now also cover
int shard specs.
Co-authored-by: Claude <[email protected]>
* Shorten comments and test docstrings
Co-authored-by: Claude <[email protected]>
* Share one helper for the rectilinear chunks/shards write errors
Co-authored-by: Claude <[email protected]>
* Fix opening a rectilinear array with a zero-length dimension into dask
zarr's read_chunk_sizes lists no chunks at all along a zero-length
dimension, and passing that empty tuple through as the preferred chunk
made dask raise "Empty tuples are not allowed in chunks" on zarr >= 3.3.
Report it as (0,) instead, which is what dask itself uses.
Co-authored-by: Claude <[email protected]>
* Clarify that rectilinear reads don't raise the minimum zarr version
Co-authored-by: Claude <[email protected]>
* Mark zero-length rectilinear test as requiring dask
Co-authored-by: Claude <[email protected]>
---------
Co-authored-by: Claude Sonnet 5 <[email protected]>
0 commit comments