Skip to content

Commit e75bcf9

Browse files
TomNicholasclaude
andauthored
Read-only support for Zarr V3 rectilinear (variable-size) chunks (#11592)
* Support reading Zarr V3 rectilinear (variable-sized) chunks Extracts the read-only half of #11279 (by Max Jones) into its own change. The read path just needs to catch the NotImplementedError that zarr_array.chunks raises for a rectilinear grid and fall back to write_chunk_sizes for the per-dimension chunk tuples. Write support (#11279) still has open correctness issues flagged in review -- align_chunks/safe_chunks validation reuses uniform-grid logic that doesn't handle partial chunks or region-start alignment -- so it's left for a separate follow-up PR rather than gating this on resolving those. Co-Authored-By: Claude Sonnet 5 <[email protected]> * Use read_chunk_sizes instead of write_chunk_sizes for rectilinear reads Addresses review feedback on #11279 (same line of code, carried over in the split): write_chunk_sizes gives outer/shard sizes, but .chunks (used in the regular-grid branch just above) gives the inner chunk shape when sharding is enabled. read_chunk_sizes keeps the two branches consistent for sharded rectilinear arrays. #11279 (comment) #11279 (comment) Co-Authored-By: Claude Sonnet 5 <[email protected]> * Add tests for mixed regular/rectilinear dims and dask-backed reads Mirrors coverage from #11279 that was missing here: a dimension mix of regular and variable-length chunks, and reading into dask arrays (checking the dask array's own .chunks, not just Dataset.chunks). Co-Authored-By: Claude Sonnet 5 <[email protected]> * Organize rectilinear chunk read tests into a parametrized class Groups the four tests into TestZarrRectilinearChunksRead, sharing the zarr-array-creation helper and parametrizing the two shape/chunk cases (pure rectilinear, mixed regular+rectilinear) across a plain read test and a dask-backed read test. Co-Authored-By: Claude Sonnet 5 <[email protected]> * Fix rectilinear test expectations for zarr's actual regular-grid check CI failed on test-py314-no-dask: my local zarr (3.2.1) let .chunks return a compact per-dim value (2, (5, 10, 5)) for a mixed regular/rectilinear array without raising, but zarr 3.3.0 (what CI installs) correctly raises NotImplementedError there per the docstring, so xarray falls back to read_chunk_sizes and reports the fully-expanded form (2, 2, 2), (5, 10, 5)) instead. Verified against zarr 3.3.0 locally and updated the expected values to match the documented/correct behavior rather than the older version's quirk. Also collapsed the parametrize down to a single expected_chunks field, since it's the same fully-expanded tuple in both the encoding check and the dask-chunks check. Co-Authored-By: Claude Sonnet 5 <[email protected]> * Handle rectilinear shard grids when reading (not just rectilinear chunks) zarr_array.shards raises NotImplementedError for a rectilinear shard grid the same way zarr_array.chunks does for a rectilinear chunk grid, but only the latter was handled. Regular inner chunks with rectilinear shards (e.g. chunks=(1,), shards=((1, 2),)) crashed xr.open_zarr with an uncaught NotImplementedError. Falls back to write_chunk_sizes (the outer/storage-level sizes, matching what "shards" means) the same way the chunks fallback uses read_chunk_sizes (the inner/read-level sizes). Reported at #11592 (comment) Co-Authored-By: Claude Sonnet 5 <[email protected]> * Address review feedback: PR reference, error message, region-write docs 1. whats-new.rst cited #11279 (the write-side PR) instead of this PR (#11592); fixed to reference both. 2/3. Trying to write a Dataset read from a rectilinear-chunked store (via round-trip to_zarr, or a region write) hit a generic "must be an int or a tuple of ints" TypeError that gave no indication this was about rectilinear chunks or how to work around it. Both cases go through the same _determine_zarr_chunks code path, so a single fix covers both: the error now names rectilinear chunks and states the two-step workaround (clear the chunk encoding *and* rechunk to a uniform size -- verified both are actually needed). Documented the region-write limitation in the rectilinear chunks docs note. 4. CI coverage: confirmed this is an upstream constraint, not something to fix here -- zarr>=3.2 requires Python>=3.12 (pypi.org/pypi/zarr/3.2.0/json), so xarray's Python-3.11-pinned environments can never resolve a zarr new enough to exercise this feature and correctly skip via requires_zarr_rectilinear_chunks. #11592 (comment) (Ian Hunt-Isaak) Co-Authored-By: Claude Sonnet 5 <[email protected]> * Validate encoding['shards'] on write; it bypassed the chunks guard Investigated the claim that reading a rectilinear-shard store and writing it back "succeeds cleanly" with valid rectilinear output. Traced zarr-python's ArrayV3Metadata.chunks/.shards properties: both gate on the identical `isinstance(self.chunk_grid, RegularChunkGridMetadata)` check, so for a store actually read through our open_store_variable fallback, encoding["chunks"] and encoding["shards"] always agree -- if the shard grid is genuinely rectilinear, both raise NotImplementedError together, and our existing encoding["chunks"] guard already catches it (verified: a round-trip from a real rectilinear-sharded store still raises TypeError, not a silent write). The part that *is* real: encoding["shards"] is passed straight through to zarr_group.create() in _create_new_array with no validation at all, unlike encoding["chunks"] (_determine_zarr_chunks). Confirmed this is pre-existing on main, unrelated to anything in this PR: hand-setting ds['var'].encoding = {"chunks": (10,), "shards": ((20, 10, 30),)} and calling to_zarr() writes a genuine rectilinear shard grid on main today, with no rectilinear reading involved at all. Added the same style of validation for encoding["shards"] that encoding["chunks"] already has, plus a regression test reproducing the hand-set-encoding case. #11592 (comment) Co-Authored-By: Claude Sonnet 5 <[email protected]> * Fix regression: int encoding['shards'] crashed the new write guard any(not isinstance(x, int) for x in shards) iterated shards without checking it was iterable first. A bare int shards value (applies to every dimension, same convention as encoding['chunks']) crashed with TypeError: 'int' object is not iterable instead of writing normally. Guard it the same way _determine_zarr_chunks already guards chunks: skip validation when shards is an int. Co-Authored-By: Claude Sonnet 5 <[email protected]> * Validate encoding before resizing an existing array on append extract_zarr_variable_encoding ran after zarr_array.resize() in set_variables, so if it raised (as it now does for rectilinear chunk encodings, which xarray can't write yet) the store was left with a grown array whose new region was never written. Move the encoding extraction ahead of the resize; it only depends on the variable, not on the region or the existing array. Co-authored-by: Claude <[email protected]> * Only flag encoding['shards'] as rectilinear when it nests sequences The new shards guard iterated whatever it was given and rejected any non-int element. zarr also accepts shards="auto" (which worked before this branch) and a config dict, both of which iterate to non-int values and so hit a misleading "looks like a rectilinear shard grid" error. Restrict the check to a list/tuple containing lists/tuples, mirroring the chunks guard, and cover "auto" in the regression test. Co-authored-by: Claude <[email protected]> * Normalise rectilinear chunk encodings across zarr-python versions What zarr-python reports for an array whose chunk grid is regular along only some dimensions, or for regular inner chunks inside rectilinear shards, differs by version: 3.2 returns a mixed tuple like (2, (5, 10, 5)) from Array.chunks, 3.3 raises and read_chunk_sizes expands the regular dimension to (2, 2, 2), and 3.4 returns the compact inner chunk shape for sharded arrays where 3.3 raises. The tests were pinned to 3.3 and failed on 3.2.x (the locally installed version) and 3.4.0 (what unpinned CI environments resolve to). Compact zarr's per-chunk size listings back to a single int along every dimension that describes a regular grid, so encoding['chunks'] and encoding['shards'] come out the same regardless of zarr version: an int per regular dimension, a tuple of sizes per rectilinear one. Test expectations updated to that representation, with the fully expanded form kept for the dask-chunks assertion. zarr-python 3.2.x also crashes on a bare-int shards spec inside its own metadata parsing, so that test case is skipped below 3.3. Co-authored-by: Claude <[email protected]> * Clarify rectilinear write errors for region and append writes The messages told users to clear the variable's chunk/shard encoding, but for writes into an existing array (region= or append_dim=) the store's own encoding is re-applied to the variable, so that advice cannot work. Say explicitly that such writes are unsupported and that the workaround only applies when writing to a new array. Co-authored-by: Claude <[email protected]> * Expand a bare int encoding['shards'] to a per-dimension tuple zarr-python 3.2.x crashes on an int shard spec ("'int' object is not iterable" from its metadata parsing), so expand it to one size per dimension in extract_zarr_variable_encoding, as zarr itself does for chunks. This keeps the documented zarr >= 3.2 minimum working without a version skip in the tests. As a side effect the dask/shard alignment checks in set_variables, which only run for a tuple spec, now also cover int shard specs. Co-authored-by: Claude <[email protected]> * Shorten comments and test docstrings Co-authored-by: Claude <[email protected]> * Share one helper for the rectilinear chunks/shards write errors Co-authored-by: Claude <[email protected]> * Fix opening a rectilinear array with a zero-length dimension into dask zarr's read_chunk_sizes lists no chunks at all along a zero-length dimension, and passing that empty tuple through as the preferred chunk made dask raise "Empty tuples are not allowed in chunks" on zarr >= 3.3. Report it as (0,) instead, which is what dask itself uses. Co-authored-by: Claude <[email protected]> * Clarify that rectilinear reads don't raise the minimum zarr version Co-authored-by: Claude <[email protected]> * Mark zero-length rectilinear test as requiring dask Co-authored-by: Claude <[email protected]> --------- Co-authored-by: Claude Sonnet 5 <[email protected]>
1 parent c125355 commit e75bcf9

5 files changed

Lines changed: 359 additions & 16 deletions

File tree

‎doc/user-guide/io.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1244,6 +1244,35 @@ ds.to_zarr(
12441244
The number of chunks on Tair matches our dask chunks, while there is now only a single
12451245
chunk in the directory stores of each coordinate.
12461246

1247+
(io.zarr.rectilinear-chunks)=
1248+
1249+
### Variable-sized (rectilinear) chunks
1250+
1251+
Zarr v3 supports _rectilinear_ chunk grids, where chunk sizes vary along one
1252+
or more dimensions. This is useful when natural data boundaries (yearly
1253+
chunks of a daily time series, per-tile spatial extents) don't align to a
1254+
regular grid. Reading such arrays requires `zarr-python >= 3.2`, with the
1255+
experimental feature enabled:
1256+
1257+
```python
1258+
import zarr
1259+
1260+
with zarr.config.set({"array.rectilinear_chunks": True}):
1261+
roundtrip = xr.open_zarr("rectilinear.zarr", zarr_format=3)
1262+
roundtrip.chunks["x"] # e.g. (10, 20, 30)
1263+
```
1264+
1265+
```{note}
1266+
xarray can currently only *read* rectilinear-chunked Zarr V3 arrays, not
1267+
write them. Rectilinear arrays must be created with `zarr-python` directly
1268+
(or another tool, such as Icechunk) before being opened with `xr.open_zarr`.
1269+
This also means a Dataset opened from a rectilinear-chunked store cannot be
1270+
written back with {py:meth}`Dataset.to_zarr`, including writing to a
1271+
`region` of the store -- both raise a `TypeError` naming the affected
1272+
variable. Write support is tracked in
1273+
[GH11279](https://github.com/pydata/xarray/pull/11279).
1274+
```
1275+
12471276
### Groups
12481277

12491278
Nested groups in zarr stores can be represented by loading the store as a

‎doc/whats-new.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,14 @@ v2026.07.1 (unreleased)
1313

1414
New Features
1515
~~~~~~~~~~~~
16+
- Support reading Zarr V3 arrays with rectilinear (variable-sized) chunk
17+
grids. Using this feature needs ``zarr-python >= 3.2`` with
18+
``zarr.config.set({"array.rectilinear_chunks": True})``; xarray's minimum
19+
supported zarr version is unchanged. Writing rectilinear chunks from
20+
xarray is not yet supported (:pull:`11592`, extracted from :pull:`11279`).
21+
By `Max Jones <https://github.com/maxrjones>`_ and
22+
`Tom Nicholas <https://github.com/TomNicholas>`_.
23+
1624
- Added `PyArrowCapsule interface <https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#arrow-pycapsule-interface>`_
1725
to :py:class:`DataArray` (``__arrow_c_schema__`` and ``__arrow_c_stream__``), enabling near zero-copy
1826
export to pyarrow, polars or duckdb.

‎xarray/backends/zarr.py‎

Lines changed: 87 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,20 @@ async def async_getitem(self, key):
316316
return self._preserve_string_dtype(value)
317317

318318

319+
def _rectilinear_encoding_error(key: str, value, name) -> TypeError:
320+
"""Error for a rectilinear (variable-sized) chunk or shard spec, which xarray
321+
can read but not yet write."""
322+
return TypeError(
323+
f"encoding[{key!r}]={value!r} for variable {name!r} is a rectilinear "
324+
"(variable-sized) grid, e.g. read from a store opened with "
325+
"zarr.config.set({'array.rectilinear_chunks': True}). xarray can read "
326+
"rectilinear grids but not yet write them, including with `region` or "
327+
"`append_dim`. To write a regular grid to a new array instead, clear "
328+
f"the encoding (`del ds[{name!r}].encoding[{key!r}]`) and make sure the "
329+
"variable's own chunks are uniform (e.g. `ds.chunk({dim: size})`)."
330+
)
331+
332+
319333
def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name):
320334
"""
321335
Given encoding chunks (possibly None or []) and variable chunks
@@ -376,6 +390,8 @@ def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name):
376390

377391
for x in enc_chunks_tuple:
378392
if not isinstance(x, int):
393+
if isinstance(x, list | tuple):
394+
raise _rectilinear_encoding_error("chunks", enc_chunks_tuple, name)
379395
raise TypeError(
380396
"zarr chunk sizes specified in `encoding['chunks']` "
381397
"must be an int or a tuple of ints. "
@@ -391,6 +407,27 @@ def _determine_zarr_chunks(enc_chunks, var_chunks, ndim, name):
391407
return enc_chunks_tuple
392408

393409

410+
def _compact_chunk_sizes(
411+
chunk_sizes: tuple[tuple[int, ...], ...],
412+
) -> tuple[int | tuple[int, ...], ...]:
413+
"""Replace a dask-style listing of chunk sizes with a single int along any
414+
dimension where it describes a regular grid, e.g. ((10, 10, 5),) -> (10,).
415+
416+
Gives a consistent representation across zarr-python versions.
417+
"""
418+
compacted: list[int | tuple[int, ...]] = []
419+
for sizes in chunk_sizes:
420+
sizes = tuple(sizes)
421+
if not sizes:
422+
# zero-length dimension: dask rejects an empty tuple here
423+
compacted.append((0,))
424+
elif len(set(sizes[:-1])) <= 1 and sizes[-1] <= sizes[0]:
425+
compacted.append(sizes[0])
426+
else:
427+
compacted.append(sizes)
428+
return tuple(compacted)
429+
430+
394431
def _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr):
395432
# Check for attributes and dimension name metadata as discussed in the Zarr encoding
396433
# specification https://docs.xarray.dev/en/stable/internals/zarr-encoding-spec.html
@@ -515,6 +552,18 @@ def extract_zarr_variable_encoding(
515552
if chunks is None:
516553
chunks = "auto"
517554
encoding["chunks"] = chunks
555+
556+
# Reject rectilinear shards (a sequence of sequences) like we do for chunks.
557+
# Zarr also accepts an int, tuple of ints, "auto" or a dict here.
558+
shards = encoding.get("shards")
559+
if isinstance(shards, list | tuple) and any(
560+
isinstance(x, list | tuple) for x in shards
561+
):
562+
raise _rectilinear_encoding_error("shards", shards, name)
563+
if isinstance(shards, integer_types):
564+
# Expand to a tuple: zarr-python 3.2.x crashes on an int shard spec.
565+
encoding["shards"] = variable.ndim * (int(shards),)
566+
518567
return encoding
519568

520569

@@ -879,18 +928,38 @@ def open_store_variable(self, name):
879928
array_wrapper = ZarrArrayWrapper(zarr_array)
880929
data = indexing.LazilyIndexedArray(array_wrapper)
881930

931+
try:
932+
chunks = tuple(zarr_array.chunks)
933+
except NotImplementedError:
934+
# Rectilinear chunk grid (zarr-python >= 3.2): `.chunks` raises, so
935+
# read the per-chunk sizes instead, e.g. ((10, 20, 30),).
936+
chunks = zarr_array.read_chunk_sizes
937+
# Normalise to an int per regular dim and a tuple per rectilinear dim,
938+
# since what zarr-python returns above varies between versions.
939+
chunks = _compact_chunk_sizes(
940+
tuple(x if isinstance(x, tuple) else (x,) for x in chunks)
941+
)
942+
preferred_chunks = dict(zip(dimensions, chunks, strict=True))
943+
882944
encoding = {
883-
"chunks": zarr_array.chunks,
884-
"preferred_chunks": dict(zip(dimensions, zarr_array.chunks, strict=True)),
945+
"chunks": chunks,
946+
"preferred_chunks": preferred_chunks,
885947
}
886948
if array_wrapper.dtype.kind == "T":
887949
encoding["dtype"] = array_wrapper.dtype
888950

951+
try:
952+
shards = zarr_array.shards
953+
except NotImplementedError:
954+
# Rectilinear shard grid: `.shards` raises, so read the per-shard
955+
# (i.e. outer/storage chunk) sizes instead, e.g. ((1, 2),).
956+
shards = _compact_chunk_sizes(zarr_array.write_chunk_sizes)
957+
889958
encoding.update(
890959
{
891960
"compressors": zarr_array.compressors,
892961
"filters": zarr_array.filters,
893-
"shards": zarr_array.shards,
962+
"shards": shards,
894963
}
895964
)
896965
if self.zarr_group.metadata.zarr_format == 3:
@@ -1219,6 +1288,21 @@ def set_variables(
12191288
else:
12201289
del v.encoding["_FillValue"]
12211290

1291+
# We need to do this for both new and existing variables to ensure we're not
1292+
# writing to a partial chunk, even though we don't use the `encoding` value
1293+
# when writing to an existing variable. See
1294+
# https://github.com/pydata/xarray/issues/8371 for details.
1295+
# Note: Ideally there should be two functions, one for validating the chunks and
1296+
# another one for extracting the encoding.
1297+
# Must run before any resize below, so a rejected encoding can't
1298+
# leave behind a resized array that was never written to.
1299+
encoding = extract_zarr_variable_encoding(
1300+
v,
1301+
raise_on_invalid=vn in check_encoding_set,
1302+
name=vn,
1303+
zarr_format=3 if is_zarr_v3_format else 2,
1304+
)
1305+
12221306
zarr_shape = None
12231307
write_region = self._write_region if self._write_region is not None else {}
12241308
write_region = {dim: write_region.get(dim, slice(None)) for dim in dims}
@@ -1244,19 +1328,6 @@ def set_variables(
12441328
zarr_shape = zarr_array.shape
12451329
region = tuple(write_region[dim] for dim in dims)
12461330

1247-
# We need to do this for both new and existing variables to ensure we're not
1248-
# writing to a partial chunk, even though we don't use the `encoding` value
1249-
# when writing to an existing variable. See
1250-
# https://github.com/pydata/xarray/issues/8371 for details.
1251-
# Note: Ideally there should be two functions, one for validating the chunks and
1252-
# another one for extracting the encoding.
1253-
encoding = extract_zarr_variable_encoding(
1254-
v,
1255-
raise_on_invalid=vn in check_encoding_set,
1256-
name=vn,
1257-
zarr_format=3 if is_zarr_v3_format else 2,
1258-
)
1259-
12601331
# When shards are specified, dask chunks must align with shard boundaries
12611332
# (not just zarr chunk boundaries) to avoid data corruption during
12621333
# parallel writes. See https://github.com/pydata/xarray/issues/10831

‎xarray/tests/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,9 @@ def get_dask_chunkmanager():
164164
requires_zarr_v3 = requires_zarr
165165
has_zarr_v3_dtypes, requires_zarr_v3_dtypes = _importorskip("zarr", "3.1.0")
166166
has_zarr_v3_async_oindex, requires_zarr_v3_async_oindex = _importorskip("zarr", "3.1.2")
167+
has_zarr_rectilinear_chunks, requires_zarr_rectilinear_chunks = _importorskip(
168+
"zarr", "3.2.0"
169+
)
167170
if has_zarr:
168171
import zarr
169172

0 commit comments

Comments
 (0)