Skip to content

Add support for rectilinear chunk grids - #954

Open
maxrjones wants to merge 43 commits into
zarr-developers:mainfrom
maxrjones:poc/unified-chunk-grid
Open

maxrjones wants to merge 43 commits into
zarr-developers:mainfrom
maxrjones:poc/unified-chunk-grid

Conversation

@maxrjones

@maxrjones maxrjones commented Apr 2, 2026 •

Copy link
Copy Markdown
Member

This is a companion PR to zarr-developers/zarr-python#3802 that adds support for rectilinear chunk grids.

Acceptance criteria:

  • Closes part of Support variable-length chunks #12
  • Tests added
  • Tests passing
  • No test coverage regression
  • Full type hint coverage
  • Changes are documented in docs/releases.md
  • New functions/methods are listed in an appropriate *.md file under docs/api
  • New functionality has documentation

@TomNicholas TomNicholas added zarr-python Relevant to zarr-python upstream zarr-specs Requires adoption of a new ZEP labels Apr 14, 2026
@maxrjones maxrjones mentioned this pull request Apr 20, 2026
8 tasks
@norlandrhagen

Copy link
Copy Markdown
Collaborator

Super excited for this! Just ran into a use-case today that it'll be great for.

@maxrjones

Copy link
Copy Markdown
Member Author

Super excited for this! Just ran into a use-case today that it'll be great for.

nice! I'll try to get back to this tomorrow. I got a bit behind on development due to travel.

Are you able to share anything about the use-case? I'm supposed to demo this feature at our project wrap-up next week and if it's related to NASA data it could be a good collaboration opportunity.

Appending now promotes the destination array's chunk grid to
rectilinear when either the incoming or existing array is already
rectilinear, or when both are regular but declare genuinely different
chunk sizes along the append axis - mirroring concatenate(), gated
behind the same array.rectilinear_chunks opt-in check.

- num_chunks() counts rectilinear chunks (len(edges)) instead of
  assuming shape // chunk_size.
- check_compatible_arrays() no longer crashes comparing a rectilinear
  array's declared chunk sizes, and excludes the append axis from the
  comparison (chunk sizes are allowed to differ there).
- resize_array() only calls zarr's Array.resize() when both sides stay
  regular with the same chunk size. Otherwise it writes the correctly
  merged chunk edges directly via copy_and_replace_metadata +
  save_metadata, since zarr's own rectilinear resize logic only ever
  appends a single new edge covering the whole size increase - wrong
  whenever the appended array itself has more than one chunk along
  that axis.

Also fixes a related bug the append test suite caught: comparing
chunk shapes via full_chunk_edges (shape-dependent boundary-truncated
edges) is wrong wherever the two arrays being compared aren't meant to
share the same shape, as in a region write comparing a small tile
against the full destination array. check_combinable_zarr_arrays and
check_compatible_arrays now compare the declared, shape-independent
chunk_grid_sizes instead; check_same_chunk_shapes normalizes a bare
int against a matching 1-tuple so the comparison still treats a
grid zarr auto-classified as regular the same as an equivalent
rectilinear declaration.

Region writes now raise a clear NotImplementedError for a rectilinear
array up front, rather than crashing on chunk_size-based alignment
math that has no equivalent for irregular chunk boundaries.

Replaces the old blanket "appending is not yet supported for
rectilinear arrays" test with tests covering the two promotion cases
and the disabled-flag error, and adds a region-write guard test.
Removes stale "appending is not yet supported" wording from the
data structures explanation, the usage guide's append section, and
the unreleased release notes entry, now that appending can promote an
array to rectilinear. Region writes remain unsupported.
# Conflicts:
#	virtualizarr/manifests/utils.py
#	virtualizarr/tests/test_manifests/test_array.py
Avoids depending on zarr.core.chunk_grids._is_rectilinear_chunks, a
private internal that could change without notice, for the trivial
nested-sequence check we actually need.
# Conflicts:
#	virtualizarr/writers/icechunk.py
Reading rectilinear chunk grids is new in xarray 2026.9.0, so the
round-trip test fails on the minimum supported xarray version.
@TomNicholas

Copy link
Copy Markdown
Member

Now that xarray v2026.09.0 has been released, this should pass CI here too... We also should not need to bump our minimum xarray version for non-rectilinear usage, which is nice.

xarray 2026.9.0 and zarr 3.2.1 are released with rectilinear chunk grid
support, so only the unreleased virtual-tiff branch is still needed.
Rectilinear TIFF strips need virtual-tiff changes that aren't released
yet (virtual-zarr/virtual-tiff#87), so drop the TIFF concat test, the
tifffile dev dependency and the rectilinear pixi env from this PR.
_warn_about_oversized_virtual_chunks runs on every opened dataset and
compared chunk shape to array shape via manifest_chunk_shape, which
raises on a rectilinear grid, so any parser returning one crashed
open_virtual_dataset.

Also reword it and the partial-chunk concat error, which blamed
regular-only chunk grids: a partial chunk before the last input still
can't be concatenated with rectilinear grids, because the stored chunk
decodes to the full chunk length.
Indexing, broadcast_to and np.where assume a regular chunk grid, so on a
rectilinear array they failed with an AttributeError from deep inside
manifest_chunk_shape. That included xarray's isel, expand_dims and
concat along a new dimension, which index under the hood. Raise
NotImplementedError naming the operation instead, and document the
limitation; supporting it is left to a follow-up.

with_fill_value_only only needed the chunk grid's shape, so compute that
via ChunkGrid, which handles rectilinear grids, instead of raising.

This branch was previously deployed

1 inactive deployment
test-release — 7840f28f Deployed Oct 16, 2026 by TomNicholas via test-built-dist #2687
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request internals zarr-python Relevant to zarr-python upstream zarr-specs Requires adoption of a new ZEP

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants