Repository navigation
sharding: lift constraint on evenly divisible subchunk shapes #34
Description
Now that the rectilinear grid is moving forward, it is time to lift the constraint that the sharding_indexed subchunk shape must evenly divide the chunk shape.
The only question is: what should the shape of the straddling subchunks be?
- Bounded by the chunk shape
- Always matching the subchunk shape, with some fill value elements that get ignored
My vote is on the former, so we can save some bytes :).
I've evaluated this with zarrs and it was fairly straightforward to support in the codec itself.
xref: zarrs/zarrs#233
Activity
d-v-b commentedon Oct 22, 2025
Bounded by chunk shape sounds good to me! I haven't looked into the zarr-python implementation (@jhamman is working on the rectilinear chunking PR) but I don't think it will be a big lift
jbms commentedon Oct 22, 2025
I'd agree with bounding it by the outer chunk shape (meaning the sub-chunk grid is also rectilinear rather than regular).
shoyer commentedon Oct 28, 2025
👍
For clarity, could you give a example of what these would look like?
jbms commentedon Oct 28, 2025
One thing is that when used with a regular grid, the original constraint, while not strictly necessary, is still likely to be a good idea in almost all cases so that the overall grid does not become non-regular accidentally. Implementations may want to "encourage" users to pick a sub-chunk shape that satisfies that constraint when using this codec with a regular grid, e.g. warn unless some other (not part of the stored metadata) option like
allow_non_regular_subchunk_gridis specified by a user. Otherwise users may arbitrarily pick some chunk shapes and happen to end up with a non-regular grid.LDeakin commentedon Dec 3, 2025
I've kicked off a PR: zarr-specs #370