Repository navigation
fix(sharding): reject variable-size codecs in index_codecs - #4397
Merged
d-v-b merged 2 commits intoSep 24, 2026
Merged
Conversation
The spec forbids variable-size codecs (e.g. compressors) in a sharding codec's `index_codecs`. ShardingCodec accepted them, and the first write then failed with a bare NotImplementedError from compute_encoded_size. Check `is_fixed_size` on every index codec at construction time, and mark the numcodecs checksum wrappers as fixed-size so they stay valid there. Closes zarr-developers#2030 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
d-v-b
marked this pull request as ready for review
September 24, 2026 09:53
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4397 +/- ##
=======================================
Coverage 94.26% 94.26%
=======================================
Files 92 92
Lines 12982 12988 +6
=======================================
+ Hits 12237 12243 +6
Misses 745 745
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found this old bug unresolved, so I'm resolving it here. AI PR etc etc
🤖 AI text below 🤖
Closes #2030.
The spec says codecs with a variable-size encoding (such as compressors) MUST NOT be used as sharding
index_codecs. The shard index is found by its byte size, so that size has to be known before the index is read.ShardingCodecaccepted such chains anyway, and the first write then failed with aNotImplementedErrorthat carried no message, raised fromcompute_encoded_size.Changes
ShardingCodec.__init__now checksis_fixed_sizeon every index codec. If any codec is not fixed-size, it raises aValueErrorthat names the offending codecs. Because the check is in__init__, it also coversfrom_dict, so opening an existing array with such an index chain fails with the same message.zarr.codecs.numcodecs.CRC32,CRC32C,Adler32,Fletcher32,JenkinsLookup3) now setis_fixed_size = True, so they stay valid as index codecs. Theircompute_encoded_sizewas already fixed (input + 4).Codec.is_fixed_sizenow says the flag also governs whether a codec can be used as an index codec.Behaviour change
is_fixed_sizedefaults toFalse, so a third-party codec that produces a fixed-size encoding but never sets the flag is now rejected inindex_codecs. One test helper intest_sharding_unit.pywas in exactly that position and now sets the flag. In-tree fixed-size codecs (bytes, crc32c, transpose, the numcodecs checksums) already declare it.Tests
test_sharding_fixed_size_index_codecs_roundtrip: writes and reads back data through several valid index chains, including transpose and a numcodecs checksum.test_sharding_codec_rejects_variable_size_index_codecs: gzip or blosc in the index chain raises the new error.I ran the full test suite locally: 10784 passed.
🤖 Generated with Claude Code