Repository navigation
Commit 8dc50b8
feat(zarr-metadata): metadata fields read against their definitions (#4434)
* feat(zarr-metadata): must_understand false refused at every extension point
Ignoring a codec gives wrong bytes as surely as ignoring a data type
gives wrong values, so no extension point may be declared ignorable:
`validate_array_metadata_v3` now refuses `must_understand: false` on a
codec or a storage transformer, as it already did on a data type, a
chunk grid and a chunk key encoding. The spec names only those three,
which this package reads as an oversight rather than a licence.
`must_understand: false` keeps its meaning where it has one: an unknown
top-level extension field, which a reader really can skip.
It changes verdicts and nothing else depends on it, so it stands on its
own as a reading of the spec. The definition layer above reads every
metadata field under the same policy.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(zarr-metadata): the fragment for must_understand false refused everywhere
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): the array schema refuses must_understand false where the validator does
The JSON schema the pydantic integration generates typed a codec and a
storage transformer with the envelope that allows `must_understand:
false`, so it accepted an array document the validator refuses. Every
extension point of an array document now has the mandatory envelope,
and the test that holds schema and runtime together covers all five.
A metadata field read on its own keeps the general envelope, as
`parse_metadata_field_v3` does.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(zarr-metadata): cite the spec lines that name the three points
The comment linked the section's opening rules, not the sentence that limits `must_understand=False` to three extension points.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs(zarr-metadata): the fragment takes this PR's number and says what breaks
Assisted-by: ClaudeCode:claude-opus-5-5
* feat(zarr-metadata): one metadata field read against its definition
The one-field layer, JSON first: the type check stands alone and feeds
the rules, and nothing is built from a configuration to judge it.
A definition is a value: the name the metadata carries, the TypedDict
its configuration is -- the one declaration of that JSON, which the
checker is compiled from -- and plain functions over that TypedDict for
the rules. Its kind is its type (`CodecDefinition`, `ChunkGridDefinition`,
...), and a generic frozen dataclass checks itself in `__post_init__`:
no `__init_subclass__`, no metaclass, no abstract base. A scope holds
definitions by kind.
Three steps, each usable on its own by a caller holding JSON: `check`
(JSON against a TypedDict, no scope: #347's, re-exported),
`Definition.judge` (the check, each nested field's envelope judged -- a
member annotated `CodecField` holds one -- then the rules), and
`resolve` (a whole field in a scope, returning `Resolved` with a
`Resolution` and every problem located). gzip, bytes, crc32c and the
regular grid are defined.
What the rules are handed is what the type says: a configuration holding
the keys its TypedDict admits and nothing else, whose nested fields'
envelopes are sound, so a rule may index a table by key or read a nested
field's name; a rule's problems land in the configuration in `judge` and
`resolve` alike. The checker hands nested fields back in the typed value,
so the branch of a union that did not match reads none. A definition
refuses at build what it could not read with -- a TypedDict that says
nothing of the keys it does not declare, a member typed
`ZarrV3MetadataFieldJSON`, an annotation that does not resolve, a name
that is not a string, a codec kind outside the three, rules or `names`
that are not functions -- and a field is read as one of the five kinds,
with or without type arguments, anything else a `TypeError`.
`configuration_of` types a read configuration as its definition's
TypedDict, and the configuration's type parameter has a default, so
`resolve(field, CodecDefinition, scope)` infers nothing unknown.
The definitions read JSON and its problems from `zarr_metadata._json`,
and a metadata field's own validators from `zarr_metadata.v3._common`,
where they move from the model, so nothing this layer adds imports the
model it supports.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat(zarr-metadata): every extension as a definition
Layer 2b interpolates 2a's single field across the package: each codec,
data type, chunk grid and chunk key encoding is a definition -- its name,
the TypedDict main already declares for its configuration, and its rules
as a function over that TypedDict.
- Codecs: blosc, zstd, transpose, scale_offset, sharding_indexed and
cast_value join gzip, bytes and crc32c. A shard's two pipelines and a
cast's target type are nested fields: their TypedDicts annotate them
`CodecField` and `DataTypeField`, statically the same envelope type,
and they are read in the scope the containing field is read in.
- Data types: the sixteen with nothing to configure take
`EmptyConfiguration`; the numpy time types share their scale-factor
rule; struct judges its field names; the raw-bytes family claims every
`r<N>` and judges its width.
- The rectilinear grid, and the default and v2 chunk key encodings.
- `canonical` per definition, and `canonicalize` for a field in scope:
blosc drops a `typesize` that `noshuffle` ignores, rectilinear
run-length encodes, and nested fields and envelopes simplify without
code of their own. The configuration it starts from holds what its
TypedDict admits at every depth, and what `canonical` gives is judged
again: one that does not hold is a `ValueError`, the definition's
fault rather than the field's.
`CORE` holds the 24 the specification defines and `CORE_AND_EXTENSIONS`
the 9 more zarr-extensions registers, as on #4379.
Against #4379's entity layer, over every metadata field in the
40k-document corpus (157,254): the same resolution for every field, and
the same canonical spelling for all 114,445 that read. Problems and
messages are the same for all but 9, sharding codecs with a stray member
in an inner codec's envelope: the shard's own rule is not asked of a
configuration whose nested envelopes are not sound, as `judge` would not
ask it, so its problem waits until the envelope is fixed. Deferred to
the layer that reads types together: whether a struct field's type is
fixed-size.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(zarr-metadata): the fragments for metadata fields read against their definitions
2a and 2b land together, here: alone, 2a's scopes hold four definitions
and read a core data type as out of scope. So 2a's fragment is named
for this PR, beside 2b's.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a field with a problem has no simplest spelling
`canonicalize` gave a spelling for a field with an unknown key, a stray
envelope member or a `must_understand` of `false`, reporting the problem
beside a form that had quietly dropped what the author wrote: `{"name":
"bytes", "configuration": {"endain": "big"}}` came back as `"bytes"`.
Now a field with any problem comes back None, with its problems. A
field with an unknown key is still read, and `resolve` still hands back
its configuration, so a consumer that tolerates one filters by kind;
the guide says the field is valid only when there is no problem at all.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(zarr-metadata): resolve refuses must_understand false wherever the model does
The definition layer judges a field's envelope with the same refusal the
model's validator applies at each extension point. One test now holds
the two together at all five points, so a change to the reading that
reaches one and not the other fails here.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* docs(zarr-metadata): one fragment for metadata fields read against their definitions
The two fragments were 2a's and 2b's, and the first still said the
scopes hold four definitions. #350 lands both, so one fragment says
what it lands.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): null is not a verdict for a field or a configuration
`resolve(None, ...)` answered "invalid" with no problem, and
`Definition.judge(None)` and `canonicalize(None, ...)` answered with no
value and no problem. `refine_json` gives JSON's `null` as None with no
problem, and each read a None refinement as "not JSON, and its problems
say why". They now ask the problems whether the value is JSON, so
`null` is read like any other value, and refused as a value of the
wrong type.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a reader's definition takes a name over from a family
`CORE_AND_EXTENSIONS.extended_with(my_r16)` still read `r16` with the
raw-bytes family, since `claimant` asked the definitions in the order
they were filed and the family came first, so a reader could not
substitute its own reading of one name a family claims. It now asks the
one filed last first, as `Context.of` and `extended_with` say: a
definition passed later takes a name over, whether the earlier one was
filed under it or claims it as a family. No two definitions the package
files claim the same name, so its own scopes read as before.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a regular grid's chunk extent may be 0
The core spec: "The chunk shape elements are non-zero when the
corresponding dimensions of the arrays have non-zero length." So 0 is
right on a dimension of length 0 -- zarr-python 3.0 and 3.1 wrote it,
and this package's own `create_default(shape=(0, 4))` does -- and a grid
alone cannot tell that dimension from one of non-zero length. The field
now refuses only a negative extent; an extent of 0 on a dimension that
is not empty is for the layer that reads the grid beside the shape.
Needed before the model reads grids through their definitions: without
it the model would refuse to read back a zero-length array it wrote.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor(zarr-metadata): no re-check of a TypedDict a definition vetted
`_configuration_checked` refused a shape that is not a TypedDict, but its
one caller hands it a definition's configuration, which the definition
vetted when it was built.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a definition's rules are checked where they are called
The rules are an extension author's code, and a slip there surfaced far
from it: a rule yielding a string failed on `.loc`, a plain function
that forgot to yield failed as `'NoneType' object is not iterable` on
valid documents, and a rule that raised let a bare `KeyError` out of a
validator with nothing to say whose it was. What the rules yield is now
checked to be what they declare, naming the definition, and an error
one raises carries a note naming the definition and where it was
reading. The refusal of an open TypedDict, and the guide, name
`typing_extensions.TypedDict`, since `typing.TypedDict` takes no
`closed` on the versions the package supports.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a family's name rules are checked as its rules are
A family's `name_rules` is the extension author's code too, and a name
rule yielding a string failed on `.loc`, as a rule did before `_ruled`.
`_ruled` now takes the call, and reads both through the same checks: what
they yield is checked, and an error one raises carries a note naming the
definition.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* fix(zarr-metadata): a definition filed again is the one filed last
`claimant` asks the definition filed last first, but a name filed again
kept its first place in its table, so extending a scope with a family
it already held did not let the family take a name back from a
definition filed after it. A name filed again now moves to the end.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* feat(zarr-metadata): codecs say whether their size is static; a shard's index takes static codecs
The sharding spec says codecs that produce a variable-sized encoded
representation, such as compressors, MUST NOT be used for index codecs,
and nothing checked it: a shard with a gzip in `index_codecs` read.
A `CodecDefinition` now says its `size`, `"static"` or `"dynamic"`:
whether the size of what it gives out is fixed by the size of what it is
handed. It is required, and checked when the definition is built, as
`kind` is. The package's codecs say what zarr-python's `is_fixed_size`
says: `bytes`, `crc32c`, `transpose`, `cast_value` and `scale_offset` are
static; `blosc`, `gzip`, `zstd` and `sharding_indexed`, whose empty inner
chunks take no bytes, are dynamic.
A member annotated `StaticCodecField` takes codecs of static size only.
The shard's TypedDict declares `index_codecs: tuple[StaticCodecField,
...]`, so the constraint lives in the one declaration of that JSON, and
any extension's container codec can say the same. Where a nested field
is read in a scope, a codec of dynamic size in such a member is an
`invalid_value` at its place. A name nothing in scope claims is left
unjudged, its size unknown with the rest of it.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(zarr-metadata): the every-definition tests say what they read
The module docstring named the fork's layer, which a reader of this package has no reason to know.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs(zarr-metadata): the definitions' fragment takes this PR's number
Assisted-by: ClaudeCode:claude-opus-5-5
* refactor(zarr-metadata): raw bits read as r* with the size their name carries
`r16` reads as the definition filed under `r*`, the name the spec's data
types table gives raw bits, with `{"bits": 16}` as its configuration; a
size that is not a positive multiple of 8 is a rule over that
configuration, reported at the field. Every name a scope looks up is now
exact, so `Context.claimant` is a dict lookup, and `Definition` loses
`names` and `name_rules`, which only raw bits used. A reader reads raw
bits its own way by defining `r*`; a data type named `r16` is refused,
since that name reads as `r*`. `canonicalize` writes the size back into
the name, in decimal, so `r008` is `r8`.
Assisted-by: ClaudeCode:claude-opus-5-5
* perf(zarr-metadata): resolve judges the envelope without walking the JSON it refined
`resolve` refines the whole field before reading it, yet its envelope
check walked each configuration member as JSON again, and each nested
field's configuration once more. `envelope_problems` judges the envelope
alone, and `resolve` asks it of the field and of each nested field;
`validate_metadata_field_v3`, for a value nothing has refined, is it
plus the JSON check of the configuration's members. For a shard holding
four codecs, a resolve makes 766 Python calls instead of 1,070.
Assisted-by: ClaudeCode:claude-opus-5-5
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent c383b22 commit 8dc50b8
53 files changed
Lines changed: 3472 additions & 123 deletions
File tree
- packages/zarr-metadata
- changes
- docs/api
- v3
- src/zarr_metadata
- model
- v3
- chunk_grid
- chunk_key_encoding
- codec
- data_type
- tests
- model
- v3
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
41 | 43 | | |
42 | 44 | | |
43 | 45 | | |
| |||
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
56 | | - | |
| 58 | + | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
53 | 52 | | |
54 | 53 | | |
55 | 54 | | |
56 | 55 | | |
57 | | - | |
58 | 56 | | |
59 | 57 | | |
60 | 58 | | |
61 | 59 | | |
62 | | - | |
63 | 60 | | |
64 | 61 | | |
65 | 62 | | |
| |||
81 | 78 | | |
82 | 79 | | |
83 | 80 | | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
84 | 86 | | |
85 | 87 | | |
86 | 88 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | 25 | | |
27 | 26 | | |
28 | 27 | | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
34 | | - | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| |||
144 | 146 | | |
145 | 147 | | |
146 | 148 | | |
147 | | - | |
148 | | - | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | 149 | | |
229 | 150 | | |
230 | 151 | | |
| |||
410 | 331 | | |
411 | 332 | | |
412 | 333 | | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
413 | 341 | | |
414 | 342 | | |
415 | 343 | | |
| |||
431 | 359 | | |
432 | 360 | | |
433 | 361 | | |
434 | | - | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
435 | 373 | | |
436 | 374 | | |
437 | 375 | | |
| |||
0 commit comments