You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bc177b3
Browse filesBrowse the repository at this point in the historyBrowse files
feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope (zarr-developers#4583)
* feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope
Add definitions for the extensions zarr-python writes. zarr-extensions
registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes`
codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own
`null_terminated_bytes` and `variable_length_bytes`, which no
specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the
default scope stays spec-pure.
Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to
CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context.
Fixes#423
Assisted-by: ClaudeCode:claude-opus-5-5
* docs(zarr-metadata): number the changelog fragments for PR 426
Assisted-by: ClaudeCode:claude-opus-5-5
* docs(zarr-metadata): number the changelog fragments for PR 4583
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(zarr-metadata): keep the old scope names and tighten the string rules
Keep `CORE` and `CORE_AND_EXTENSIONS` importable from `zarr_metadata.v3`
and `zarr_metadata.v3.definition` as the same scopes with a
DeprecationWarning, so a consumer such as pydantic-zarr keeps working
across the rename.
Check that `vlen-utf8` encodes `string` and `vlen-bytes` encodes `bytes`
(or zarr-python's `variable_length_bytes`), as zarr-extensions says,
instead of accepting any data type whose values vary in size.
Do not count trailing null bytes against the size of a
`null_terminated_bytes` fill value, since they are padding. Spell a
`fixed_length_utf32` fill value without its trailing U+0000 padding, so
`"a"` and `"a\u0000"` have one canonical spelling.
Assisted-by: ClaudeCode:claude-opus-5-5
* feat(zarr-metadata): read zarr-python's raw_bytes and rename CORE_V2
Add zarr-python's `raw_bytes` data type (NumPy's `V`) to
ZARR_PYTHON_CONTEXT, so the scope reads every data type zarr-python
writes. Its names start with `ZarrPython`, since `RawBytes` already names
the spec's raw bits. A fill value longer than `length_bytes` is a problem,
because zarr-python cuts it short, and the canonical spelling pads it with
null bytes, as zarr-python writes it.
Rename CORE_V2 to CORE_V2_CONTEXT, so every scope's name ends in
`_CONTEXT`. The old name keeps working with a DeprecationWarning, through
the same `renamed_scope` helper as the v3 names, which moves to
`zarr_metadata._renamed`.
Share the base64 decoding and spelling of fill values in `_byte.py`, and
say that the vlen codecs' `configuration` key is optional, since
zarr-python writes an empty object.
Assisted-by: ClaudeCode:claude-opus-5-5
* refactor(zarr-metadata): declare the v2 scope's old name once
Move CORE_V2_CONTEXT and its table of old names into `v2/_core.py`, as
`v3/_core.py` holds the v3 scopes, so `zarr_metadata.v2` and
`zarr_metadata.v2.definition` share one table.
Say why a `fixed_length_utf32` fill value counts its trailing U+0000 code
points, as the spec requires, while `null_terminated_bytes` follows
zarr-python and does not, and test a value that only its padding makes too
long.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(zarr-metadata): give main's strategies the renamed scopes and new definitions
The Hypothesis strategies merged from main (zarr-developers#4575) imported `CORE`,
`CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the
`_CONTEXT` names, and their coverage test requires a strategy for every
definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add
strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each
vlen codec draws nothing for a data type it does not read.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(zarr-metadata): draw zarr-python's own data types in the testing strategies
`DEFAULT_STRATEGIES` holds a strategy for each definition the package
defines, but `null_terminated_bytes`, `variable_length_bytes` and
`raw_bytes` had none, so `fields()` raised `LookupError` for them. Draw
each, and check the strategies against `ZARR_PYTHON_CONTEXT`, which
holds every definition. Say that the documents in
test_zarr_python_scope.py leave out the compressors zarr-python adds.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* test(zarr-metadata): pin the registered extensions to CORE_AND_EXTENSIONS_CONTEXT
The strategies are now checked against `ZARR_PYTHON_CONTEXT`, which no
longer proves that `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`
are in `CORE_AND_EXTENSIONS_CONTEXT`. Assert it directly.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* refactor(zarr-metadata): let the vlen strategies ask their codec's rules
Each vlen codec strategy named its compatible data types a second time;
draw nothing for exactly the chunks the codec's own chunk rules refuse,
so the two cannot drift. Draw `fixed_length_utf32` fill values with
trailing U+0000 padding too, which the canonical spelling removes.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: packages/zarr-metadata/README.md
+10-8Lines changed: 10 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,7 +41,7 @@ closely model the content of the Zarr specifications, such as:
41
41
`TypedDict` with its rules, and the scope that decides which definitions
42
42
read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
43
43
definition and scope of their format, such as `BYTES_CODEC` and
44
-
`CORE_AND_EXTENSIONS`.
44
+
`CORE_AND_EXTENSIONS_CONTEXT`.
45
45
-**JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
46
46
`check` checks a JSON value against any `TypedDict` and reports every
47
47
problem with its location.
@@ -104,7 +104,9 @@ of context-free consistency rules, e.g.:
104
104
In a v3 document, validation also checks each extension point: the data
105
105
type, the chunk grid, the chunk key encoding, each codec, and each storage
106
106
transformer. Each one is checked by the definition that claims its name in
107
-
a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
107
+
a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
108
+
`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
109
+
specification registers.
108
110
If the definition rejects a configuration, the reader rejects it too,
109
111
and a configuration key the definition does not declare is reported as
110
112
`unknown_key`. If no definition in the scope claims a name, the reader
@@ -232,13 +234,13 @@ each field:
232
234
```python test="true" session="readme"
233
235
from zarr_metadata.model import read_array_metadata_v3
234
236
from zarr_metadata.definition import canonical_of, with_problems
235
-
from zarr_metadata.v3.definition importCORE
237
+
from zarr_metadata.v3.definition importCORE_CONTEXT
236
238
237
239
reading = read_array_metadata_v3(raw)
238
240
beyond_core = [
239
241
loc
240
242
for loc, field in reading.fields()
241
-
if field.name isnotNoneandCORE.claimant(field.read_as, field.name) isNone
243
+
if field.name isnotNoneandCORE_CONTEXT.claimant(field.read_as, field.name) isNone
242
244
]
243
245
simplest = {
244
246
loc: canonical_of(field, problems)
@@ -283,15 +285,15 @@ scope claims is an `UnclaimedField` too, also when it sits inside another
283
285
field, such as a codec inside a shard. A model's `unclaimed_fields` lists
284
286
every field nothing in scope claims, at any depth, by its location, and
285
287
leaves out the members that waive `must_understand`: a reader can open
286
-
the node when it is empty. `CORE` holds
288
+
the node when it is empty. `CORE_CONTEXT` holds
287
289
`CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
288
290
`consolidated_metadata` as the envelope the spec fixes. A group's
289
291
`consolidated()` reads the documents that member lists, in the group's
290
292
own scope.
291
293
292
294
A v3 model consists of its document and the scope it was read in.
293
295
`ZarrV3ArrayMetadata(document, context=None)` reads the document in the
294
-
scope (`CORE_AND_EXTENSIONS` when none is given) and raises
296
+
scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
295
297
`MetadataValidationError` with every problem, so an invalid model is never
296
298
built. `to_json` returns the document as it was written, and
297
299
`to_key_value` writes that same document to store bytes.
@@ -322,7 +324,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
322
324
`read_consolidated_metadata_v3`, which returns problems instead of
323
325
raising them. A document with a problem then fails only when it is read.
324
326
Pass `context=group.context` to read the documents in the group's own
325
-
scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
327
+
scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
326
328
327
329
To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
328
330
pass it to the group's `update`. The group reads the listed models in its
@@ -332,7 +334,7 @@ models as entries. Such an entry is accepted when its claims refine into
332
334
the model's scope, and refused at its path otherwise. To consolidate
333
335
children that were read in several scopes, consolidate them in the
334
336
`union` of those scopes. Every reader takes `context=None` for the
335
-
default scope, `CORE_AND_EXTENSIONS`.
337
+
default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
336
338
337
339
Two models are equal when they represent the same document, even if each
338
340
is spelled differently. Values the package interprets (each field, and the
Add the `ZARR_PYTHON_CONTEXT` scope, which reads what zarr-python writes: `CORE_AND_EXTENSIONS_CONTEXT` plus zarr-python's own data types `null_terminated_bytes` (NumPy's `S`), `raw_bytes` (NumPy's `V`) and `variable_length_bytes`, which no specification registers. `CORE_AND_EXTENSIONS_CONTEXT` now also reads the `fixed_length_utf32` data type (NumPy's `U`) and the `vlen-utf8` and `vlen-bytes` codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.
Rename the scopes `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` to `CORE_CONTEXT`, `CORE_AND_EXTENSIONS_CONTEXT` and `CORE_V2_CONTEXT`, so that a scope's name says it is a context. The old names, imported from `zarr_metadata.v3`, `zarr_metadata.v3.definition`, `zarr_metadata.v2` or `zarr_metadata.v2.definition`, still refer to the same scopes with a `DeprecationWarning`, and will be removed in a later release.
Copy file name to clipboardExpand all lines: packages/zarr-metadata/docs/parsing-validation.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -161,7 +161,7 @@ This procedure yields one of two values:
161
161
162
162
`zarr-metadata` does not run each stage as a separate function. Instead, a single read of the document collects the problems from every stage, so a caller gets the full list of problems at once.
163
163
164
-
The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE` holds only the definitions from the Zarr V3 spec.
164
+
The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS_CONTEXT`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE_CONTEXT` holds only the definitions from the Zarr V3 spec. `ZARR_PYTHON_CONTEXT` adds to `CORE_AND_EXTENSIONS_CONTEXT` the data types that zarr-python writes but no specification registers (`null_terminated_bytes` for NumPy's `S`, `raw_bytes` for NumPy's `V`, and `variable_length_bytes`), so a strict check of what zarr-python writes passes it.
165
165
166
166
`read_array_metadata_v3` reads a JSON document and returns a "reading". `reading.problems` is a tuple of the problems, and `reading.metadata` is the validated model, or `None` if there are problems. Each problem has a `loc`, which is its position in the document as a tuple, a `kind`, like `missing_key` or `invalid_value`, and a human-readable `message`:
167
167
@@ -193,10 +193,10 @@ To read one field on its own, such as a codec, use `resolve` from `zarr_metadata
`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS.update(codecs={**CORE_AND_EXTENSIONS.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
228
+
`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS_CONTEXT.update(codecs={**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
229
229
230
230
[^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the field `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`.
0 commit comments