Skip to content

Commit 310075a

Browse files
d-v-bclaude
andauthored
refactor(zarr-metadata)!: call an extension-point value a declaration (zarr-developers#4592)
* refactor(zarr-metadata)!: call a JSON key a field and an extension-point value a declaration Match zarrs and the v3 spec. A name/value pair in a JSON metadata object is now a field everywhere (it was a member), which frees "field" from its old term-of-art meaning: a value at an extension point that a scope resolves, which is now a declaration. The public names for that concept follow: AcceptedField and its siblings become AcceptedDeclaration etc., the ZarrV3*Field aliases become ZarrV3*Declaration, ZarrV3MetadataFieldJSON becomes ZarrV3DeclarationJSON, a reading's fields() becomes declarations(), and unclaimed_fields becomes unclaimed_declarations. Messages that said member now say field. The pickle-rebuild functions keep their names so existing pickles load. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): say which fields hold declarations and complete the rename's changelog Review fixes for the field/declaration rename: an extra field is a declaration written as the field alone, the array models' plain fields are the ones that hold no declaration, must_understand_fields covers extra fields only, and the context guide matches definitions to declarations. The fragment drops four names that were never public and adds spec_fields, the declaration wording of messages, and the renamed JSON Schema $defs keys. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the field/declaration fragment after its PR Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): keep the field/declaration fragment unnumbered until it goes upstream towncrier links fragments to upstream zarr-python PRs, and upstream #432 is an unrelated merged PR; this change has only a fork PR so far. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the field/declaration fragment after its fork PR CI's changelog check requires an integer PR number, so an orphan fragment cannot land. 432 is the fork PR; renumber it to the upstream PR when this goes upstream. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): stop claiming the v3 spec prefers "field" over "member" The v3 core spec uses both words for a name/value pair in a metadata object ("the member chunk_shape", "the fill_value metadata field"), so only zarrs motivates the choice. Also say "context", not "scope". Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * refactor(zarr-metadata)!: keep "member" for a JSON name/value pair Drop the member-to-field half of the rename. The v3 core spec uses "field", "key" and "member" for a name/value pair without preference, RFC 8259 says "member", and the package already said "member", so only zarrs motivated the change. The field-to-declaration half stays: it removes the collision between the extension-point sense of "field" and the JSON sense in extra_fields, ExtraFieldDefinition and friends. Every line that upstream main writes with "member" is restored by aligning the two trees token by token. spec_fields is spec_members again, member_changes keeps its name, and ZarrV3ConsolidatedMetadata takes member= again. Prose that said "extra member" or "extension member" now says "extra field" and "extension field", after the names. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * refactor(zarr-metadata): finish the member and declaration wording roborev flagged Four docstrings said "the fields that hold no declaration" for members; the pickle-rebuild comment said the functions keep their _declaration names when they keep their _field names; and two test helpers in the strategies tests still said "fields" for declarations. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * test(zarr-metadata): name the schema-acceptance test after the declaration it reads Last test in the package whose name said "field" for a declaration. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * refactor(zarr-metadata): restore "member" where the rename changed a name in both senses Thirteen test names changed in both the member and the declaration sense at once, so the token alignment that restored "member" skipped them; they take their upstream spelling with only the declaration half applied. Seven docstrings get the same treatment, and the two envelope docstrings that said "an extra field" for an undeclared envelope member now say "an extra envelope member", so "extra field" keeps one meaning. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * docs(zarr-metadata): say the model checks the members of a document Last module docstring with "field" in the name/value-pair sense. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> * docs(zarr-metadata): number the declaration-rename fragment after upstream PR 4592 Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]> --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 7105939 commit 310075a

94 files changed

Lines changed: 2065 additions & 1907 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/zarr-metadata/README.md‎

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ closely model the content of the Zarr specifications, such as:
4848
- **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem
4949
values every reader returns, and the error that carries them.
5050
- **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the
51-
JSON Schema of a `TypedDict`, a v3 field, or a whole `zarr.json`.
51+
JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
5252
- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/),
5353
requires Pydantic 2.13 or newer): each model as a Pydantic field type that
5454
validates raw documents through the same strict parser.
@@ -213,23 +213,23 @@ with one of them is refused:
213213
`read_array_metadata_v3` reads a document once and returns everything the
214214
read found:
215215

216-
- each field as the scope read it: an `AcceptedField` when a definition
217-
claims its name, an `UnclaimedField` when none does, or a
218-
`RefusedField`, together with the field's location and the kind it was
216+
- each declaration as the scope read it: an `AcceptedDeclaration` when a definition
217+
claims its name, an `UnclaimedDeclaration` when none does, or a
218+
`RefusedDeclaration`, together with the declaration's location and the kind it was
219219
read as;
220220
- each codec, with the chunk it receives;
221221
- every problem;
222222
- the model, when there are no problems.
223223

224224
`from_json` returns that model, or raises the problems. A consumer can
225-
apply its own policy by walking the fields, without reading anything
226-
twice. `with_problems` pairs each field with its problems: those located
227-
in the field and in the fields it contains. This matches how zod's
225+
apply its own policy by walking the declarations, without reading anything
226+
twice. `with_problems` pairs each declaration with its problems: those located
227+
in the declaration and in the declarations it contains. This matches how zod's
228228
`flattenError` groups issues. `canonical_of` returns the simplest
229-
spelling of a field that has no problems, without reading it again. For
230-
example, the code below finds the fields that go beyond the core spec (a
231-
field with no name is already a problem), and the simplest spelling of
232-
each field:
229+
spelling of a declaration that has no problems, without reading it again. For
230+
example, the code below finds the declarations that go beyond the core spec (a
231+
declaration with no name is already a problem), and the simplest spelling of
232+
each declaration:
233233

234234
```python test="true" session="readme"
235235
from zarr_metadata.model import read_array_metadata_v3
@@ -239,12 +239,13 @@ from zarr_metadata.v3.definition import CORE_CONTEXT
239239
reading = read_array_metadata_v3(raw)
240240
beyond_core = [
241241
loc
242-
for loc, field in reading.fields()
243-
if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None
242+
for loc, declaration in reading.declarations()
243+
if declaration.name is not None
244+
and CORE_CONTEXT.claimant(declaration.read_as, declaration.name) is None
244245
]
245246
simplest = {
246-
loc: canonical_of(field, problems)
247-
for loc, field, problems in with_problems(reading.fields(), reading.problems)
247+
loc: canonical_of(declaration, problems)
248+
for loc, declaration, problems in with_problems(reading.declarations(), reading.problems)
248249
}
249250
metadata = reading.metadata # None when reading.problems is not empty
250251
```
@@ -274,16 +275,17 @@ string against those rules.
274275
A top-level member that the spec does not define is an extra field. A
275276
scope can hold a definition for one, by its key, in the
276277
`array_extra_fields` or `group_extra_fields` slot of `ZarrV3Context`. A
277-
member whose key a definition claims is read by that definition, like any
278-
other field. Any other member is kept as written, as an `UnclaimedField`.
278+
member whose key a definition claims is read by that definition: an extra field is a declaration
279+
written as the member alone, `{key: value}`. Any other extra field is kept as
280+
written, as an `UnclaimedDeclaration`.
279281
The scope is what a reader recognizes, so the model's
280282
`must_understand_fields` lists each unclaimed member that does not say
281283
`"must_understand": false`: the spec requires a reader to fail to open a
282-
node with any such member. It covers members only. A data type, chunk
284+
node with any such member. It covers extra fields only. A data type, chunk
283285
grid, chunk key encoding, codec or storage transformer that nothing in
284-
scope claims is an `UnclaimedField` too, also when it sits inside another
285-
field, such as a codec inside a shard. A model's `unclaimed_fields` lists
286-
every field nothing in scope claims, at any depth, by its location, and
286+
scope claims is an `UnclaimedDeclaration` too, also when it sits inside another
287+
declaration, such as a codec inside a shard. A model's `unclaimed_declarations` lists
288+
every declaration nothing in scope claims, at any depth, by its location, and
287289
leaves out the members that waive `must_understand`: a reader can open
288290
the node when it is empty. `CORE_CONTEXT` holds
289291
`CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
@@ -298,9 +300,9 @@ scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
298300
built. `to_json` returns the document as it was written, and
299301
`to_key_value` writes that same document to store bytes.
300302

301-
Every typed member is a view of that read: each field as the scope read
302-
it (an `AcceptedField` or an `UnclaimedField`), and `shape`, `attributes`
303-
and the other members as the read refined them. These views are read-only
303+
Every typed attribute is a view of that read: each declaration as the scope read
304+
it (an `AcceptedDeclaration` or an `UnclaimedDeclaration`), and `shape`, `attributes`
305+
and the members that hold no declaration as the read refined them. These views are read-only
304306
at every level: a list in the document is shown as a tuple, and an object
305307
as a read-only mapping. `to_json` returns plain containers.
306308

@@ -337,7 +339,7 @@ children that were read in several scopes, consolidate them in the
337339
default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
338340

339341
Two models are equal when they represent the same document, even if each
340-
is spelled differently. Values the package interprets (each field, and the
342+
is spelled differently. Values the package interprets (each declaration, and the
341343
fill value against its data type) are compared by their canonical
342344
spelling, as returned by `canonical_of` and `canonical_fill_value`. For
343345
example, `"NaN"` and `"0x7fc00000"` are the same `float32` fill value,
@@ -357,7 +359,7 @@ the same data. The v3 spec defines its blosc codec, and says `noshuffle`
357359
ignores `typesize`. A v2 node with no `.zattrs` equals one with an
358360
empty `.zattrs`, as a v3 document with and without an empty
359361
`attributes` are equal. Values the package does
360-
not interpret (attributes, extra fields, and the configuration of a field
362+
not interpret (attributes, extra fields, and the configuration of a declaration
361363
that nothing in the scope claims) are compared as JSON text. JSON text
362364
distinguishes `true` from `1` and `-0.0` from `0.0`, and treats `NaN` as
363365
equal to itself. Equal models have equal hashes, but may write different
@@ -367,11 +369,11 @@ model holds nothing that can be changed in place.
367369
`node_metadata_json_schema_v3` writes the rules the readers apply as a
368370
JSON Schema (draft 2020-12). Use it in an editor that checks a `zarr.json`
369371
while you write it, or with a validator in another language. Each
370-
extension point is described as a field as its scope reads it: a claimed
372+
extension point is described as a declaration as its scope reads it: a claimed
371373
configuration follows its definition's TypedDict, including bounds, and a
372374
name that nothing in the scope claims may have any configuration. The fill
373375
value must be a valid fill value for the data type the document names.
374-
`field_json_schema_v3(kind, *, context=None)` writes the schema of one field, and
376+
`declaration_json_schema_v3(kind, *, context=None)` writes the schema of one declaration, and
375377
`json_schema` writes the schema of any TypedDict as `check` reads it. All
376378
three are in `zarr_metadata.json_schema`. A schema describes each member on its own, not the rules that
377379
relate members to each other, so a document the schema accepts may still

‎packages/zarr-metadata/changes/4574.bugfix.1.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
writes it. A record's problem was located under a `fields` key the
33
document does not have, at `("dtype", "fields", 0, 1)`; it is now at
44
`("dtype", 0, 1)`, and carries the value found there as its `input`.
5-
`fields()` places each record's type there too. A definition's
5+
`declarations()` places each record's type there too. A definition's
66
functions still find the record types in `nested` at
77
`("fields", 0, 1)`, their place in the configuration.

‎packages/zarr-metadata/changes/4574.bugfix.7.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
A definition's rules are located where the document writes the field. A
1+
A definition's rules are located where the document writes the declaration. A
22
`bytes` codec written as `"bytes"` or `{"name": "bytes"}`, with values
33
of several bytes, reports its missing `endian` at
44
`codecs.N.configuration`, the configuration it does not write, as a

‎packages/zarr-metadata/changes/4574.bugfix.9.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
problems in the merged document, so a `.zarray` that wrote `attributes`
66
beside a `.zattrs` that was not an object gave two problems at
77
`("attributes",)`, a location in neither file. An array node's reading
8-
locates its fields there too: its new `at` is `(".zarray",)`, so
9-
`fields()` gives `(".zarray", "dtype")` and `with_problems` pairs each
10-
field with its problems. Its model reads the merged document, so the
11-
model's own `reading` locates the field at `("dtype",)`, and no longer
8+
locates its declarations there too: its new `at` is `(".zarray",)`, so
9+
`declarations()` gives `(".zarray", "dtype")` and `with_problems` pairs each
10+
declaration with its problems. Its model reads the merged document, so the
11+
model's own `reading` locates the declaration at `("dtype",)`, and no longer
1212
equals the node reading.
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Added `zarr_metadata.testing.strategies`, Hypothesis strategies that draw valid Zarr v3 metadata fields, behind a new `testing` extra (`pip install zarr-metadata[testing]`). There is one strategy for every codec, data type, chunk grid and chunk key encoding the package defines that holds no other field. `fields` draws every envelope spelling the spec allows, or only the canonical one with `spellings="canonical"`. Codecs can be drawn for the `Chunk` they are handed, chunk grids for an array shape, and fill values for a data type. The strategies are a mapping keyed by definition, so an extension author can add strategies for their own definitions.
1+
Added `zarr_metadata.testing.strategies`, Hypothesis strategies that draw valid Zarr v3 metadata declarations, behind a new `testing` extra (`pip install zarr-metadata[testing]`). There is one strategy for every codec, data type, chunk grid and chunk key encoding the package defines that holds no other declaration. `declarations` draws every envelope spelling the spec allows, or only the canonical one with `spellings="canonical"`. Codecs can be drawn for the `Chunk` they are handed, chunk grids for an array shape, and fill values for a data type. The strategies are a mapping keyed by definition, so an extension author can add strategies for their own definitions.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
Rename the word for a value at an extension point: it is now a *declaration* (it was a *field*). A name/value pair in a JSON metadata object stays a *member*. The package already said *field* in the JSON sense in `extra_fields`, `ExtraFieldDefinition`, `ZarrV3ExtensionField` and `must_understand_fields`, after the spec's "additional fields", so one word meant two things. The public names change with no aliases:
2+
3+
- `AcceptedField`, `UnclaimedField`, `RefusedField` and `ResolvedField` are now `AcceptedDeclaration`, `UnclaimedDeclaration`, `RefusedDeclaration` and `ResolvedDeclaration`.
4+
- `ZarrV3DataTypeField`, `ZarrV3ChunkGridField`, `ZarrV3ChunkKeyEncodingField`, `ZarrV3CodecField`, `ZarrV3StaticCodecField`, `ZarrV3StorageTransformerField` and `ZarrV2DataTypeField` are now the same names ending in `Declaration`, and a kind's `field_aliases` is now `declaration_aliases`.
5+
- `ZarrV3MetadataFieldJSON` is now `ZarrV3DeclarationJSON`, and `field_json_schema_v3` is now `declaration_json_schema_v3`.
6+
- A reading's `fields()` is now `declarations()`, and a model's `unclaimed_fields` is now `unclaimed_declarations`.
7+
- The first parameter of `with_problems` is now `declarations` (it was `fields`).
8+
- The JSON Schemas that `node_metadata_json_schema_v3` and `declaration_json_schema_v3` write name each alias under `$defs` by its new name, so a `$ref` to `#/$defs/ZarrV3CodecField` is now `#/$defs/ZarrV3CodecDeclaration`, and likewise for the other kinds.
9+
10+
Messages that said *field* or *metadata field* for an extension-point value now say *declaration*, as in "expected a declaration (string or extension object)". Prose that said *extra member* or *extension member* now says *extra field* and *extension field*, after the names. `extra_fields`, `ExtraFieldDefinition`, `ZarrV3ExtensionField`, `StructField` and the pydantic `...MetadataField` types keep their names.

‎packages/zarr-metadata/docs/api/index.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ title: API reference
77
The package is organized to mirror the structure of the Zarr specifications:
88

99
- [`zarr_metadata.json_schema`](json_schema.md) — JSON Schema output for
10-
any `TypedDict`, one v3 field, or a whole v3 `zarr.json`
10+
any `TypedDict`, one v3 declaration, or a whole v3 `zarr.json`
1111
- [`zarr_metadata.model`](model.md) — document models (each one a
1212
document and the scope it was read in), the readers that return
1313
problems instead of raising them, and the `UNSET` sentinel
@@ -30,8 +30,8 @@ The package is organized to mirror the structure of the Zarr specifications:
3030
[chunk key encodings](v3/chunk_key_encoding.md), [codecs](v3/codec.md),
3131
and [data types](v3/data_type.md)
3232
- [`zarr_metadata.definition`](definition.md) — what both formats share
33-
for reading a field: `Definition`, `resolve`, the three results of a
34-
reading (`AcceptedField`, `UnclaimedField`, `RefusedField`), and the
33+
for reading a declaration: `Definition`, `resolve`, the three results of a
34+
reading (`AcceptedDeclaration`, `UnclaimedDeclaration`, `RefusedDeclaration`), and the
3535
scope type `Context`. Its module docstring is the guide
3636
- [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds
3737
of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec

‎packages/zarr-metadata/docs/api/v3/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ title: v3
66
options:
77
members: false
88

9-
::: zarr_metadata.v3.ZarrV3MetadataFieldJSON
9+
::: zarr_metadata.v3.ZarrV3DeclarationJSON
1010

1111
::: zarr_metadata.v3.ZarrV3NamedConfigJSON
1212

0 commit comments

Comments
 (0)