Skip to content

Commit 6eb15aa

Browse files
authored
feat: align the validators with zarr-metadata 0.6.0 (#3)
* fix(conformance): check the corpus against zarr-metadata 0.6.0 zarr-metadata 0.6.0 removed the validate_* functions the conformance script imported, so the CI conformance job, which installs the latest release, fails on main. The script now uses the 0.6 readers in their default scopes, and CI and the justfile pin the release, so a new release changes the contract only when the pin and the corpus change. The structural validators follow the 0.6 verdicts: the unknown_key kind, invalid_type for literals of the wrong JSON type, extension name rules, must_understand: false refused at every array extension point, a shallow group reader with a full consolidated_metadata reader (node paths, hierarchy, nested listings), open v2 .zarray documents, typestr and field record checks with exact locations, and node reading in .zmetadata. A new v3_consolidated corpus kind covers the consolidated reader. The Python runner removes the "fields" segment 0.6.0 puts in v2 dtype locations, which the document does not have. Assisted-by: ClaudeCode:claude-opus-5-5 * feat(semantics): read v3 fields through zarr-metadata 0.6.0's definitions The semantic layer is now a port of the definition layer of zarr-metadata 0.6.0: each codec, chunk grid, chunk key encoding and data type in its CORE_AND_EXTENSIONS scope has a definition with a typed configuration (typed.ts ports typed_json.check, so problems land at the same location with the same kind), rules, and, for codecs, chunk rules, transitions and nested pipelines (codec/pipeline.ts ports read_pipeline). definition.ts ports resolve: accepted, refused and unclaimed fields, nested field reading, and the gating that keeps a refused field out of later rules. The corpus gains semantic cases with Python's verdicts as expectations, and a differential run of about 100,000 generated and mutated v3 array documents finds no difference from the Python readers apart from the int/float spelling that JSON.parse cannot keep. Two TS bugs are fixed on the way: a rectilinear grid's chunk lengths may exceed the dimension (the spec says "MUST equal or exceed"), and no rule reads a shape with a negative length. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(validation): compare listed groups and refuse an empty codec id A differential run against zarr-metadata 0.6.0 found two structural rules the port lacked. A group listed both in a consolidated listing and in a listed group's own listing must read the same in both: the same attributes and extra fields. Arrays are not compared yet, since the reference compares their canonical spellings. A v2 codec id must not be empty. Assisted-by: ClaudeCode:claude-opus-5-5 * feat: read and write non-finite numbers in attributes zarr-python writes attributes with the defaults of Python's json module, so a NaN attribute is stored as a bare NaN, which JSON.parse refuses: loadStoreJson failed on such stores. zarr-metadata 0.6.0 reads these tokens and accepts non-finite numbers in user data (attributes, extension fields nothing interprets, a group's consolidated envelope entries, .zattrs entries) while refusing them where the spec interprets a value. loadStoreJson now parses the three tokens outside strings, the validators follow the reference on where they are allowed, and dumpStoreJson takes allowNonFinite to write them back. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: describe the two validation layers against zarr-metadata 0.6.0 The README described the semantic layer as TS-only. It now describes both layers as a port of the 0.6.0 readers, lists the names the semantic layer interprets, the problem kinds, and the known differences: no v2 semantic layer, no comparison of an array listed twice in nested consolidated metadata, and the int/float spelling JSON.parse loses. Assisted-by: ClaudeCode:claude-opus-5-5 * chore: ignore Python bytecode Running the conformance script, or importing it, writes scripts/__pycache__, which a broad git add picked up once. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(validation): read a group's listed documents once in validateMetadataV3 For a group, validateMetadataV3 checked each listed document twice: as user data in the consolidated envelope check, and as a node when it read the listing. A non-JSON value in a listed document, such as a Map in its attributes, was reported twice at the same location. The dispatcher now checks only the envelope before it reads the listed documents. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(changeset): note the removed ChunkGridVerdict export The type was exported from the package root and is gone with the old semantic machinery; the changelog now says so. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(store): refuse a bare NaN or Infinity where an object key belongs loadStoreJson replaced every bare NaN, Infinity and -Infinity outside a string with a marker before parsing, key positions included, so text that is not JSON, such as {NaN: 1}, parsed into an object keyed by the marker. A parse whose result holds a marker key now fails as invalid_json, as Python's json.loads does. The new test imports validateMetadataV3 statically, like the rest of the file. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(conformance): rewrite v2 dtype locations only inside .zarray entries The runner removed the "fields" segment from any .zmetadata location whose third part is "dtype", so a .zattrs entry with a user key named dtype would have been rewritten too. Only a .zarray entry's dtype is. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: note the precision of 64-bit integer range checks JSON.parse maps integers past 2^53 to the nearest double, so the edges of the int64, uint64 and numpy time tick ranges are checked only as precisely as a double holds them. The README's known differences and the integer data type factory now say so. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(validation): keep a "__proto__" key as an ordinary key JSON.parse reads a "__proto__" key as an ordinary own key, but plain assignment treats it as the object's prototype. The typed checks, the nested field put-back, and the listed-group comparison copied keys by assignment, so a struct field named "__proto__" got a false missing_key for its fill value, and two group listings differing only in such a member compared equal. Keys are now defined as own data properties. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(validation): check a listed document that is no node as user data validateGroupMetadataV3 checks each document a consolidated listing holds as user data, but the readers that read the listed documents as nodes checked one with no usable node_type only for its node_type and zarr_format, so a non-JSON value in it went unreported there. Its other members are now checked as user data too. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(store): read large documents that hold a non-finite number The fallback parser that reads bare NaN and Infinity tokens rebuilt the text one character at a time, and its key check spread each array into a single push call, which overflows the stack for a large array: a .zmetadata with a long array and a NaN attribute failed as invalid_json. The text is now copied in runs between tokens and the check pushes one element at a time; a 25 MB document parses in about a second. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(validation): tell NaN from null when comparing listed groups Two listings of one group were compared as JSON text written by JSON.stringify, which writes NaN and the infinities as null, so a group listed with a NaN attribute in one place and null in the other compared equal. Non-finite numbers are now written as their own tokens, as Python's json module writes them, and the reference reports the same mismatch. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: say that a .zattrs with no sibling node is not checked validateConsolidatedMetadataV2 follows the reference reader, which forms no node from a .zattrs entry alone and so does not check it to be an object. The deprecated validateConsolidatedDocumentsV2 checked it before; its note and the changeset now say so, and a test pins it. Assisted-by: ClaudeCode:claude-opus-5-5 * fix: return long problem lists instead of throwing RangeError Problem lists were merged with push(...list), which passes each problem as a call argument. Now that loadStoreJson reads bare NaN tokens, a store document can put a large array of non-finite numbers in a field that must be finite, and the validators threw RangeError instead of returning one problem per element. Lists are now appended in a loop, and the smallest inner chunk length is found without spreading either. Assisted-by: ClaudeCode:claude-opus-5-5 * fix(semantics): count consolidated listing depth as the structural layer does The semantic layer started a group's own listing at depth 1 and the structural reader at depth 0, so with the same limit the semantic layer stopped one level earlier. Both now start at 0. Assisted-by: ClaudeCode:claude-opus-5-5 * test(conformance): pin how .zmetadata keys with extra slashes are read zarr-metadata 0.6.0 drops empty segments when it builds a node path from a .zmetadata key, so "/a/.zgroup" and "a//b/.zarray" name the nodes "a" and "a/b". The corpus now records that; the difference from the v3 consolidated path rules is reported in d-v-b/zarr-python#417. Assisted-by: ClaudeCode:claude-opus-5-5
1 parent 72cb4a8 commit 6eb15aa

67 files changed

Lines changed: 9334 additions & 1727 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"zarr-metadata": minor
3+
---
4+
5+
Stores written by zarr-python with a NaN or infinite number in attributes can now be read. zarr-python writes such a value (an xarray `_FillValue` or a CF `missing_value`, for example) as a bare `NaN`, `Infinity` or `-Infinity`, which `loadStoreJson` refused as invalid JSON. It now reads these tokens, and the validators accept non-finite numbers in attributes, in extension fields nothing interprets, and in `.zattrs` entries, as zarr-metadata 0.6.0 does, while still refusing them wherever the spec interprets a value, such as `fill_value`. `dumpStoreJson(value, { allowNonFinite: true })` writes them back as the same tokens.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
"zarr-metadata": minor
3+
---
4+
5+
The semantic layer (`validateArraySemanticsV3`, `validateSemanticsV3`) now applies the rules zarr-metadata 0.6.0 applies in its default scope, and the conformance corpus checks it against that release. Together with the structural validators it reports what the Python readers report.
6+
7+
- Every codec, chunk grid, chunk key encoding and data type the reference defines is read through its definition: `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, `sharding_indexed`, `transpose` and `zstd`; `regular` and `rectilinear`; `default` and `v2`; and every core and convention data type. A configuration value out of range, of the wrong type, or outside its closed set is a problem, and so is a key the configuration does not declare (`unknown_key`). A missing configuration is reported at `<field>.configuration`.
8+
- The codecs are read as a pipeline: array -> array codecs, one array -> bytes codec, then bytes -> bytes codecs, each judged against the chunk it receives. A `bytes` codec needs an `endian` for values of several bytes and refuses variable-length values; a shard's index codecs must be of static size and receive `uint64` values; a shard's inner chunk length is reported per axis.
9+
- Fill values report `invalid_type` for the wrong JSON type and `invalid_value` for a value out of range. A struct needs at least one field, distinct non-empty names, and fixed-size field types.
10+
- Fixed: a rectilinear grid's chunk lengths may exceed the dimension's length, as the spec allows ("MUST equal or exceed"); they were required to sum to it exactly. Fixed: no rule reads a `shape` that has its own problem.
11+
- The `ChunkGridVerdict` type is no longer exported: the semantic layer's internal machinery changed, and the configuration types (`RegularChunkGridConfiguration` and the others) remain the public description of what each name takes.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
"zarr-metadata": minor
3+
---
4+
5+
The structural validators now report what zarr-metadata 0.6.0, the Python reference, reports, and the conformance corpus is checked against that release. This changes some verdicts:
6+
7+
- `must_understand: false` is refused on codecs and storage transformers, as on `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec gives wrong bytes.
8+
- Every v3 metadata field name must be a valid extension name (`^[a-z][a-z0-9-_.]+$`) or a URI.
9+
- A key that a closed object does not declare is reported with the new kind `unknown_key` instead of `invalid_value`. A value of the wrong JSON type for a fixed literal (`zarr_format: "3"`) is `invalid_type`. A metadata field object without a `name` is `missing_key`.
10+
- `consolidated_metadata: null` is refused. `validateGroupMetadataV3` checks only the envelope of `consolidated_metadata`; `validateConsolidatedMetadataV3` reads the listed documents, including the rules that their keys are node paths, that they form a hierarchy, and that a group listed twice reads the same in both listings. `validateMetadataV3` still reads both for a group.
11+
- A v2 `.zarray` may hold members the spec does not define, `filters: []` is accepted, a v2 dtype string must be a NumPy typestr, a codec id must not be empty, and problems in codec ids and field records are located at the id or record. `validateConsolidatedMetadataV2` now reads each `.zarray` and `.zgroup` entry as a node, so `validateConsolidatedDocumentsV2` is deprecated; neither checks a `.zattrs` entry that has no node beside it.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,5 +26,6 @@ jobs:
2626
- uses: actions/checkout@v4
2727
- uses: astral-sh/setup-uv@v5
2828
# The shared corpus is the contract with the Python reference
29-
# implementation; this runs it against the released PyPI package.
30-
- run: uv run --with zarr-metadata --no-project python scripts/check_conformance.py
29+
# implementation; this runs it against the pinned PyPI release, so a new
30+
# release changes the contract only when the pin and the corpus change.
31+
- run: uv run --with 'zarr-metadata==0.6.0' --no-project python scripts/check_conformance.py

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@ dist/
33
*.vsix
44
*.tsbuildinfo
55
site/
6+
__pycache__/

‎README.md‎

Lines changed: 43 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -76,20 +76,44 @@ package's flat problem lists; `flattenTree`/`treeOf` convert between the
7676
two, and the flat path+kind form remains the cross-language interchange
7777
format the conformance corpus asserts on.
7878

79-
The structural validators check key presence, value shapes, and fixed
80-
literals without interpreting extension points, matching the Python
81-
package's layering. The semantic layer (`validateSemanticsV3` /
82-
`validateArraySemanticsV3`) is TS-only and sits on top, organized by
83-
content: `chunk-grid/`, `codec/`, and `data-type/` hold one module per
84-
interpreted name, each with its syntax (configuration and fill-value
85-
types) and semantics (required members, cross-field prose rules — grid
86-
arity and sums, transpose permutations, sharding divisibility, fill_value
87-
vs data type). Interpreted data types cover the core scalars, the `r<N>`
88-
raw-bits family, and the established zarr-extensions conventions —
89-
`string`, `bytes`, `numpy.datetime64`/`timedelta64`, and `struct` (whose
90-
fill is judged per field, recursively). `semantics.ts` is the
91-
document-walking orchestrator. Unrecognized names are skipped — the
92-
extension name space is open.
79+
Validation has two layers, and together they report what the Python
80+
readers (`read_array_metadata_v3` and friends, zarr-metadata 0.6.0) report
81+
in their default scope:
82+
83+
- The structural validators (`validate*`) check key presence, value
84+
shapes, fixed literals, the metadata field envelope and its extension
85+
names, v2 typestrs and field records, and the hierarchy a consolidated
86+
listing forms, without interpreting what a name means.
87+
- The semantic layer (`validateSemanticsV3` / `validateArraySemanticsV3`)
88+
reads each v3 extension point through the definition that claims its
89+
name, organized by content: `chunk-grid/`, `chunk-key-encoding/`,
90+
`codec/`, and `data-type/` hold one module per interpreted name, each
91+
with its configuration type and rules. Then the cross-field rules run:
92+
the fill value against the data type, the chunk grid against the shape,
93+
and the codecs as a pipeline, each judged against the chunk it receives
94+
(array -> array, then one array -> bytes, then bytes -> bytes; `endian`
95+
for multi-byte values; a shard's inner chunks and its `uint64` index).
96+
The interpreted names are those of the reference's
97+
`CORE_AND_EXTENSIONS` scope: the codecs `blosc`, `bytes`, `cast_value`,
98+
`crc32c`, `gzip`, `scale_offset`, `sharding_indexed`, `transpose` and
99+
`zstd`; the `regular` and `rectilinear` grids; the `default` and `v2`
100+
chunk key encodings; and the core data types, `r<N>` raw bits, `string`,
101+
`bytes`, `numpy.datetime64`/`timedelta64` and `struct`. Unrecognized
102+
names are left unjudged — the extension name space is open — and so is
103+
everything that would depend on them.
104+
105+
A value of the wrong JSON type is an `invalid_type`, a value of the right
106+
type outside its closed set or range is an `invalid_value`, and a key a
107+
closed object does not declare is an `unknown_key`.
108+
109+
Known differences from the reference implementation: v2 documents get
110+
the structural checks only (the reference also checks numpy dtype
111+
families, fill values against the dtype, and numcodecs configurations);
112+
two listings of one array in nested consolidated metadata are not
113+
compared; and `JSON.parse` cannot tell `1.0` from `1`, nor two integers
114+
past 2^53 that round to the same double, so the edges of the 64-bit
115+
integer ranges (`int64`, `uint64`, numpy time ticks) are checked only as
116+
precisely as a double holds them.
93117

94118
Consumers include the
95119
[Zarr Metadata VS Code extension](https://github.com/d-v-b/vscode-zarr).
@@ -98,9 +122,10 @@ Consumers include the
98122

99123
The [conformance corpus](https://github.com/d-v-b/zarr-metadata.js/tree/main/conformance) is the contract with the Python
100124
reference implementation: shared JSON fixtures asserting the exact problem
101-
set (loc + kind) each document must produce. Both test suites run every
102-
case; a spec change updates the corpus first, turning both implementations
103-
red until each is fixed.
125+
set (loc + kind) each document must produce, for structure and semantics
126+
alike. Both test suites run every case, the Python side against a pinned
127+
release; a spec change updates the corpus first, turning both
128+
implementations red until each is fixed.
104129

105130
## Development
106131

@@ -113,7 +138,7 @@ just install # npm install
113138
just build # tsc → dist/
114139
just test # vitest: conformance corpus + unit tests
115140
just typecheck # sources and tests, including type-level assertions
116-
just conformance # corpus vs the Python reference (PyPI by default)
141+
just conformance # corpus vs the Python reference (the pinned PyPI release by default)
117142
just check # everything CI runs
118143
```
119144

‎conformance/README.md‎

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,24 @@ both implementations red until each is fixed.
77

88
## Format
99

10-
One JSON file per document kind (`v3_array.json`, `v3_group.json`,
11-
`v2_array.json`, `v2_group.json`, `v2_consolidated.json`), each an array of
12-
cases:
10+
One JSON file per document kind, each an array of cases:
11+
12+
| file | document | Python reader | TypeScript |
13+
| - | - | - | - |
14+
| `v3_array.json` | a v3 array `zarr.json` | `read_array_metadata_v3` | `validateArrayMetadataV3` + `validateArraySemanticsV3` |
15+
| `v3_group.json` | a v3 group `zarr.json` | `read_group_metadata_v3` | `validateGroupMetadataV3` |
16+
| `v3_consolidated.json` | a v3 group's `consolidated_metadata` member | `read_consolidated_metadata_v3` | `validateConsolidatedMetadataV3` + the semantic layer on each listed array |
17+
| `v2_array.json` | a merged `.zarray` + `.zattrs` | `read_array_metadata_v2` | `validateArrayMetadataV2` |
18+
| `v2_group.json` | a merged `.zgroup` + `.zattrs` | `read_group_metadata_v2` | `validateGroupMetadataV2` |
19+
| `v2_consolidated.json` | a `.zmetadata` | `read_consolidated_metadata_v2` | `validateConsolidatedMetadataV2` |
20+
21+
Each Python reader runs in its default scope (`CORE_AND_EXTENSIONS` for v3,
22+
`CORE_V2` for v2), so the expected problems are its full verdict: structure,
23+
and the rules of every extension the scope defines. The TypeScript package
24+
has no v2 semantic layer yet, so v2 fixtures keep their dtypes, fill values
25+
and codec configurations valid apart from what each case tests.
26+
27+
A case looks like this:
1328

1429
```json
1530
{
@@ -21,9 +36,10 @@ cases:
2136
}
2237
```
2338

24-
- An empty `problems` array means the document is structurally valid.
39+
- An empty `problems` array means the document is valid.
2540
- Expected problems assert on `loc` (path from document root) and `kind`
26-
(`missing_key` | `invalid_type` | `invalid_value` | `invalid_json`) only.
41+
(`missing_key` | `invalid_type` | `invalid_value` | `invalid_json` |
42+
`unknown_key`) only.
2743
Human-readable messages are deliberately NOT asserted: they legitimately
2844
differ between languages (Python `repr` vs JS formatting).
2945
- Comparison is order-insensitive: implementations may emit problems in any
@@ -39,6 +55,11 @@ cases:
3955
distinguishes them. Python rejects `zarr_format: 3.0`; JS structurally
4056
cannot.
4157

58+
- The Python runner removes one known location defect of the reference
59+
implementation before comparing: zarr-metadata 0.6.0 reports a problem in
60+
a v2 field record at `["dtype", "fields", 0]`, a location the document
61+
does not have, where the corpus holds `["dtype", 0]`.
62+
4263
## Runners
4364

4465
- TypeScript: `test/conformance.test.ts` (vitest).

0 commit comments

Comments
 (0)