Skip to content

feat: align the validators with zarr-metadata 0.6.0 - #3

Merged
d-v-b merged 19 commits into
mainfrom
feat/zarr-metadata-0.6
Oct 28, 2026
Merged

d-v-b merged 19 commits into
mainfrom
feat/zarr-metadata-0.6

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 28, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

The validators now report what zarr-metadata 0.6.0, the Python reference, reports, and the conformance corpus is checked against that release. The semantic layer is a port of the 0.6.0 definitions for every codec, chunk grid, chunk key encoding and data type in CORE_AND_EXTENSIONS, and stores with NaN or infinite numbers in attributes can now be read.

Why

zarr-metadata 0.6.0 removed the validate_* functions that scripts/check_conformance.py imported. The CI conformance job installs the latest release, so it fails on main since 0.6.0 was published. The script now uses the 0.6 read_* readers in their default scopes, and CI and the justfile pin zarr-metadata==0.6.0.

Changes

  • Structural layer (validation.ts), following the 0.6 verdicts:
    • a new problem kind, unknown_key;
    • invalid_type for a fixed literal of the wrong JSON type;
    • extension name rules;
    • must_understand: false refused at every array extension point, codecs and storage transformers included;
    • consolidated_metadata: null refused;
    • validateGroupMetadataV3 checks only the consolidated envelope, while validateConsolidatedMetadataV3 reads the listed documents (node paths, hierarchy, nested listings) and validateMetadataV3 still reads both;
    • v2 .zarray extra members and filters: [] accepted, typestr and field record checks with exact locations, and .zmetadata entries read as nodes.
  • Semantic layer (definition.ts, typed.ts, codec/, chunk-grid/, chunk-key-encoding/, data-type/):
    • fields are read through definitions with typed configurations, rules, and a codec pipeline reader;
    • typed.ts ports typed_json.check, so problems land at the same location with the same kind.
  • TS bugs fixed:
    • rectilinear chunk lengths may exceed the dimension, as the spec allows;
    • no rule reads a shape with a negative length.
  • Non-finite numbers: loadStoreJson reads the bare NaN/Infinity/-Infinity tokens zarr-python writes, the validators accept them where the reference does, and dumpStoreJson takes allowNonFinite.
  • Corpus: a new v3_consolidated kind, and semantic cases whose expected problems are the Python reader's output: 184 cases in all.

Testing

  • npm run typecheck, npx vitest run (291 tests), and the Python conformance run against 0.6.0 (184/184) pass.
  • Five roborev reviews of the whole branch (jobs 1112, 1116, 1120, 1124, 1127). Each finding was fixed in its own commit or answered and closed. The fixes include a .pyc removed from history, {NaN: 1} refused as invalid JSON, __proto__ keys kept as ordinary keys, and long problem lists no longer throwing RangeError.
  • A differential run of about 100,000 generated and mutated documents through both implementations finds no unexplained differences:

Not in this PR

  • A v2 semantic layer (numpy dtype families, fill values against the dtype, numcodecs configurations).
  • Comparing an array listed twice in nested consolidated metadata, which needs canonical spellings of every field.
Notes for reviewers

🤖 Generated with Claude Code

d-v-b added 19 commits October 28, 2026 18:55
zarr-metadata 0.6.0 removed the validate_* functions the conformance
script imported, so the CI conformance job, which installs the latest
release, fails on main. The script now uses the 0.6 readers in their
default scopes, and CI and the justfile pin the release, so a new
release changes the contract only when the pin and the corpus change.

The structural validators follow the 0.6 verdicts: the unknown_key
kind, invalid_type for literals of the wrong JSON type, extension name
rules, must_understand: false refused at every array extension point,
a shallow group reader with a full consolidated_metadata reader (node
paths, hierarchy, nested listings), open v2 .zarray documents, typestr
and field record checks with exact locations, and node reading in
.zmetadata. A new v3_consolidated corpus kind covers the consolidated
reader. The Python runner removes the "fields" segment 0.6.0 puts in
v2 dtype locations, which the document does not have.

Assisted-by: ClaudeCode:claude-opus-5-5
…ions

The semantic layer is now a port of the definition layer of
zarr-metadata 0.6.0: each codec, chunk grid, chunk key encoding and data
type in its CORE_AND_EXTENSIONS scope has a definition with a typed
configuration (typed.ts ports typed_json.check, so problems land at the
same location with the same kind), rules, and, for codecs, chunk rules,
transitions and nested pipelines (codec/pipeline.ts ports read_pipeline).
definition.ts ports resolve: accepted, refused and unclaimed fields,
nested field reading, and the gating that keeps a refused field out of
later rules.

The corpus gains semantic cases with Python's verdicts as expectations,
and a differential run of about 100,000 generated and mutated v3 array
documents finds no difference from the Python readers apart from the
int/float spelling that JSON.parse cannot keep.

Two TS bugs are fixed on the way: a rectilinear grid's chunk lengths may
exceed the dimension (the spec says "MUST equal or exceed"), and no rule
reads a shape with a negative length.

Assisted-by: ClaudeCode:claude-opus-5-5
A differential run against zarr-metadata 0.6.0 found two structural
rules the port lacked. A group listed both in a consolidated listing and
in a listed group's own listing must read the same in both: the same
attributes and extra fields. Arrays are not compared yet, since the
reference compares their canonical spellings. A v2 codec id must not be
empty.

Assisted-by: ClaudeCode:claude-opus-5-5
zarr-python writes attributes with the defaults of Python's json module,
so a NaN attribute is stored as a bare NaN, which JSON.parse refuses:
loadStoreJson failed on such stores. zarr-metadata 0.6.0 reads these
tokens and accepts non-finite numbers in user data (attributes, extension
fields nothing interprets, a group's consolidated envelope entries,
.zattrs entries) while refusing them where the spec interprets a value.

loadStoreJson now parses the three tokens outside strings, the
validators follow the reference on where they are allowed, and
dumpStoreJson takes allowNonFinite to write them back.

Assisted-by: ClaudeCode:claude-opus-5-5
The README described the semantic layer as TS-only. It now describes
both layers as a port of the 0.6.0 readers, lists the names the semantic
layer interprets, the problem kinds, and the known differences: no v2
semantic layer, no comparison of an array listed twice in nested
consolidated metadata, and the int/float spelling JSON.parse loses.

Assisted-by: ClaudeCode:claude-opus-5-5
Running the conformance script, or importing it, writes
scripts/__pycache__, which a broad git add picked up once.

Assisted-by: ClaudeCode:claude-opus-5-5
…dataV3

For a group, validateMetadataV3 checked each listed document twice: as
user data in the consolidated envelope check, and as a node when it read
the listing. A non-JSON value in a listed document, such as a Map in its
attributes, was reported twice at the same location. The dispatcher now
checks only the envelope before it reads the listed documents.

Assisted-by: ClaudeCode:claude-opus-5-5
The type was exported from the package root and is gone with the old
semantic machinery; the changelog now says so.

Assisted-by: ClaudeCode:claude-opus-5-5
loadStoreJson replaced every bare NaN, Infinity and -Infinity outside a
string with a marker before parsing, key positions included, so text
that is not JSON, such as {NaN: 1}, parsed into an object keyed by the
marker. A parse whose result holds a marker key now fails as
invalid_json, as Python's json.loads does. The new test imports
validateMetadataV3 statically, like the rest of the file.

Assisted-by: ClaudeCode:claude-opus-5-5
The runner removed the "fields" segment from any .zmetadata location
whose third part is "dtype", so a .zattrs entry with a user key named
dtype would have been rewritten too. Only a .zarray entry's dtype is.

Assisted-by: ClaudeCode:claude-opus-5-5
JSON.parse maps integers past 2^53 to the nearest double, so the edges
of the int64, uint64 and numpy time tick ranges are checked only as
precisely as a double holds them. The README's known differences and
the integer data type factory now say so.

Assisted-by: ClaudeCode:claude-opus-5-5
JSON.parse reads a "__proto__" key as an ordinary own key, but plain
assignment treats it as the object's prototype. The typed checks, the
nested field put-back, and the listed-group comparison copied keys by
assignment, so a struct field named "__proto__" got a false missing_key
for its fill value, and two group listings differing only in such a
member compared equal. Keys are now defined as own data properties.

Assisted-by: ClaudeCode:claude-opus-5-5
validateGroupMetadataV3 checks each document a consolidated listing
holds as user data, but the readers that read the listed documents as
nodes checked one with no usable node_type only for its node_type and
zarr_format, so a non-JSON value in it went unreported there. Its other
members are now checked as user data too.

Assisted-by: ClaudeCode:claude-opus-5-5
The fallback parser that reads bare NaN and Infinity tokens rebuilt the
text one character at a time, and its key check spread each array into
a single push call, which overflows the stack for a large array: a
.zmetadata with a long array and a NaN attribute failed as invalid_json.
The text is now copied in runs between tokens and the check pushes one
element at a time; a 25 MB document parses in about a second.

Assisted-by: ClaudeCode:claude-opus-5-5
Two listings of one group were compared as JSON text written by
JSON.stringify, which writes NaN and the infinities as null, so a group
listed with a NaN attribute in one place and null in the other compared
equal. Non-finite numbers are now written as their own tokens, as
Python's json module writes them, and the reference reports the same
mismatch.

Assisted-by: ClaudeCode:claude-opus-5-5
validateConsolidatedMetadataV2 follows the reference reader, which forms
no node from a .zattrs entry alone and so does not check it to be an
object. The deprecated validateConsolidatedDocumentsV2 checked it
before; its note and the changeset now say so, and a test pins it.

Assisted-by: ClaudeCode:claude-opus-5-5
Problem lists were merged with push(...list), which passes each problem
as a call argument. Now that loadStoreJson reads bare NaN tokens, a
store document can put a large array of non-finite numbers in a field
that must be finite, and the validators threw RangeError instead of
returning one problem per element. Lists are now appended in a loop, and
the smallest inner chunk length is found without spreading either.

Assisted-by: ClaudeCode:claude-opus-5-5
…yer does

The semantic layer started a group's own listing at depth 1 and the
structural reader at depth 0, so with the same limit the semantic layer
stopped one level earlier. Both now start at 0.

Assisted-by: ClaudeCode:claude-opus-5-5
zarr-metadata 0.6.0 drops empty segments when it builds a node path from
a .zmetadata key, so "/a/.zgroup" and "a//b/.zarray" name the nodes "a"
and "a/b". The corpus now records that; the difference from the v3
consolidated path rules is reported in d-v-b/zarr-python#417.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
d-v-b merged commit 6eb15aa into main Oct 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant