Repository navigation
feat: align the validators with zarr-metadata 0.6.0 - #3
Merged
Merged
Conversation
zarr-metadata 0.6.0 removed the validate_* functions the conformance script imported, so the CI conformance job, which installs the latest release, fails on main. The script now uses the 0.6 readers in their default scopes, and CI and the justfile pin the release, so a new release changes the contract only when the pin and the corpus change. The structural validators follow the 0.6 verdicts: the unknown_key kind, invalid_type for literals of the wrong JSON type, extension name rules, must_understand: false refused at every array extension point, a shallow group reader with a full consolidated_metadata reader (node paths, hierarchy, nested listings), open v2 .zarray documents, typestr and field record checks with exact locations, and node reading in .zmetadata. A new v3_consolidated corpus kind covers the consolidated reader. The Python runner removes the "fields" segment 0.6.0 puts in v2 dtype locations, which the document does not have. Assisted-by: ClaudeCode:claude-opus-5-5
…ions The semantic layer is now a port of the definition layer of zarr-metadata 0.6.0: each codec, chunk grid, chunk key encoding and data type in its CORE_AND_EXTENSIONS scope has a definition with a typed configuration (typed.ts ports typed_json.check, so problems land at the same location with the same kind), rules, and, for codecs, chunk rules, transitions and nested pipelines (codec/pipeline.ts ports read_pipeline). definition.ts ports resolve: accepted, refused and unclaimed fields, nested field reading, and the gating that keeps a refused field out of later rules. The corpus gains semantic cases with Python's verdicts as expectations, and a differential run of about 100,000 generated and mutated v3 array documents finds no difference from the Python readers apart from the int/float spelling that JSON.parse cannot keep. Two TS bugs are fixed on the way: a rectilinear grid's chunk lengths may exceed the dimension (the spec says "MUST equal or exceed"), and no rule reads a shape with a negative length. Assisted-by: ClaudeCode:claude-opus-5-5
A differential run against zarr-metadata 0.6.0 found two structural rules the port lacked. A group listed both in a consolidated listing and in a listed group's own listing must read the same in both: the same attributes and extra fields. Arrays are not compared yet, since the reference compares their canonical spellings. A v2 codec id must not be empty. Assisted-by: ClaudeCode:claude-opus-5-5
zarr-python writes attributes with the defaults of Python's json module, so a NaN attribute is stored as a bare NaN, which JSON.parse refuses: loadStoreJson failed on such stores. zarr-metadata 0.6.0 reads these tokens and accepts non-finite numbers in user data (attributes, extension fields nothing interprets, a group's consolidated envelope entries, .zattrs entries) while refusing them where the spec interprets a value. loadStoreJson now parses the three tokens outside strings, the validators follow the reference on where they are allowed, and dumpStoreJson takes allowNonFinite to write them back. Assisted-by: ClaudeCode:claude-opus-5-5
The README described the semantic layer as TS-only. It now describes both layers as a port of the 0.6.0 readers, lists the names the semantic layer interprets, the problem kinds, and the known differences: no v2 semantic layer, no comparison of an array listed twice in nested consolidated metadata, and the int/float spelling JSON.parse loses. Assisted-by: ClaudeCode:claude-opus-5-5
Running the conformance script, or importing it, writes scripts/__pycache__, which a broad git add picked up once. Assisted-by: ClaudeCode:claude-opus-5-5
…dataV3 For a group, validateMetadataV3 checked each listed document twice: as user data in the consolidated envelope check, and as a node when it read the listing. A non-JSON value in a listed document, such as a Map in its attributes, was reported twice at the same location. The dispatcher now checks only the envelope before it reads the listed documents. Assisted-by: ClaudeCode:claude-opus-5-5
The type was exported from the package root and is gone with the old semantic machinery; the changelog now says so. Assisted-by: ClaudeCode:claude-opus-5-5
loadStoreJson replaced every bare NaN, Infinity and -Infinity outside a
string with a marker before parsing, key positions included, so text
that is not JSON, such as {NaN: 1}, parsed into an object keyed by the
marker. A parse whose result holds a marker key now fails as
invalid_json, as Python's json.loads does. The new test imports
validateMetadataV3 statically, like the rest of the file.
Assisted-by: ClaudeCode:claude-opus-5-5
The runner removed the "fields" segment from any .zmetadata location whose third part is "dtype", so a .zattrs entry with a user key named dtype would have been rewritten too. Only a .zarray entry's dtype is. Assisted-by: ClaudeCode:claude-opus-5-5
JSON.parse maps integers past 2^53 to the nearest double, so the edges of the int64, uint64 and numpy time tick ranges are checked only as precisely as a double holds them. The README's known differences and the integer data type factory now say so. Assisted-by: ClaudeCode:claude-opus-5-5
JSON.parse reads a "__proto__" key as an ordinary own key, but plain assignment treats it as the object's prototype. The typed checks, the nested field put-back, and the listed-group comparison copied keys by assignment, so a struct field named "__proto__" got a false missing_key for its fill value, and two group listings differing only in such a member compared equal. Keys are now defined as own data properties. Assisted-by: ClaudeCode:claude-opus-5-5
validateGroupMetadataV3 checks each document a consolidated listing holds as user data, but the readers that read the listed documents as nodes checked one with no usable node_type only for its node_type and zarr_format, so a non-JSON value in it went unreported there. Its other members are now checked as user data too. Assisted-by: ClaudeCode:claude-opus-5-5
The fallback parser that reads bare NaN and Infinity tokens rebuilt the text one character at a time, and its key check spread each array into a single push call, which overflows the stack for a large array: a .zmetadata with a long array and a NaN attribute failed as invalid_json. The text is now copied in runs between tokens and the check pushes one element at a time; a 25 MB document parses in about a second. Assisted-by: ClaudeCode:claude-opus-5-5
Two listings of one group were compared as JSON text written by JSON.stringify, which writes NaN and the infinities as null, so a group listed with a NaN attribute in one place and null in the other compared equal. Non-finite numbers are now written as their own tokens, as Python's json module writes them, and the reference reports the same mismatch. Assisted-by: ClaudeCode:claude-opus-5-5
validateConsolidatedMetadataV2 follows the reference reader, which forms no node from a .zattrs entry alone and so does not check it to be an object. The deprecated validateConsolidatedDocumentsV2 checked it before; its note and the changeset now say so, and a test pins it. Assisted-by: ClaudeCode:claude-opus-5-5
Problem lists were merged with push(...list), which passes each problem as a call argument. Now that loadStoreJson reads bare NaN tokens, a store document can put a large array of non-finite numbers in a field that must be finite, and the validators threw RangeError instead of returning one problem per element. Lists are now appended in a loop, and the smallest inner chunk length is found without spreading either. Assisted-by: ClaudeCode:claude-opus-5-5
…yer does The semantic layer started a group's own listing at depth 1 and the structural reader at depth 0, so with the same limit the semantic layer stopped one level earlier. Both now start at 0. Assisted-by: ClaudeCode:claude-opus-5-5
zarr-metadata 0.6.0 drops empty segments when it builds a node path from a .zmetadata key, so "/a/.zgroup" and "a//b/.zarray" name the nodes "a" and "a/b". The corpus now records that; the difference from the v3 consolidated path rules is reported in d-v-b/zarr-python#417. Assisted-by: ClaudeCode:claude-opus-5-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
The validators now report what zarr-metadata 0.6.0, the Python reference, reports, and the conformance corpus is checked against that release. The semantic layer is a port of the 0.6.0 definitions for every codec, chunk grid, chunk key encoding and data type in
CORE_AND_EXTENSIONS, and stores withNaNor infinite numbers in attributes can now be read.Why
zarr-metadata 0.6.0 removed the
validate_*functions thatscripts/check_conformance.pyimported. The CI conformance job installs the latest release, so it fails onmainsince 0.6.0 was published. The script now uses the 0.6read_*readers in their default scopes, and CI and the justfile pinzarr-metadata==0.6.0.Changes
validation.ts), following the 0.6 verdicts:unknown_key;invalid_typefor a fixed literal of the wrong JSON type;must_understand: falserefused at every array extension point, codecs and storage transformers included;consolidated_metadata: nullrefused;validateGroupMetadataV3checks only the consolidated envelope, whilevalidateConsolidatedMetadataV3reads the listed documents (node paths, hierarchy, nested listings) andvalidateMetadataV3still reads both;.zarrayextra members andfilters: []accepted, typestr and field record checks with exact locations, and.zmetadataentries read as nodes.definition.ts,typed.ts,codec/,chunk-grid/,chunk-key-encoding/,data-type/):typed.tsportstyped_json.check, so problems land at the same location with the same kind.shapewith a negative length.loadStoreJsonreads the bareNaN/Infinity/-Infinitytokens zarr-python writes, the validators accept them where the reference does, anddumpStoreJsontakesallowNonFinite.v3_consolidatedkind, and semantic cases whose expected problems are the Python reader's output: 184 cases in all.Testing
npm run typecheck,npx vitest run(291 tests), and the Python conformance run against 0.6.0 (184/184) pass..pycremoved from history,{NaN: 1}refused as invalid JSON,__proto__keys kept as ordinary keys, and long problem lists no longer throwingRangeError.1.0vs1, whichJSON.parsecannot tell apart;Not in this PR
Notes for reviewers
fieldssegment that 0.6.0 puts in v2 dtype problem locations (zarr-metadata 0.6.0: v2 dtype problem locations, the "struct" dtype string, and a doubled consolidated metadata problem zarr-python#417, item 1). Remove that once the reference is fixed.validateConsolidatedDocumentsV2is deprecated but still works, and vscode-zarr's dedupe by range and kind absorbs the overlap withvalidateConsolidatedMetadataV2. Several messages changed.ChunkGridVerdicttype export is noted in one of them.🤖 Generated with Claude Code