Repository navigation
Commit 6eb15aa
authored
feat: align the validators with zarr-metadata 0.6.0 (#3)
* fix(conformance): check the corpus against zarr-metadata 0.6.0
zarr-metadata 0.6.0 removed the validate_* functions the conformance
script imported, so the CI conformance job, which installs the latest
release, fails on main. The script now uses the 0.6 readers in their
default scopes, and CI and the justfile pin the release, so a new
release changes the contract only when the pin and the corpus change.
The structural validators follow the 0.6 verdicts: the unknown_key
kind, invalid_type for literals of the wrong JSON type, extension name
rules, must_understand: false refused at every array extension point,
a shallow group reader with a full consolidated_metadata reader (node
paths, hierarchy, nested listings), open v2 .zarray documents, typestr
and field record checks with exact locations, and node reading in
.zmetadata. A new v3_consolidated corpus kind covers the consolidated
reader. The Python runner removes the "fields" segment 0.6.0 puts in
v2 dtype locations, which the document does not have.
Assisted-by: ClaudeCode:claude-opus-5-5
* feat(semantics): read v3 fields through zarr-metadata 0.6.0's definitions
The semantic layer is now a port of the definition layer of
zarr-metadata 0.6.0: each codec, chunk grid, chunk key encoding and data
type in its CORE_AND_EXTENSIONS scope has a definition with a typed
configuration (typed.ts ports typed_json.check, so problems land at the
same location with the same kind), rules, and, for codecs, chunk rules,
transitions and nested pipelines (codec/pipeline.ts ports read_pipeline).
definition.ts ports resolve: accepted, refused and unclaimed fields,
nested field reading, and the gating that keeps a refused field out of
later rules.
The corpus gains semantic cases with Python's verdicts as expectations,
and a differential run of about 100,000 generated and mutated v3 array
documents finds no difference from the Python readers apart from the
int/float spelling that JSON.parse cannot keep.
Two TS bugs are fixed on the way: a rectilinear grid's chunk lengths may
exceed the dimension (the spec says "MUST equal or exceed"), and no rule
reads a shape with a negative length.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(validation): compare listed groups and refuse an empty codec id
A differential run against zarr-metadata 0.6.0 found two structural
rules the port lacked. A group listed both in a consolidated listing and
in a listed group's own listing must read the same in both: the same
attributes and extra fields. Arrays are not compared yet, since the
reference compares their canonical spellings. A v2 codec id must not be
empty.
Assisted-by: ClaudeCode:claude-opus-5-5
* feat: read and write non-finite numbers in attributes
zarr-python writes attributes with the defaults of Python's json module,
so a NaN attribute is stored as a bare NaN, which JSON.parse refuses:
loadStoreJson failed on such stores. zarr-metadata 0.6.0 reads these
tokens and accepts non-finite numbers in user data (attributes, extension
fields nothing interprets, a group's consolidated envelope entries,
.zattrs entries) while refusing them where the spec interprets a value.
loadStoreJson now parses the three tokens outside strings, the
validators follow the reference on where they are allowed, and
dumpStoreJson takes allowNonFinite to write them back.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs: describe the two validation layers against zarr-metadata 0.6.0
The README described the semantic layer as TS-only. It now describes
both layers as a port of the 0.6.0 readers, lists the names the semantic
layer interprets, the problem kinds, and the known differences: no v2
semantic layer, no comparison of an array listed twice in nested
consolidated metadata, and the int/float spelling JSON.parse loses.
Assisted-by: ClaudeCode:claude-opus-5-5
* chore: ignore Python bytecode
Running the conformance script, or importing it, writes
scripts/__pycache__, which a broad git add picked up once.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(validation): read a group's listed documents once in validateMetadataV3
For a group, validateMetadataV3 checked each listed document twice: as
user data in the consolidated envelope check, and as a node when it read
the listing. A non-JSON value in a listed document, such as a Map in its
attributes, was reported twice at the same location. The dispatcher now
checks only the envelope before it reads the listed documents.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs(changeset): note the removed ChunkGridVerdict export
The type was exported from the package root and is gone with the old
semantic machinery; the changelog now says so.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(store): refuse a bare NaN or Infinity where an object key belongs
loadStoreJson replaced every bare NaN, Infinity and -Infinity outside a
string with a marker before parsing, key positions included, so text
that is not JSON, such as {NaN: 1}, parsed into an object keyed by the
marker. A parse whose result holds a marker key now fails as
invalid_json, as Python's json.loads does. The new test imports
validateMetadataV3 statically, like the rest of the file.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(conformance): rewrite v2 dtype locations only inside .zarray entries
The runner removed the "fields" segment from any .zmetadata location
whose third part is "dtype", so a .zattrs entry with a user key named
dtype would have been rewritten too. Only a .zarray entry's dtype is.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs: note the precision of 64-bit integer range checks
JSON.parse maps integers past 2^53 to the nearest double, so the edges
of the int64, uint64 and numpy time tick ranges are checked only as
precisely as a double holds them. The README's known differences and
the integer data type factory now say so.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(validation): keep a "__proto__" key as an ordinary key
JSON.parse reads a "__proto__" key as an ordinary own key, but plain
assignment treats it as the object's prototype. The typed checks, the
nested field put-back, and the listed-group comparison copied keys by
assignment, so a struct field named "__proto__" got a false missing_key
for its fill value, and two group listings differing only in such a
member compared equal. Keys are now defined as own data properties.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(validation): check a listed document that is no node as user data
validateGroupMetadataV3 checks each document a consolidated listing
holds as user data, but the readers that read the listed documents as
nodes checked one with no usable node_type only for its node_type and
zarr_format, so a non-JSON value in it went unreported there. Its other
members are now checked as user data too.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(store): read large documents that hold a non-finite number
The fallback parser that reads bare NaN and Infinity tokens rebuilt the
text one character at a time, and its key check spread each array into
a single push call, which overflows the stack for a large array: a
.zmetadata with a long array and a NaN attribute failed as invalid_json.
The text is now copied in runs between tokens and the check pushes one
element at a time; a 25 MB document parses in about a second.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(validation): tell NaN from null when comparing listed groups
Two listings of one group were compared as JSON text written by
JSON.stringify, which writes NaN and the infinities as null, so a group
listed with a NaN attribute in one place and null in the other compared
equal. Non-finite numbers are now written as their own tokens, as
Python's json module writes them, and the reference reports the same
mismatch.
Assisted-by: ClaudeCode:claude-opus-5-5
* docs: say that a .zattrs with no sibling node is not checked
validateConsolidatedMetadataV2 follows the reference reader, which forms
no node from a .zattrs entry alone and so does not check it to be an
object. The deprecated validateConsolidatedDocumentsV2 checked it
before; its note and the changeset now say so, and a test pins it.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix: return long problem lists instead of throwing RangeError
Problem lists were merged with push(...list), which passes each problem
as a call argument. Now that loadStoreJson reads bare NaN tokens, a
store document can put a large array of non-finite numbers in a field
that must be finite, and the validators threw RangeError instead of
returning one problem per element. Lists are now appended in a loop, and
the smallest inner chunk length is found without spreading either.
Assisted-by: ClaudeCode:claude-opus-5-5
* fix(semantics): count consolidated listing depth as the structural layer does
The semantic layer started a group's own listing at depth 1 and the
structural reader at depth 0, so with the same limit the semantic layer
stopped one level earlier. Both now start at 0.
Assisted-by: ClaudeCode:claude-opus-5-5
* test(conformance): pin how .zmetadata keys with extra slashes are read
zarr-metadata 0.6.0 drops empty segments when it builds a node path from
a .zmetadata key, so "/a/.zgroup" and "a//b/.zarray" name the nodes "a"
and "a/b". The corpus now records that; the difference from the v3
consolidated path rules is reported in d-v-b/zarr-python#417.
Assisted-by: ClaudeCode:claude-opus-5-51 parent 72cb4a8 commit 6eb15aa
67 files changed
Lines changed: 9334 additions & 1727 deletions
File tree
- .changeset
- .github/workflows
- conformance
- scripts
- src
- chunk-grid
- chunk-key-encoding
- codec
- data-type
- test
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
93 | 117 | | |
94 | 118 | | |
95 | 119 | | |
| |||
98 | 122 | | |
99 | 123 | | |
100 | 124 | | |
101 | | - | |
102 | | - | |
103 | | - | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
104 | 129 | | |
105 | 130 | | |
106 | 131 | | |
| |||
113 | 138 | | |
114 | 139 | | |
115 | 140 | | |
116 | | - | |
| 141 | + | |
117 | 142 | | |
118 | 143 | | |
119 | 144 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
12 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
13 | 28 | | |
14 | 29 | | |
15 | 30 | | |
| |||
21 | 36 | | |
22 | 37 | | |
23 | 38 | | |
24 | | - | |
| 39 | + | |
25 | 40 | | |
26 | | - | |
| 41 | + | |
| 42 | + | |
27 | 43 | | |
28 | 44 | | |
29 | 45 | | |
| |||
39 | 55 | | |
40 | 56 | | |
41 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
42 | 63 | | |
43 | 64 | | |
44 | 65 | | |
| |||
0 commit comments