fix: spec compliance — rectilinear overflow, positive regular chunks, exact int64/uint64 fills by d-v-b · Pull Request #2 · d-v-b/zarr-metadata.js

d-v-b · 2026-09-15T14:51:37Z

🤖 AI text below 🤖

Fixes from a spec-compliance audit against zarr-specs (fc7dd9c) and the zarr-extensions rectilinear README. Every change is in the TS-only semantic layer or the JSON value model. The Python conformance corpus is untouched (67/67 still pass).

Changes

Rectilinear grid was too strict (src/chunk-grid/rectilinear.ts). The spec says "The sum of the edge lengths MUST equal or exceed L. Overflowing L by multiple chunks is permitted." The library required an exact sum, so it rejected documents zarr-python writes after resize shrinks an array. It now reports only sums below the extent.

Regular grid was too lax (src/chunk-grid/regular.ts). The spec says "Chunk sizes must be greater than zero." chunk_shape entries of 0 or less passed, and a 0 even silently satisfied the sharding divisibility check. Each such entry is now reported, and the grid is no longer used when checking sharding divisibility.

int64/uint64 fill ranges were unverifiable. JSON.parse rounds 2^63 and 2^63 - 1 to the same double, so an out-of-range fill passed.

New decodeStoreJson(text | bytes) parses like JSON.parse, but integer literals beyond Number.MAX_SAFE_INTEGER become bigint. It uses the reviver's context.source (Node ≥ 21); on older runtimes such literals round as before. loadStoreJson uses it.
bigint is now a JSON integer in JSONValue, validateJson and dimension sequences.
dumpStoreJson writes bigints as exact integer literals. JSON.stringify alone throws on them.
Integer fills are range-checked exactly against bigint bounds. Number fills beyond the safe range, which come from bare JSON.parse, keep the old lenient double comparison, so a valid max int64 is never rejected.
The README examples now use decodeStoreJson.

The two changesets are patch (grids) and minor (bigint).

Not addressed (from the same audit)

Codec kind/order rules aren't enforced in the library; vscode-zarr enforces them.
bytes endian isn't required for multi-byte types.
index_codecs is never walked.
v2 dtype grammar and fill encodings aren't checked.
v2 .zarray extra keys are errors although the spec says SHOULD NOT; changing that needs a corpus and Python change.
filters: [] is rejected.
Extension name syntax isn't checked.

Test plan

just check: typecheck, vitest (147), and Python conformance (67/67)
New tests:
- valid: bigint bounds, a rectilinear overflow, a bigint float fill
- errors: out-of-range int64/uint64, non-positive regular sizes, rectilinear shortfall, and an end-to-end JSON text → decodeStoreJson → rejection of 2^63
- decodeStoreJson / dumpStoreJson round-trip

🤖 Generated with Claude Code

…gular sizes) - rectilinear: the zarr-extensions spec says "The sum of the edge lengths MUST equal or exceed L. Overflowing L by multiple chunks is permitted." An exact sum was required, rejecting e.g. documents zarr-python writes after shrinking an array. - regular: the spec says "Chunk sizes must be greater than zero"; 0 and negative chunk_shape entries were accepted (and a 0 size silently satisfied sharding divisibility). Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <[email protected]>

JSON.parse rounds 2^63 and 2^63 - 1 to the same double, so an out-of-range int64/uint64 fill value (data-types spec: "within the representable range of the data type") was indistinguishable from a valid one and passed. - decodeStoreJson(text | bytes): JSON.parse, but integer literals beyond Number.MAX_SAFE_INTEGER decode to bigint via reviver source-text access (Node >= 21; rounds as before elsewhere). loadStoreJson uses it. - bigint is a JSON integer: JSONValue, validateJson, dimension sequences. - dumpStoreJson writes bigints as exact integer literals. - Integer fills are range-checked exactly with bigint bounds; float and numpy temporal fills accept bigints. Rounded number fills keep the lenient double comparison so valid extremes aren't rejected. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <[email protected]>

Per the zarr-extensions READMEs (pinned commit 4da7b37): - struct: "Field names MUST be unique"; "Variable-length data types (e.g. "string") MUST NOT be used as field types"; core data types "MUST be a string". Duplicates previously made the fill check go silent instead of reporting. Descriptors gain a configIssues hook, applied recursively through nested struct fields. - numpy.datetime64/timedelta64: fill "within the range [-2^63, 2^63 - 1]" or "NaT"; any integer was accepted. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

d-v-b · 2026-09-15T19:44:35Z

🤖 AI text below 🤖

Third commit, from checking the interpreted zarr-extensions data types against their READMEs (pinned commit 4da7b37):

struct: "Field names MUST be unique" — duplicates made the fill check go silent instead of reporting. "Variable-length data types (e.g. "string") MUST NOT be used as field types" and core field types "MUST be a string" were unchecked. Descriptors gain an optional configIssues hook (ConfigContext), applied recursively through nested struct fields.
numpy.datetime64 / numpy.timedelta64: fill "within the range [-2^63, 2^63 - 1]" — any integer passed (1e19).

string, bytes, r<N> (whose README just points at the core spec) and the struct fill rules match already. Still uninterpreted, by design: the sub-byte ints, bfloat16/float8_*/float4/float6, complex_*, fixed_length_utf32, and the legacy structured alias.

The structural v2 validators (corpus-governed, Python parity) accept any string as dtype and any JSON as fill_value. The v2 spec is stricter: "The byte order MUST be specified"; datetime/timedelta types "MUST also include the units"; float fills "MUST" use "NaN"/"Infinity"/"-Infinity"; fixed-length byte string and structured fills "MUST be encoded as an ASCII string using the standard Base64 alphabet". validateSemanticsV2 / validateArraySemanticsV2 apply these (plus NumPy item sizes, integer ranges, int64-or-"NaT" temporal fills, and structured-dtype field rules), mirroring the TS-only v3 layer; the consolidated form walks .zmetadata's .zarray entries. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

d-v-b · 2026-09-15T19:49:22Z

🤖 AI text below 🤖

Fourth commit: a v2 semantic layer (validateSemanticsV2 / validateArraySemanticsV2, TS-only like the v3 one; corpus untouched, 67/67). The structural validators accept any string as dtype and any JSON as fill_value; the spec says:

"The byte order MUST be specified" — "f8", "float64" now rejected.
datetime/timedelta "MUST also include the units" — "<M8" rejected; NumPy multipliers like "<m8[10s]" accepted.
float fills "MUST" use "NaN"/"Infinity"/"-Infinity" — "nan", "inf" rejected.
fixed-length byte string (|S, |V) and structured fills "MUST be encoded as an ASCII string using the standard Base64 alphabet".

Plus NumPy's item sizes per kind ("<i3" rejected), integer ranges (bigint-exact), int64-or-"NaT" temporal fills, non-scalar fills, and structured-dtype rules (duplicate field names, negative subarray shapes, recursively). validateSemanticsV2 also walks the .zarray entries of a .zmetadata document. null fills are always accepted; |b1 and U fills are only required to be scalars, since the spec fixes no encoding for them.

This removes the "v2 dtype grammar / fill encodings unchecked" items from the not-addressed list. Remaining there: codec kind/order in the library, bytes endian, index_codecs walking, .zarray extra-key severity and filters: [] (both need corpus + Python changes), extension-name syntax.

Two structural v2 rules were stricter than the spec: - .zarray extra members were rejected. The spec: "Other keys SHOULD NOT be present within the metadata object and SHOULD be ignored by implementations" — a recommendation, unlike .zgroup's "Other keys MUST NOT be present". Extras are now tolerated structurally and reported by the v2 semantic layer as advisories. - filters: [] was rejected ("expected at least one filter"). The spec: "A list of JSON objects providing codec configurations, or null" — an empty list is a list; zarr-python only warns on it. Corpus cases 5 and 8 of conformance/v2_array.json now expect no problems. Per conformance/README.md the corpus changes first: the Python reference implementation's conformance run is red until it follows. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

d-v-b · 2026-09-15T19:53:56Z

🤖 AI text below 🤖

Fifth commit: a conformance corpus change, per the corpus-first workflow in conformance/README.md. Two v2 structural rules are stricter than the spec:

Case (`conformance/v2_array.json`)	Spec (`v2.0.rst`)	Was	Now
5 — extra member in `.zarray`	line 91: "Other keys SHOULD NOT be present within the metadata object and SHOULD be ignored by implementations" (vs `.zgroup` line 313: "MUST NOT")	`invalid_value` at the key	no problems
8 — `filters: []`	line 77: "A list of JSON objects providing codec configurations, or `null`" — no minimum	`invalid_value` at `filters`	no problems

TS follows in the same commit (arrayMetadataV2Problems no longer calls unexpectedKeys; the empty-filters check is gone). So the SHOULD NOT isn't lost, validateArraySemanticsV2 now reports extra .zarray members as advisories (invalid_value, message "unexpected document member (the spec says other keys SHOULD NOT be present)"), which vscode-zarr already maps to warnings.

CI on this PR is red by design until the Python package follows: just conformance → 65/67, failing exactly cases 5 and 8. The Python changes needed (paths in zarr-python/packages/zarr-metadata):

src/zarr_metadata/model/_validation.py:589 — drop the _unexpected_keys(ARRAY_METADATA_STANDARD_KEYS_V2, …) call in the v2 array validator (keep the .zgroup one at :804).
src/zarr_metadata/model/_validation.py:647 — drop the "expected at least one filter" problem.
src/zarr_metadata/_pydantic_schema.py:41 — ZarrV2FilterPipelineJSON: remove Field(min_length=1).
src/zarr_metadata/model/_array.py:476 — review the model-level "unexpected document member" for the v2 array model (same rule at the from_json layer).
Tests to invert: tests/model/test_array.py:1384 test_array_v2_rejects_unknown_document_member, tests/model/test_pydantic_module.py:185 test_v2_array_schema_rejects_empty_filters.

.zgroup stays closed on both sides — that one is a MUST NOT.

d-v-b · 2026-09-16T12:32:47Z

🤖 AI text below 🤖

The Python side of the corpus change is up: zarr-developers/zarr-python#4365. Against that checkout, just conformance here is back to 67/67; CI on this PR stays red until it's released to PyPI.

d-v-b · 2026-09-16T13:10:16Z

🤖 AI text below 🤖

Corpus audit note: v3_array.json case 1 declares "data_type": {"name": "int32"} valid, which the current core spec text (data_type section: core types "must be ... provided as a string") contradicts. Left as is deliberately — zarr-developers/zarr-specs#392 / zarr-developers/zarr-specs#393 resolve the spec in the corpus's favour.

Commit-pinned zarr-specs / zarr-extensions permalinks with line ranges, so each citation points at the exact sentence. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

d-v-b and others added 2 commits September 15, 2026 16:34

d-v-b mentioned this pull request Sep 15, 2026

fix: reject on-disk v2 attributes, validate .zattrs, report every missing key d-v-b/vscode-zarr#1

Merged

2 tasks done

d-v-b mentioned this pull request Sep 16, 2026

fix(zarr-metadata): v2 array document is open and filters may be empty zarr-developers/zarr-python#4365

Merged

3 tasks done

d-v-b and others added 3 commits September 16, 2026 15:48

docs: link the spec text cited in docstrings and comments

606e9ec

Commit-pinned zarr-specs / zarr-extensions permalinks with line ranges, so each citation points at the exact sentence. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

docs: put spec permalinks on their own comment lines

796564c

Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

docs: link the spec text behind the pre-existing v2/v3 statements

86dcb85

Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <[email protected]>

Navigation Menu

Repository navigation

fix: spec compliance — rectilinear overflow, positive regular chunks, exact int64/uint64 fills - #2

fix: spec compliance — rectilinear overflow, positive regular chunks, exact int64/uint64 fills#2
d-v-b wants to merge 8 commits into
mainfrom
fix/spec-audit

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 16, 2026

Uh oh!

d-v-b commented Sep 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

d-v-b commented Sep 15, 2026

Changes

Not addressed (from the same audit)

Test plan

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 15, 2026

Uh oh!

d-v-b commented Sep 16, 2026

Uh oh!

d-v-b commented Sep 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant