Skip to content

Commit c0600ee

Browse files
d-v-bclaude
andcommitted
feat(zarr-metadata): canonicalize a document, or report why it cannot be
One function from a syntactically valid document to either that document in its simplest equivalent spelling or every reason it is not semantically valid. Canonical is defined per metadata variety as the simplest form with the same semantics. The generic half — an entity with nothing to configure collapsing to its bare name, a defaulted `must_understand` dropping while an explicit `false` survives — the model layer's round trip already performs exactly, so it is delegated rather than written twice. What is new is per-variety: blosc drops a `typesize` the spec says is ignored under `noshuffle`, and a rectilinear dimension run-length encodes, since `[size, count]` is the spelling that does not grow with the chunk count. `dimension_names` of nothing but nulls says what omitting it says. Each canonicalization lives with the entity it belongs to, in `v3/codec/blosc.py` and `v3/chunk_grid/rectilinear.py`, because they are pure transforms over already-valid metadata and need nothing from the rules layer. Two spellings that look collapsible are left alone on purpose, and are tested: a dimension-level bare integer repeats until it covers the extent rather than naming a fixed list, and a one-element list names exactly one chunk. They produce identical extents but different coverage verdicts, so collapsing either would change meaning under a resize. Asserted over the generated corpus: canonicalizing twice changes nothing further, and canonicalizing never changes a verdict — the second is what would catch a simplification that quietly says something else. Assisted-by: ClaudeCode:claude-opus-5 Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent aa05d84 commit c0600ee

7 files changed

Lines changed: 392 additions & 1 deletion

File tree

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
Added `canonicalize_array_metadata_v3`, which answers with
2+
`Canonical[T] | Invalid`: a semantically valid document in its simplest
3+
equivalent spelling, or every reason it is not valid.
4+
5+
Canonical is decided per metadata variety, and always means the simplest
6+
form expressing the same semantics. An entity whose configuration carries
7+
nothing collapses to its bare name, and a `must_understand` of `true` —
8+
the default — is dropped while an explicit `false` is kept. `blosc` drops
9+
a `typesize` that `shuffle: "noshuffle"` renders ignored, the spec saying
10+
of that case that "the value is ignored". A rectilinear dimension's chunk
11+
sizes run-length encode, because `[size, count]` is the spelling that does
12+
not grow with the number of chunks. `dimension_names` of nothing but nulls
13+
says what omitting the field says.
14+
15+
Two spellings that look collapsible are deliberately left alone: a
16+
dimension-level bare integer is a *step* that repeats until it covers the
17+
extent, so it is not equivalent to any fixed list, and a one-element list
18+
declares exactly one chunk rather than as many as it takes. Expanding
19+
either would pin a grid that currently adapts.
20+
21+
Two properties are asserted over generated documents: canonicalizing twice
22+
changes nothing further, and canonicalizing never changes a verdict.

‎packages/zarr-metadata/src/zarr_metadata/rules/__init__.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@
33
`zarr_metadata.model` checks JSON structure. This module also checks
44
cross-field constraints such as fill-value compatibility, codec
55
ordering, and dimension counts. Its `validate_*` and `parse_*` functions
6-
mirror the model API.
6+
mirror the model API, and `canonicalize_array_metadata_v3` answers with
7+
either the document in its simplest equivalent spelling or every reason
8+
it is not valid.
79
810
Rules target canonical metadata and may be stricter than readers that
911
coerce inputs. Unknown entity names are left unjudged. Known entities
@@ -12,6 +14,11 @@
1214
round-trips preserve those unmodeled members.
1315
"""
1416

17+
from zarr_metadata.rules._canonical import (
18+
Canonical,
19+
Invalid,
20+
canonicalize_array_metadata_v3,
21+
)
1522
from zarr_metadata.rules._documents import (
1623
parse_array_metadata_v2,
1724
parse_array_metadata_v3,
@@ -34,9 +41,12 @@
3441
"ZARR_V3_ARRAY_RULES",
3542
"ZARR_V3_GROUP",
3643
"ZARR_V3_GROUP_RULES",
44+
"Canonical",
45+
"Invalid",
3746
"Rule",
3847
"RuleCheck",
3948
"applicable",
49+
"canonicalize_array_metadata_v3",
4050
"parse_array_metadata_v2",
4151
"parse_array_metadata_v3",
4252
"parse_group_metadata_v2",
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
"""One document in, one canonical document or one report of why not.
2+
3+
`canonicalize_array_metadata_v3` takes a *syntactically* valid document —
4+
one the model layer has already accepted, so every member is present and
5+
typed as its TypedDict declares — and answers with `Canonical[T] | Invalid`: either the same document in
6+
canonical form or every reason it is not semantically valid. Testing the
7+
literal `valid` field narrows to one or the other.
8+
9+
Canonical means the simplest spelling with the same meaning, decided per
10+
metadata variety:
11+
12+
- an entity whose configuration carries nothing collapses to its bare
13+
name, and a `must_understand` of `true` (the default) is dropped while
14+
an explicit `false` is kept, because that one says something. This part
15+
the model layer already performs, so it is delegated rather than
16+
reimplemented.
17+
- `blosc` drops a `typesize` that `shuffle: "noshuffle"` renders ignored.
18+
- a rectilinear dimension's chunk sizes run-length encode, because that
19+
is the spelling that does not grow with the number of chunks.
20+
- `dimension_names` of nothing but nulls says what omitting the field
21+
says.
22+
23+
Two properties are worth holding on to, and
24+
`tests/rules/test_canonical.py` asserts both: canonicalizing twice
25+
changes nothing further, and canonicalizing never changes a verdict.
26+
"""
27+
28+
from __future__ import annotations
29+
30+
from collections.abc import Mapping
31+
from dataclasses import dataclass
32+
from typing import TYPE_CHECKING, Generic, Literal, TypeVar, cast
33+
34+
from zarr_metadata.model._array import ZarrV3ArrayMetadata
35+
from zarr_metadata.rules._documents import validate_array_metadata_v3
36+
from zarr_metadata.v3._shape import entity_name
37+
from zarr_metadata.v3.chunk_grid.rectilinear import (
38+
RECTILINEAR_CHUNK_GRID_NAME,
39+
canonical_chunk_shapes,
40+
)
41+
from zarr_metadata.v3.codec.blosc import BLOSC_CODEC_NAME
42+
from zarr_metadata.v3.codec.blosc import canonical_configuration as canonical_blosc
43+
44+
if TYPE_CHECKING:
45+
from zarr_metadata.model._validation import ValidationProblem
46+
from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON
47+
from zarr_metadata.v3.chunk_grid.rectilinear import RectilinearDimSpec
48+
49+
DocumentT = TypeVar("DocumentT")
50+
51+
52+
@dataclass(frozen=True, slots=True)
53+
class Canonical(Generic[DocumentT]):
54+
"""A semantically valid document, in its simplest equivalent spelling."""
55+
56+
document: DocumentT
57+
valid: Literal[True] = True
58+
59+
60+
@dataclass(frozen=True, slots=True)
61+
class Invalid:
62+
"""Every reason a document is not semantically valid; never empty."""
63+
64+
problems: tuple[ValidationProblem, ...]
65+
valid: Literal[False] = False
66+
67+
def __post_init__(self) -> None:
68+
if len(self.problems) == 0:
69+
msg = "Invalid requires at least one validation problem"
70+
raise ValueError(msg)
71+
72+
73+
def _canonical_entity(value: object) -> object:
74+
"""One entity's configuration in its simplest equivalent form.
75+
76+
Only varieties with something to say appear here; everything else is
77+
handed to the generic collapse unchanged.
78+
"""
79+
original: object = value
80+
name = entity_name(value)
81+
if name is None or not isinstance(value, Mapping):
82+
return original
83+
entry: Mapping[str, object] = cast("Mapping[str, object]", value)
84+
configuration = entry.get("configuration")
85+
if not isinstance(configuration, Mapping):
86+
return original
87+
members: Mapping[str, object] = cast("Mapping[str, object]", configuration)
88+
if name == BLOSC_CODEC_NAME:
89+
members = canonical_blosc(members)
90+
elif name == RECTILINEAR_CHUNK_GRID_NAME:
91+
shapes = members.get("chunk_shapes")
92+
if isinstance(shapes, tuple):
93+
specs = cast("tuple[RectilinearDimSpec, ...]", shapes)
94+
members = {**members, "chunk_shapes": canonical_chunk_shapes(specs)}
95+
if members is configuration:
96+
return original
97+
return {**entry, "configuration": members}
98+
99+
100+
def _canonical_document(document: Mapping[str, object]) -> dict[str, object]:
101+
"""Per-variety canonicalization, before the generic collapse."""
102+
out = dict(document)
103+
for field in ("chunk_grid", "chunk_key_encoding", "data_type"):
104+
if field in out:
105+
out[field] = _canonical_entity(out[field])
106+
codecs = out.get("codecs")
107+
if isinstance(codecs, tuple):
108+
entries = cast("tuple[object, ...]", codecs)
109+
out["codecs"] = tuple(_canonical_entity(codec) for codec in entries)
110+
names = out.get("dimension_names")
111+
if isinstance(names, tuple) and all(
112+
entry is None for entry in cast("tuple[object, ...]", names)
113+
):
114+
# Every dimension unnamed says what saying nothing says.
115+
del out["dimension_names"]
116+
return out
117+
118+
119+
def canonicalize_array_metadata_v3(
120+
document: ZarrV3ArrayMetadataJSON,
121+
) -> Canonical[ZarrV3ArrayMetadataJSON] | Invalid:
122+
"""`document` in canonical form, or every reason it is not valid.
123+
124+
Expects a document the model layer has already accepted. Passing one
125+
it has not is not an error — the composition problems are reported the
126+
same way — but the structural problems come back too, and the result
127+
is `Invalid` rather than a canonical document.
128+
"""
129+
problems = validate_array_metadata_v3(document)
130+
if len(problems) != 0:
131+
return Invalid(problems)
132+
canonical = _canonical_document(document)
133+
# The generic collapse — shorthand names, defaulted `must_understand` —
134+
# is what the model layer's round trip already performs.
135+
return Canonical(ZarrV3ArrayMetadata.from_json(canonical).to_json())
136+
137+
138+
__all__ = [
139+
"Canonical",
140+
"Invalid",
141+
"canonicalize_array_metadata_v3",
142+
]

‎packages/zarr-metadata/src/zarr_metadata/v3/chunk_grid/rectilinear.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,11 +47,53 @@ class RectilinearChunkGridObject(TypedDict, closed=True):
4747
https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1562-L1564
4848
"""
4949

50+
51+
def canonical_dim_spec(spec: RectilinearDimSpec) -> RectilinearDimSpec:
52+
"""One dimension's chunk sizes in their simplest equivalent form.
53+
54+
Runs of equal sizes collapse to `[size, count]` pairs, because that is
55+
the spelling that does not grow with the number of chunks: a million
56+
equal chunks is two numbers, not a million. A run of one stays a bare
57+
size, and `[size, 1]` collapses to one, since a pair says nothing extra
58+
there. Adjacent spellings of the same size merge, which is what makes
59+
this idempotent: `[[32, 2], 32]` and `[32, [32, 2]]` both become
60+
`[[32, 3]]`.
61+
62+
A dimension-level bare integer is left alone. It is a *step* that
63+
repeats until it covers the extent, so it is not equivalent to any
64+
fixed list — expanding it would pin a grid that currently adapts, and
65+
the two would diverge the moment the array were resized. For the same
66+
reason a one-element list is never collapsed to a bare integer:
67+
`[32]` declares exactly one chunk and `32` declares as many as it takes.
68+
69+
Assumes a spec the shape validator has already accepted.
70+
"""
71+
if not isinstance(spec, tuple):
72+
return spec
73+
runs: list[tuple[int, int]] = []
74+
for entry in spec:
75+
size, count = entry if isinstance(entry, tuple) else (entry, 1)
76+
if len(runs) != 0 and runs[-1][0] == size:
77+
runs[-1] = (size, runs[-1][1] + count)
78+
else:
79+
runs.append((size, count))
80+
return tuple(size if count == 1 else (size, count) for size, count in runs)
81+
82+
83+
def canonical_chunk_shapes(
84+
chunk_shapes: tuple[RectilinearDimSpec, ...],
85+
) -> tuple[RectilinearDimSpec, ...]:
86+
"""Every dimension's chunk sizes in their simplest equivalent form."""
87+
return tuple(canonical_dim_spec(spec) for spec in chunk_shapes)
88+
89+
5090
__all__ = [
5191
"RECTILINEAR_CHUNK_GRID_NAME",
5292
"RectilinearChunkGridConfiguration",
5393
"RectilinearChunkGridMetadata",
5494
"RectilinearChunkGridName",
5595
"RectilinearChunkGridObject",
5696
"RectilinearDimSpec",
97+
"canonical_chunk_shapes",
98+
"canonical_dim_spec",
5799
]

‎packages/zarr-metadata/src/zarr_metadata/v3/codec/blosc.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/blosc/index.html
55
"""
66

7+
from collections.abc import Mapping
78
from typing import Final, Literal, NotRequired
89

910
from typing_extensions import TypedDict
@@ -63,6 +64,22 @@ class BloscCodecObject(TypedDict, closed=True):
6364
https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L1562-L1564 (short-hand names only "if no configuration metadata is required")
6465
"""
6566

67+
68+
def canonical_configuration(configuration: Mapping[str, object]) -> Mapping[str, object]:
69+
"""A blosc configuration in its simplest equivalent form.
70+
71+
Under `shuffle: "noshuffle"` the spec says of `typesize` that "the
72+
value is ignored", so whatever it holds carries no meaning and two
73+
documents differing only there describe the same codec. Dropping it
74+
makes that equality visible.
75+
76+
Assumes a configuration the shape validator has already accepted.
77+
"""
78+
if configuration.get("shuffle") != BLOSC_NO_SHUFFLE or "typesize" not in configuration:
79+
return configuration
80+
return {key: value for key, value in configuration.items() if key != "typesize"}
81+
82+
6683
__all__ = [
6784
"BLOSC_CNAME",
6885
"BLOSC_CODEC_NAME",
@@ -74,4 +91,5 @@ class BloscCodecObject(TypedDict, closed=True):
7491
"BloscCodecName",
7592
"BloscCodecObject",
7693
"BloscShuffle",
94+
"canonical_configuration",
7795
]

0 commit comments

Comments
 (0)