Skip to content

Commit daef49b

Browse files
authored
feat(zarr-metadata): export base64_fill_value_rules for data type definitions (#4609)
* feat(zarr-metadata): export base64_fill_value_rules for data type definitions Export the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it as `fill_value_rules` or yield from it. `bytes` and `variable_length_bytes` now share it. The parsing and validation guide shows a data type that defines `fill_value_rules`, built with `union`. Rebased onto main as one commit; the earlier history carried the merged ZARR_PYTHON_CONTEXT branch. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragment for #4609 Assisted-by: ClaudeCode:claude-opus-5-5
1 parent e12ca8d commit daef49b

7 files changed

Lines changed: 96 additions & 27 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Export `base64_fill_value_rules` from `zarr_metadata.v3.data_type.bytes`, the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it as `fill_value_rules` for a `Base64Bytes` fill value, or yield from it in a rule of its own. The guide to parsing and validation now shows how to define a data type with `fill_value_rules`.

‎packages/zarr-metadata/docs/parsing-validation.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,4 +271,48 @@ reading = read_array_metadata_v3(raw, context=context)
271271

272272
`union` keeps every definition the context has, and raises `ScopeConflictError` if `MY_CODEC` has a name the context already files with another definition. To replace a definition on purpose, use `copy`, which, like pydantic's `model_copy(update=...)`, replaces whole each slot it is given and keeps the others. A slot can be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS_CONTEXT.copy(update={"codecs": {**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": MY_GZIP}})` replaces the gzip definition with your own. Passing `copy(update={"codecs": [MY_CODEC]})` instead would leave `MY_CODEC` as the only codec.
273273

274+
A data type's definition also says what its fill value may be. `fill_value` is the fill value's JSON shape, as a type annotation. `fill_value_rules` is a function of the configuration, the nested declarations and the fill value, that yields each problem the shape cannot express. For example, `Base64Bytes` is a string, so an annotation accepts any string, and the rule `base64_fill_value_rules` rejects one that is not base64. A rule of your own can yield from it, then check what depends on the configuration:
275+
276+
```python test="true" session="parsing-validation"
277+
import base64
278+
279+
from typing_extensions import TypedDict
280+
281+
from zarr_metadata.definition import fill_value_problems
282+
from zarr_metadata.problems import ValidationProblem
283+
from zarr_metadata.v3.data_type.bytes import Base64Bytes, base64_fill_value_rules
284+
from zarr_metadata.v3.definition import DataTypeDefinition
285+
286+
287+
class PackedConfiguration(TypedDict, closed=True):
288+
length_bytes: int
289+
290+
291+
def packed_fill_value_rules(configuration, nested, value):
292+
# The fill value is base64, of at most `length_bytes` bytes.
293+
problems = list(base64_fill_value_rules(configuration, nested, value))
294+
yield from problems
295+
if not problems and len(base64.b64decode(value)) > configuration["length_bytes"]:
296+
yield ValidationProblem((), "expected fewer bytes than the type holds", "invalid_value")
297+
298+
299+
PACKED = DataTypeDefinition(
300+
name="acme.packed",
301+
configuration=PackedConfiguration,
302+
fill_value=Base64Bytes,
303+
fill_value_rules=packed_fill_value_rules,
304+
)
305+
context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(data_types=[PACKED]))
306+
data_type, _ = resolve(
307+
{"name": "acme.packed", "configuration": {"length_bytes": 2}}, DataTypeDefinition, context
308+
)
309+
for fill_value in ("YWI=", "not base64!", "YWJj"):
310+
print(fill_value, [problem.message for problem in fill_value_problems(data_type, fill_value)])
311+
# YWI= []
312+
# not base64! ['expected standard-alphabet base64, got "not base64!"']
313+
# YWJj ['expected fewer bytes than the type holds']
314+
```
315+
316+
A reader of a document in this context checks its fill value with these rules, and so does a pydantic field. A JSON Schema written for the context does not contain the rules, so it accepts any string here.
317+
274318
[^1]: The one exception is the [`r*` data type](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#data-types), which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects. The declaration `"r16"` keeps its name. The `r*` definition reads it with the configuration `{"bits": 16}`.

‎packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
BytesDataTypeName,
3838
BytesFillValue,
3939
base64_bytes,
40+
base64_fill_value_rules,
4041
)
4142
from zarr_metadata.v3.data_type.complex64 import (
4243
COMPLEX64_DATA_TYPE,
@@ -338,6 +339,7 @@
338339
"ZarrPythonRawBytesDataTypeName",
339340
"ZarrPythonRawBytesFillValue",
340341
"base64_bytes",
342+
"base64_fill_value_rules",
341343
"hex_float16",
342344
"hex_float32",
343345
"hex_float64",

‎packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,23 @@ def base64_problems(value: str) -> Iterator[ValidationProblem]:
2828
)
2929

3030

31+
def base64_fill_value_rules(
32+
configuration: object, nested: object, value: object
33+
) -> Iterator[ValidationProblem]:
34+
"""Check that a fill value written as a string is standard-alphabet base64: a data type's `fill_value_rules`.
35+
36+
`Base64Bytes` is a `NewType` of `str`, so a fill value annotated with
37+
it accepts any string. Pass this rule as the data type's
38+
`fill_value_rules` to reject a string that is not base64. A value of
39+
any other shape is not checked here, so the rule also serves a fill
40+
value that may be written another way, such as
41+
`tuple[ByteValue, ...] | Base64Bytes`. A rule that checks more, such
42+
as the number of bytes, yields from this one first.
43+
"""
44+
if isinstance(value, str):
45+
yield from base64_problems(value)
46+
47+
3148
def base64_decoded(value: str) -> bytes | None:
3249
"""The bytes that `value` encodes in standard-alphabet base64; None when it is not such base64."""
3350
try:
@@ -41,4 +58,10 @@ def base64_spelling(data: bytes) -> str:
4158
return base64.b64encode(data).decode("ascii")
4259

4360

44-
__all__ = ["ByteValue", "base64_decoded", "base64_problems", "base64_spelling"]
61+
__all__ = [
62+
"ByteValue",
63+
"base64_decoded",
64+
"base64_fill_value_rules",
65+
"base64_problems",
66+
"base64_spelling",
67+
]

‎packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py‎

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,8 @@
55
"""
66

77
import base64
8-
from collections.abc import Iterator
98
from typing import Final, Literal
109

11-
from zarr_metadata._json import ValidationProblem
1210
from zarr_metadata._numpy import Base64Bytes, base64_bytes
1311
from zarr_metadata.definition._definition import (
1412
EmptyConfiguration,
@@ -18,7 +16,11 @@
1816
DataTypeDefinition,
1917
variable_length,
2018
)
21-
from zarr_metadata.v3.data_type._byte import ByteValue, base64_problems, base64_spelling
19+
from zarr_metadata.v3.data_type._byte import (
20+
ByteValue,
21+
base64_fill_value_rules,
22+
base64_spelling,
23+
)
2224

2325
BYTES_DATA_TYPE_NAME: Final = "bytes"
2426
"""The `data_type` value for the variable-length `bytes` type."""
@@ -34,14 +36,6 @@
3436
"""
3537

3638

37-
def _fill_value_rules(
38-
configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue
39-
) -> Iterator[ValidationProblem]:
40-
"""Check that a string fill value is standard-alphabet base64. A fill value written as byte values is not checked here."""
41-
if isinstance(value, str):
42-
yield from base64_problems(value)
43-
44-
4539
def _fill_value_canonical(
4640
configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue
4741
) -> str:
@@ -60,7 +54,7 @@ def _fill_value_canonical(
6054
name=BYTES_DATA_TYPE_NAME,
6155
configuration=EmptyConfiguration,
6256
fill_value=BytesFillValue,
63-
fill_value_rules=_fill_value_rules,
57+
fill_value_rules=base64_fill_value_rules,
6458
fill_value_canonical=_fill_value_canonical,
6559
storage=variable_length,
6660
)
@@ -79,4 +73,5 @@ def _fill_value_canonical(
7973
"BytesDataTypeName",
8074
"BytesFillValue",
8175
"base64_bytes",
76+
"base64_fill_value_rules",
8277
]

‎packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py‎

Lines changed: 2 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,12 @@
1010
"""
1111

1212
import base64
13-
from collections.abc import Iterator
1413
from typing import Final, Literal
1514

16-
from zarr_metadata._json import ValidationProblem
1715
from zarr_metadata._numpy import Base64Bytes
1816
from zarr_metadata.definition._definition import EmptyConfiguration, Nested
1917
from zarr_metadata.v3._definition import DataTypeDefinition, variable_length
20-
from zarr_metadata.v3.data_type._byte import base64_problems, base64_spelling
18+
from zarr_metadata.v3.data_type._byte import base64_fill_value_rules, base64_spelling
2119

2220
VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME: Final = "variable_length_bytes"
2321
"""The `data_type` value for the `variable_length_bytes` type."""
@@ -29,13 +27,6 @@
2927
"""Permitted JSON shape of the `fill_value` field for `variable_length_bytes`: a `Base64Bytes` string, which is what zarr-python writes and reads."""
3028

3129

32-
def _fill_value_rules(
33-
configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue
34-
) -> Iterator[ValidationProblem]:
35-
"""Check that the fill value is standard-alphabet base64."""
36-
yield from base64_problems(value)
37-
38-
3930
def _fill_value_canonical(
4031
configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue
4132
) -> str:
@@ -47,7 +38,7 @@ def _fill_value_canonical(
4738
name=VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME,
4839
configuration=EmptyConfiguration,
4940
fill_value=VariableLengthBytesFillValue,
50-
fill_value_rules=_fill_value_rules,
41+
fill_value_rules=base64_fill_value_rules,
5142
fill_value_canonical=_fill_value_canonical,
5243
storage=variable_length,
5344
)

‎packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
1-
"""Cover the `base64_bytes` brand validator.
1+
"""Cover the `base64_bytes` brand validator and the `base64_fill_value_rules` fill value rule.
22
33
The pydantic-driven fixture tests don't enforce the base64 alphabet or
44
length-multiple-of-4 constraint because `Base64Bytes` is a `NewType`,
55
which pydantic treats as plain `str`. Direct coverage of the validator
6-
function lives here.
6+
function and the rule built on it lives here.
77
"""
88

99
from __future__ import annotations
1010

1111
import pytest
1212

13-
from zarr_metadata.v3.data_type.bytes import base64_bytes
13+
from zarr_metadata.v3.data_type.bytes import base64_bytes, base64_fill_value_rules
1414

1515
VALID = [
1616
"", # empty is valid base64 (length 0, multiple of 4)
@@ -41,3 +41,16 @@ def test_valid(value: str) -> None:
4141
def test_invalid(value: str) -> None:
4242
with pytest.raises(ValueError, match="standard-alphabet base64"):
4343
base64_bytes(value)
44+
45+
46+
# A value written in another shape, such as byte values, is the annotation's to check.
47+
@pytest.mark.parametrize("value", [*VALID, (1, 2, 3), None])
48+
def test_base64_fill_value_rules_accepts(value: object) -> None:
49+
assert list(base64_fill_value_rules({}, (), value)) == []
50+
51+
52+
@pytest.mark.parametrize("value", INVALID)
53+
def test_error_base64_fill_value_rules_rejects_a_string_that_is_not_base64(value: str) -> None:
54+
assert [(p.loc, p.kind) for p in base64_fill_value_rules({}, (), value)] == [
55+
((), "invalid_value")
56+
]

0 commit comments

Comments
 (0)