zarr-metadata 0.6.0: v2 dtype problem locations, the "struct" dtype string, and a doubled consolidated metadata problem #417

🤖 AI text below 🤖

zarr-metadata 0.6.0 reports some problems at locations the document does not have, reads one internal name as a valid v2 dtype, and reports one consolidated metadata problem twice. These were found by a differential run of the TypeScript port (zarr-metadata.js) against the 0.6.0 readers in their default scopes; every example below was checked against zarr-metadata==0.6.0.

Shared setup for the examples:

from zarr_metadata.model import (
    read_array_metadata_v2, read_array_metadata_v3,
    read_consolidated_metadata_v2, read_consolidated_metadata_v3,
)

v2 = {"zarr_format": 2, "shape": [4], "chunks": [2], "compressor": None,
      "fill_value": None, "order": "C", "filters": None}
G = lambda **kw: {"zarr_format": 3, "node_type": "group", **kw}
env = lambda md: {"kind": "inline", "must_understand": False, "metadata": md}

1. v2 field record problems are located under a fields key the document does not have

The v2 struct dtype definition reads a list of field records as the configuration {"fields": [...]} and locates its problems in that configuration. A problem in a record is therefore reported at ("dtype", "fields", <index>, ...), but the document has the records directly under dtype.

read_array_metadata_v2({**v2, "dtype": [["a"]]}).problems
# loc ('dtype', 'fields', 0)        expected ('dtype', 0)
read_array_metadata_v2({**v2, "dtype": [["a", [["b", 5]]]]}).problems
# loc ('dtype', 'fields', 0, 1, 'fields', 0, 1)        expected ('dtype', 0, 1, 0, 1)
read_consolidated_metadata_v2(
    {"zarr_consolidated_format": 1, "metadata": {"x/.zarray": {**v2, "dtype": [["a"]]}}}
).problems
# loc ('metadata', 'x/.zarray', 'dtype', 'fields', 0)

Expected: each problem is located in the document, without the fields segment. A tool that places a diagnostic at a problem's loc cannot find these locations. The zarr-metadata.js conformance runner currently removes the segment before it compares results.

2. The v2 dtype string "struct" is read as a struct dtype

typestr_problem in v2/_definition.py returns no problem for STRUCT_NAME, the name the struct definition is filed under. So a document that writes "struct" as its dtype is not refused as a non-typestr. It is read as a struct dtype with no configuration:

read_array_metadata_v2({**v2, "dtype": "struct"}).problems
# [(('dtype',), 'missing_key', "'struct' requires a configuration")]
read_array_metadata_v2({**v2, "dtype": [["a", "struct"]]}).problems
# [(('dtype', 'fields', 0, 1), 'missing_key', "'struct' requires a configuration")]
read_array_metadata_v2({**v2, "dtype": "strukt"}).problems
# [(('dtype',), 'invalid_value', 'expected a NumPy typestr -- ...')]

Expected: "struct" gets the same invalid_value problem as "strukt". The filing name is an implementation detail, not part of the v2 dtype grammar.

3. A listed group with its own problem is also reported as "reads otherwise"

The docstring of _nested_listing_problems in model/_group_v3.py says "a document with its own problem is reported for that problem". A listed group whose attributes are not an object gets a second problem anyway, because its members hold attributes=None, which _read_alike compares as "null" against the other listing's "{}". A group with a missing zarr_format gets only its own problem, so the two cases disagree.

read_consolidated_metadata_v3(
    env({"g": G(consolidated_metadata=env({"x": G(attributes=5)})), "g/x": G()})
).problems
# ('metadata', 'g', 'consolidated_metadata', 'metadata', 'x', 'attributes') invalid_type
# ('metadata', 'g', 'consolidated_metadata', 'metadata', 'x') invalid_value
#     "expected the document the group lists at "/g/x", got one that reads otherwise"
read_consolidated_metadata_v3(
    env({"g": G(consolidated_metadata=env({"x": {"node_type": "group"}})), "g/x": G()})
).problems
# ('metadata', 'g', 'consolidated_metadata', 'metadata', 'x', 'zarr_format') missing_key

The same happens when the outer listing's copy (g/x) is the one with the problem. Expected: the comparison is skipped when either document has a problem of its own, as the docstring says.

4. The v2 field record message names only the three-element form

A record may have two elements ([name, dtype]) or three ([name, dtype, shape]), and a two-element record is accepted. When a record has neither length, the message names only one form:

read_array_metadata_v2({**v2, "dtype": [["a", "<i4", [2], "x"]]}).problems
# 'expected an array of 3 elements, got ["a", "<i4", [2], "x"]'

Expected: the message names both forms.

5. Smaller inconsistencies

These may be intended. They are listed because the v2 and v3 readers disagree.

• A non-finite number in a member the spec does not define is accepted in a v3 document, but refused in a v2 .zarray:

  A = {"zarr_format": 3, "node_type": "array", "shape": [4], "data_type": "int8",
       "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2]}},
       "chunk_key_encoding": "default", "fill_value": 0, "codecs": ["bytes"]}
  read_array_metadata_v3({**A, "foo": float("nan")}).problems             # []
  read_array_metadata_v2({**v2, "dtype": "<i4", "foo": float("nan")}).problems
  # [(('foo',), 'invalid_value', 'non-finite float nan is not JSON')]

• A v3 consolidated key with an empty segment is refused, but the same path in a v2 .zmetadata key is accepted and normalized without a problem, and so is a key with a leading /:

  read_consolidated_metadata_v3(env({"a//b": G()})).problems
  # [(('metadata', 'a//b'), 'invalid_value', '... holds an empty name between two "/"')]
  read_consolidated_metadata_v2({"zarr_consolidated_format": 1, "metadata": {
      "a//.zgroup": {"zarr_format": 2}, "/b/.zgroup": {"zarr_format": 2}}}).problems
  # []

• An attributes member written inside a .zmetadata entry gets a different kind for arrays and groups: invalid_value in a .zarray entry, unknown_key in a .zgroup entry. Both messages say the attributes belong in .zattrs.