Skip to content

Commit 92e0051

Browse files
committed
fix(indexing): validate raw index values against wire bounds
Accept valid finite and one-sided bounds with eager validation shared by both JSON loaders. Validate before affine adjustment and simplification; immutable validated maps need no retained constraint. Assisted-by: Codex:GPT-6
1 parent 79ec188 commit 92e0051

8 files changed

Lines changed: 162 additions & 82 deletions

File tree

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
Reject constrained `index_array_bounds` when lowering JSON to an engine transform or output map, instead of silently dropping the constraint. Message normalization continues to preserve bounds for consumers that support them.
1+
Validate inclusive `index_array_bounds` against all raw index values when loading transforms and output maps from JSON. Accept valid finite and one-sided constraints, and reject out-of-bounds values eagerly before offset, stride, or map simplification. Validated immutable maps need not retain the constraints; message normalization preserves the original bounds.

‎packages/zarr-indexing/docs/ndsel.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -102,14 +102,19 @@ in this example is not a general round-trip guarantee: implicit flags are
102102
removed and degenerate array maps are collapsed.
103103

104104
`index_array_bounds` constrains raw index-array values before the map's offset
105-
and stride are applied. The message layer preserves these bounds, but the
106-
engine cannot retain them through map operations. Both `IndexTransform.from_json`
107-
and `output_index_map_from_json` therefore raise `NdselError("invalid_json", ...)`
108-
when bounds differ from `["-inf", "+inf"]`. This includes one-sided constraints,
109-
empty and singleton arrays, and zero-stride maps. Omitted or explicitly unbounded
110-
bounds remain supported. Use the message layer to preserve constrained documents
111-
for consumers that support them; lowering never silently discards an index-array
112-
constraint.
105+
and stride are applied. Both `IndexTransform.from_json` and
106+
`output_index_map_from_json` validate every supplied value against the inclusive
107+
bounds when loading. Finite and one-sided bounds are supported; omitted bounds
108+
and `["-inf", "+inf"]` impose no additional constraint. Values outside the
109+
bounds raise `NdselError("invalid_json", ...)`, including in singleton arrays
110+
and zero-stride maps. Empty arrays satisfy any well-formed, ordered bounds.
111+
112+
Validation is eager: an invalid entry rejects the entire map even if a later
113+
selection would avoid that entry. After validation the engine owns immutable
114+
index coordinates, so it need not retain the bounds; serialization emits
115+
unbounded constraints for non-degenerate maps. Message normalization preserves
116+
the original bounds without checking array contents. This implementation does
117+
not defer bounds errors until individual positions are accessed.
113118

114119
A canonical body carrying a
115120
`"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a

‎packages/zarr-indexing/src/zarr_indexing/_wire.py‎

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
33
Package-private: the types that serialize themselves (`IndexDomain`,
44
`IndexTransform`, the output map kinds) all need these, so they cannot live in
5-
any one of them, and they are not API. The three engine constraints named in
6-
[`zarr_indexing.json`][zarr_indexing.json] — finite bounds, implicit bounds
7-
lowering by value, integer `index_array` content — are enforced here.
5+
any one of them, and they are not API. Domain bounds, index-array bounds,
6+
implicit bounds lowering by value, and integer `index_array` content are
7+
handled here, as described in [`zarr_indexing.json`][zarr_indexing.json].
88
"""
99

1010
from __future__ import annotations
@@ -13,24 +13,34 @@
1313

1414
import numpy as np
1515

16-
from zarr_indexing.messages import NdselError
16+
from zarr_indexing.messages import NdselError, validate_index_array_bounds
1717

1818
if TYPE_CHECKING:
1919
from zarr_indexing.domain import IndexDomain
2020
from zarr_indexing.json import BoundJSON
2121

2222

23-
def check_index_array_bounds(bounds: list[int | str], where: str) -> None:
24-
"""Reject constraints the engine cannot retain through map operations.
23+
def check_index_array_bounds(array: np.ndarray[Any, Any], bounds: Any, where: str) -> None:
24+
"""Validate every raw index value against an inclusive interval.
2525
26-
Bounds constrain raw index values before offset/stride are applied. Even
27-
currently valid values do not justify silently erasing a wire constraint.
26+
Validate interval syntax even for empty arrays. Once checked, immutable
27+
index coordinates need no retained constraint. Use Python integer extrema
28+
to avoid overflow or floating-point rounding at integer limits.
2829
"""
29-
if bounds != ["-inf", "+inf"]:
30+
lo, hi = validate_index_array_bounds(bounds, where)
31+
if array.size == 0 or (lo == "-inf" and hi == "+inf"):
32+
return
33+
minimum, maximum = int(array.min()), int(array.max())
34+
if (
35+
lo == "+inf"
36+
or hi == "-inf"
37+
or (isinstance(lo, int) and minimum < lo)
38+
or (isinstance(hi, int) and maximum > hi)
39+
):
3040
raise NdselError(
3141
"invalid_json",
32-
f"{where}.index_array_bounds {bounds!r} is unsupported by the engine; "
33-
'only omitted bounds or ["-inf", "+inf"] can be lowered',
42+
f"{where}.index_array values [{minimum}, {maximum}] are outside "
43+
f"index_array_bounds {bounds!r}",
3444
)
3545

3646

‎packages/zarr-indexing/src/zarr_indexing/json.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,9 @@
1414
1515
The engine lowering rules include:
1616
17-
- **Unconstrained index arrays.** Only omitted `index_array_bounds` or
18-
`["-inf", "+inf"]` can be lowered. Other constraints raise `NdselError`
19-
rather than being lost during serialization or map operations.
17+
- **Validated index arrays.** Raw index values must satisfy the inclusive
18+
`index_array_bounds` before offset and stride are applied. Lowering checks
19+
all supplied values eagerly; validated immutable maps need not retain bounds.
2020
- **Finite bounds.** An `IndexDomain` addresses a finite array, so a canonical
2121
body carrying a `"-inf"`/`"+inf"` bound cannot be lowered; `from_json` raises.
2222
- **Implicit bounds lower by value.** The `[n]`-bracket implicit/explicit flag
@@ -136,8 +136,8 @@ class OutputIndexMapJSON(TypedDict, total=False):
136136
"""Wire bounds on index-array values; `["-inf", "+inf"]` if unconstrained.
137137
138138
The message layer preserves these inclusive constraints on raw index values.
139-
Engine lowering rejects bounds other than `["-inf", "+inf"]`; serialization
140-
emits unconstrained bounds for non-degenerate array maps.
139+
Engine lowering validates all values eagerly before offset and stride.
140+
Serialization emits unconstrained bounds for validated non-degenerate maps.
141141
"""
142142

143143

‎packages/zarr-indexing/src/zarr_indexing/messages.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -556,7 +556,7 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
556556
if has_index_array:
557557
stride = _check_int(raw["stride"], f"{where}.stride") if "stride" in raw else 1
558558
bounds = (
559-
_check_index_array_bounds(raw["index_array_bounds"], where)
559+
validate_index_array_bounds(raw["index_array_bounds"], where)
560560
if "index_array_bounds" in raw
561561
else ["-inf", "+inf"]
562562
)
@@ -585,7 +585,8 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
585585
return {"offset": offset}
586586

587587

588-
def _check_index_array_bounds(value: Any, where: str) -> list[int | str]:
588+
def validate_index_array_bounds(value: Any, where: str) -> list[int | str]:
589+
"""Validate the syntax and ordering of an inclusive index-array interval."""
589590
if not isinstance(value, list) or len(value) != 2:
590591
raise NdselError(
591592
"invalid_json",

‎packages/zarr-indexing/src/zarr_indexing/output_map.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -357,8 +357,9 @@ def output_index_map_from_json(data: OutputIndexMapJSON) -> OutputIndexMap:
357357
selects an array map, `input_dimension` selects a dimension map, and
358358
neither selects a constant map.
359359
360-
Array maps with `index_array_bounds` other than `["-inf", "+inf"]` raise
361-
`NdselError`, because the engine cannot retain those constraints.
360+
Validate every raw index value against the inclusive `index_array_bounds`
361+
before applying offset and stride. Out-of-bounds values raise `NdselError`
362+
at load time. Validated immutable maps do not retain the bounds.
362363
363364
Examples
364365
--------
@@ -370,9 +371,10 @@ def output_index_map_from_json(data: OutputIndexMapJSON) -> OutputIndexMap:
370371
from zarr_indexing._wire import check_index_array_bounds, lower_index_array
371372

372373
if "index_array" in data:
373-
check_index_array_bounds(data.get("index_array_bounds", ["-inf", "+inf"]), "output")
374+
array = lower_index_array(data["index_array"], "index_array")
375+
check_index_array_bounds(array, data.get("index_array_bounds", ["-inf", "+inf"]), "output")
374376
return ArrayMap(
375-
index_array=lower_index_array(data["index_array"], "index_array"),
377+
index_array=array,
376378
offset=data.get("offset", 0),
377379
stride=data.get("stride", 1),
378380
)

‎packages/zarr-indexing/src/zarr_indexing/transform.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -793,8 +793,10 @@ def from_json(cls, data: IndexTransformJSON) -> IndexTransform:
793793
that omitted fields — identity `output`, default bounds and labels —
794794
are filled and validated, then lowered to the engine representation.
795795
Lower-rank `index_array`s are widened to the full input rank on the way
796-
in. Constrained `index_array_bounds` raise `NdselError`: only omitted
797-
bounds or `["-inf", "+inf"]` are supported by the engine.
796+
in. Every supplied raw index value is checked against the inclusive
797+
`index_array_bounds` before offset, stride, or map simplification.
798+
Out-of-bounds values raise `NdselError` immediately, even if a later
799+
selection would avoid them. Validated immutable maps do not retain bounds.
798800
799801
Examples
800802
--------
@@ -842,8 +844,8 @@ def from_json(cls, data: IndexTransformJSON) -> IndexTransform:
842844
for i, om in enumerate(body["output"]):
843845
if "index_array" in om:
844846
where = f"output[{i}]"
845-
check_index_array_bounds(om["index_array_bounds"], where)
846847
arr = lower_index_array(om["index_array"], f"{where}.index_array")
848+
check_index_array_bounds(arr, om["index_array_bounds"], where)
847849
# ndsel leaves index-array rank unvalidated, so an external
848850
# producer may send an array of lower rank that broadcasts
849851
# against the domain. Widen it here, on the way in, so every
Lines changed: 108 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,130 @@
1-
"""Engine lowering must never silently discard index-array constraints."""
1+
"""Index-array bounds are validated against raw coordinates during loading."""
22

33
from __future__ import annotations
44

5-
from typing import TYPE_CHECKING
5+
from typing import TYPE_CHECKING, Any
66

77
import numpy as np
88
import pytest
99

10-
from zarr_indexing import IndexTransform, normalize_ndsel
10+
from zarr_indexing import ConstantMap, IndexTransform, normalize_ndsel
1111
from zarr_indexing.messages import NdselError
1212
from zarr_indexing.output_map import output_index_map_from_json
1313

1414
if TYPE_CHECKING:
1515
from zarr_indexing.json import IndexTransformJSON, OutputIndexMapJSON
1616

1717

18-
@pytest.mark.parametrize("bounds", [[0, 3], ["-inf", 3], [0, "+inf"], [0, 0]])
19-
@pytest.mark.parametrize("values", [[], [1], [-1, 1, 3]])
20-
@pytest.mark.parametrize(("offset", "stride"), [(0, 1), (10, -2), (7, 0)])
18+
def load(values: Any, bounds: Any, entry: str, offset: int, stride: int) -> Any:
19+
output: OutputIndexMapJSON = {"index_array": values, "offset": offset, "stride": stride}
20+
if bounds is not None:
21+
output["index_array_bounds"] = bounds
22+
if entry == "output_map":
23+
return output_index_map_from_json(output)
24+
body: IndexTransformJSON = {"input_shape": list(np.asarray(values).shape), "output": [output]}
25+
return IndexTransform.from_json(body)
26+
27+
2128
@pytest.mark.parametrize("entry", ["transform", "output_map"])
22-
def test_lowering_rejects_unsupported_index_array_bounds(
23-
bounds: list[int | str], values: list[int], offset: int, stride: int, entry: str
29+
@pytest.mark.parametrize(("offset", "stride"), [(0, 1), (10, -2), (7, 0)])
30+
@pytest.mark.parametrize(
31+
("values", "bounds"),
32+
[
33+
([-3, 0, 4], [-3, 4]),
34+
([-3, 0, 4], ["-inf", 4]),
35+
([-3, 0, 4], [-3, "+inf"]),
36+
([-3, 0, 4], ["-inf", "+inf"]),
37+
([-3, 0, 4], None),
38+
([2], [2, 2]),
39+
([], [2, 2]),
40+
([], ["+inf", "+inf"]),
41+
([], ["-inf", "-inf"]),
42+
([[0, 2], [2, 0]], [0, 2]),
43+
],
44+
)
45+
def test_valid_bounds_preserve_values_and_roundtrip(
46+
values: Any, bounds: Any, entry: str, offset: int, stride: int
2447
) -> None:
25-
output: OutputIndexMapJSON = {
26-
"index_array": values,
27-
"index_array_bounds": bounds,
28-
"offset": offset,
29-
"stride": stride,
30-
}
31-
body: IndexTransformJSON = {
32-
"input_inclusive_min": [0],
33-
"input_shape": [len(values)],
34-
"output": [output],
35-
}
36-
# JSON-only processing preserves constraints for capable consumers.
37-
canonical = normalize_ndsel({"kind": "transform", **body})
38-
assert canonical["output"][0]["index_array_bounds"] == bounds
48+
model = np.asarray(values, dtype=np.intp)
49+
loaded = load(values, bounds, entry, offset, stride)
50+
expected = offset + stride * model
3951
if entry == "transform":
40-
with pytest.raises(NdselError, match="index_array_bounds.*unsupported") as exc:
41-
IndexTransform.from_json(canonical)
42-
assert exc.value.reason == "invalid_json"
52+
points = np.array(list(np.ndindex(model.shape)), dtype=np.intp).reshape(-1, model.ndim)
53+
restored = IndexTransform.from_json(loaded.to_json())
54+
np.testing.assert_array_equal(loaded.apply_many(points).reshape(model.shape), expected)
55+
np.testing.assert_array_equal(restored.apply_many(points).reshape(model.shape), expected)
56+
# Further selection and affine adjustment operate on validated coordinates.
57+
if model.ndim == 1 and model.size > 1:
58+
selected = loaded.oindex[[model.size - 1, 0]].translate((3,))
59+
np.testing.assert_array_equal(
60+
selected.apply_many([[0], [1]]).ravel(), expected[[-1, 0]] + 3
61+
)
4362
else:
44-
with pytest.raises(NdselError, match="index_array_bounds.*unsupported") as exc:
45-
output_index_map_from_json(output)
46-
assert exc.value.reason == "invalid_json"
63+
restored = output_index_map_from_json(loaded.to_json())
64+
if isinstance(restored, ConstantMap):
65+
np.testing.assert_array_equal(np.full(model.shape, restored.offset), expected)
66+
else:
67+
assert restored == loaded
68+
np.testing.assert_array_equal(loaded.offset + loaded.stride * loaded.index_array, expected)
69+
if bounds is not None:
70+
canonical = normalize_ndsel(
71+
{
72+
"kind": "transform",
73+
"input_shape": list(model.shape),
74+
"output": [{"index_array": values, "index_array_bounds": bounds}],
75+
}
76+
)
77+
assert canonical["output"][0]["index_array_bounds"] == bounds
4778

4879

49-
@pytest.mark.parametrize("explicit_bounds", [False, True])
50-
@pytest.mark.parametrize("values", [[], [1], [-1, 1, 3]])
80+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
5181
@pytest.mark.parametrize(("offset", "stride"), [(0, 1), (10, -2), (7, 0)])
52-
def test_unconstrained_index_array_roundtrip_evaluation(
53-
explicit_bounds: bool, values: list[int], offset: int, stride: int
82+
@pytest.mark.parametrize(
83+
("values", "bounds"),
84+
[
85+
([-1, 0], [0, 2]),
86+
([0, 3], [0, 2]),
87+
([3], [2, 2]),
88+
([-1], [0, "+inf"]),
89+
([3], ["-inf", 2]),
90+
([0], ["+inf", "+inf"]),
91+
([0], ["-inf", "-inf"]),
92+
([[0, 1], [1, 3]], [0, 2]),
93+
],
94+
)
95+
def test_out_of_bounds_raw_values_are_rejected(
96+
values: Any, bounds: Any, entry: str, offset: int, stride: int
5497
) -> None:
55-
output: OutputIndexMapJSON = {"index_array": values, "offset": offset, "stride": stride}
56-
if explicit_bounds:
57-
output["index_array_bounds"] = ["-inf", "+inf"]
58-
body: IndexTransformJSON = {
59-
"input_inclusive_min": [0],
60-
"input_shape": [len(values)],
61-
"output": [output],
62-
}
63-
transform = IndexTransform.from_json(body)
64-
restored = IndexTransform.from_json(transform.to_json())
65-
points = np.arange(len(values)).reshape(-1, 1)
66-
expected = np.array([offset + stride * value for value in values]).reshape(-1, 1)
67-
np.testing.assert_array_equal(transform.apply_many(points), expected)
68-
np.testing.assert_array_equal(restored.apply_many(points), expected)
69-
# The standalone decoder follows the same omission/default policy.
70-
assert output_index_map_from_json(output) == transform.output[0]
98+
with pytest.raises(NdselError, match="index_array_bounds") as exc:
99+
load(values, bounds, entry, offset, stride)
100+
assert exc.value.reason == "invalid_json"
101+
102+
103+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
104+
@pytest.mark.parametrize("values", [[], [1]])
105+
@pytest.mark.parametrize("bounds", [[], [0], [0, 1, 2], [False, 2], [0.5, 2], ["bad", 2]])
106+
def test_malformed_bounds_are_rejected(values: Any, bounds: Any, entry: str) -> None:
107+
with pytest.raises(NdselError) as exc:
108+
load(values, bounds, entry, 0, 1)
109+
assert exc.value.reason == "invalid_json"
110+
111+
112+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
113+
@pytest.mark.parametrize("values", [[], [1]])
114+
def test_reversed_bounds_are_rejected(values: Any, entry: str) -> None:
115+
with pytest.raises(NdselError) as exc:
116+
load(values, [2, 0], entry, 0, 1)
117+
assert exc.value.reason == "bounds_out_of_order"
118+
119+
120+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
121+
@pytest.mark.parametrize("value", [int(np.iinfo(np.intp).min), int(np.iinfo(np.intp).max)])
122+
def test_integer_extreme_bounds_are_exact(entry: str, value: int) -> None:
123+
result = load([value], [value, value], entry, 0, 1)
124+
if entry == "transform":
125+
assert result.apply((0,)) == (value,)
126+
else:
127+
assert int(result.index_array[0]) == value
128+
bounds = [value + 1, "+inf"] if value < 0 else ["-inf", value - 1]
129+
with pytest.raises(NdselError, match="index_array_bounds"):
130+
load([value], bounds, entry, 0, 1)

0 commit comments

Comments
 (0)