Skip to content

Commit 34dd17c

Browse files
authored
fix(indexing): validate index-array bounds during JSON loading (#4347)
* docs(indexing): ground design and integration claims in current behavior Assisted-by: Codex:GPT-6 * docs(indexing): correct reader lazy-array and cache contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate wire boundaries and clarify format contracts Assisted-by: Codex:GPT-6 * fix(indexing): validate selector bounds and shared dependencies Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6 * docs(indexing): reconcile reader contracts and record audit fixes Assisted-by: Codex:GPT-6 * docs(indexing): reconcile audit with current partition implementation Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6 * docs(indexing): clarify planning coverage and benchmark measurement boundaries Assisted-by: Codex:GPT-6 * fix(indexing): group signed chunk coordinates without collisions Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6 * docs(indexing): state remaining planner limits precisely Assisted-by: Codex:GPT-6 * fix(indexing): reject unsupported wire index array bounds Assisted-by: Codex:GPT-6 * docs(indexing): clarify wire bounds rejection contract Assisted-by: Codex:GPT-6 * docs(indexing): number audit changelog entries for PR 4345 Assisted-by: Codex:GPT-6 * docs(indexing): describe current contracts in docstrings Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6 * fix(indexing): validate raw index values against wire bounds Accept valid finite and one-sided bounds with eager validation shared by both JSON loaders. Validate before affine adjustment and simplification; immutable validated maps need no retained constraint. Assisted-by: Codex:GPT-6
1 parent 1f13742 commit 34dd17c

8 files changed

Lines changed: 198 additions & 13 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Validate inclusive `index_array_bounds` against all raw index values when loading transforms and output maps from JSON. Accept valid finite and one-sided constraints, and reject out-of-bounds values eagerly before offset, stride, or map simplification. Validated immutable maps need not retain the constraints; message normalization preserves the original bounds.

‎packages/zarr-indexing/docs/ndsel.md‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,22 @@ assert t.to_json() == canonical
9999
map kind has a `to_json`; `output_index_map_from_json` dispatches the wire's
100100
structurally discriminated union back to the right kind. Exact JSON equality
101101
in this example is not a general round-trip guarantee: implicit flags are
102-
removed, finite `index_array_bounds` are not retained or enforced by the
103-
engine, and degenerate array maps are collapsed.
102+
removed and degenerate array maps are collapsed.
103+
104+
`index_array_bounds` constrains raw index-array values before the map's offset
105+
and stride are applied. Both `IndexTransform.from_json` and
106+
`output_index_map_from_json` validate every supplied value against the inclusive
107+
bounds when loading. Finite and one-sided bounds are supported; omitted bounds
108+
and `["-inf", "+inf"]` impose no additional constraint. Values outside the
109+
bounds raise `NdselError("invalid_json", ...)`, including in singleton arrays
110+
and zero-stride maps. Empty arrays satisfy any well-formed, ordered bounds.
111+
112+
Validation is eager: an invalid entry rejects the entire map even if a later
113+
selection would avoid that entry. After validation the engine owns immutable
114+
index coordinates, so it need not retain the bounds; serialization emits
115+
unbounded constraints for non-degenerate maps. Message normalization preserves
116+
the original bounds without checking array contents. This implementation does
117+
not defer bounds errors until individual positions are accessed.
104118

105119
A canonical body carrying a
106120
`"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a

‎packages/zarr-indexing/src/zarr_indexing/_wire.py‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
33
Package-private: the types that serialize themselves (`IndexDomain`,
44
`IndexTransform`, the output map kinds) all need these, so they cannot live in
5-
any one of them, and they are not API. The three engine constraints named in
6-
[`zarr_indexing.json`][zarr_indexing.json] — finite bounds, implicit bounds
7-
lowering by value, integer `index_array` content — are enforced here.
5+
any one of them, and they are not API. Domain bounds, index-array bounds,
6+
implicit bounds lowering by value, and integer `index_array` content are
7+
handled here, as described in [`zarr_indexing.json`][zarr_indexing.json].
88
"""
99

1010
from __future__ import annotations
@@ -13,13 +13,37 @@
1313

1414
import numpy as np
1515

16-
from zarr_indexing.messages import NdselError
16+
from zarr_indexing.messages import NdselError, validate_index_array_bounds
1717

1818
if TYPE_CHECKING:
1919
from zarr_indexing.domain import IndexDomain
2020
from zarr_indexing.json import BoundJSON
2121

2222

23+
def check_index_array_bounds(array: np.ndarray[Any, Any], bounds: Any, where: str) -> None:
24+
"""Validate every raw index value against an inclusive interval.
25+
26+
Validate interval syntax even for empty arrays. Once checked, immutable
27+
index coordinates need no retained constraint. Use Python integer extrema
28+
to avoid overflow or floating-point rounding at integer limits.
29+
"""
30+
lo, hi = validate_index_array_bounds(bounds, where)
31+
if array.size == 0 or (lo == "-inf" and hi == "+inf"):
32+
return
33+
minimum, maximum = int(array.min()), int(array.max())
34+
if (
35+
lo == "+inf"
36+
or hi == "-inf"
37+
or (isinstance(lo, int) and minimum < lo)
38+
or (isinstance(hi, int) and maximum > hi)
39+
):
40+
raise NdselError(
41+
"invalid_json",
42+
f"{where}.index_array values [{minimum}, {maximum}] are outside "
43+
f"index_array_bounds {bounds!r}",
44+
)
45+
46+
2347
def lower_bound(bound: BoundJSON, where: str) -> int:
2448
"""Lower a canonical bound to a finite integer, rejecting infinities.
2549

‎packages/zarr-indexing/src/zarr_indexing/json.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@
1414
1515
The engine lowering rules include:
1616
17+
- **Validated index arrays.** Raw index values must satisfy the inclusive
18+
`index_array_bounds` before offset and stride are applied. Lowering checks
19+
all supplied values eagerly; validated immutable maps need not retain bounds.
1720
- **Finite bounds.** An `IndexDomain` addresses a finite array, so a canonical
1821
body carrying a `"-inf"`/`"+inf"` bound cannot be lowered; `from_json` raises.
1922
- **Implicit bounds lower by value.** The `[n]`-bracket implicit/explicit flag
@@ -132,8 +135,9 @@ class OutputIndexMapJSON(TypedDict, total=False):
132135
index_array_bounds: list[IndexValueJSON]
133136
"""Wire bounds on index-array values; `["-inf", "+inf"]` if unconstrained.
134137
135-
The engine currently discards this field on load and does not enforce
136-
finite bounds against the array values. Serialization emits unconstrained bounds.
138+
The message layer preserves these inclusive constraints on raw index values.
139+
Engine lowering validates all values eagerly before offset and stride.
140+
Serialization emits unconstrained bounds for validated non-degenerate maps.
137141
"""
138142

139143

‎packages/zarr-indexing/src/zarr_indexing/messages.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -556,7 +556,7 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
556556
if has_index_array:
557557
stride = _check_int(raw["stride"], f"{where}.stride") if "stride" in raw else 1
558558
bounds = (
559-
_check_index_array_bounds(raw["index_array_bounds"], where)
559+
validate_index_array_bounds(raw["index_array_bounds"], where)
560560
if "index_array_bounds" in raw
561561
else ["-inf", "+inf"]
562562
)
@@ -585,7 +585,8 @@ def _normalize_output_map(raw: Any, where: str) -> dict[str, Any]:
585585
return {"offset": offset}
586586

587587

588-
def _check_index_array_bounds(value: Any, where: str) -> list[int | str]:
588+
def validate_index_array_bounds(value: Any, where: str) -> list[int | str]:
589+
"""Validate the syntax and ordering of an inclusive index-array interval."""
589590
if not isinstance(value, list) or len(value) != 2:
590591
raise NdselError(
591592
"invalid_json",

‎packages/zarr-indexing/src/zarr_indexing/output_map.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -357,18 +357,24 @@ def output_index_map_from_json(data: OutputIndexMapJSON) -> OutputIndexMap:
357357
selects an array map, `input_dimension` selects a dimension map, and
358358
neither selects a constant map.
359359
360+
Validate every raw index value against the inclusive `index_array_bounds`
361+
before applying offset and stride. Out-of-bounds values raise `NdselError`
362+
at load time. Validated immutable maps do not retain the bounds.
363+
360364
Examples
361365
--------
362366
>>> output_index_map_from_json({"offset": 5})
363367
ConstantMap(offset=5)
364368
>>> output_index_map_from_json({"offset": 0, "stride": 2, "input_dimension": 1})
365369
DimensionMap(input_dimension=1, offset=0, stride=2)
366370
"""
367-
from zarr_indexing._wire import lower_index_array
371+
from zarr_indexing._wire import check_index_array_bounds, lower_index_array
368372

369373
if "index_array" in data:
374+
array = lower_index_array(data["index_array"], "index_array")
375+
check_index_array_bounds(array, data.get("index_array_bounds", ["-inf", "+inf"]), "output")
370376
return ArrayMap(
371-
index_array=lower_index_array(data["index_array"], "index_array"),
377+
index_array=array,
372378
offset=data.get("offset", 0),
373379
stride=data.get("stride", 1),
374380
)

‎packages/zarr-indexing/src/zarr_indexing/transform.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -793,7 +793,10 @@ def from_json(cls, data: IndexTransformJSON) -> IndexTransform:
793793
that omitted fields — identity `output`, default bounds and labels —
794794
are filled and validated, then lowered to the engine representation.
795795
Lower-rank `index_array`s are widened to the full input rank on the way
796-
in.
796+
in. Every supplied raw index value is checked against the inclusive
797+
`index_array_bounds` before offset, stride, or map simplification.
798+
Out-of-bounds values raise `NdselError` immediately, even if a later
799+
selection would avoid them. Validated immutable maps do not retain bounds.
797800
798801
Examples
799802
--------
@@ -805,6 +808,7 @@ def from_json(cls, data: IndexTransformJSON) -> IndexTransform:
805808
True
806809
"""
807810
from zarr_indexing._wire import (
811+
check_index_array_bounds,
808812
full_rank_index_array,
809813
lower_bound,
810814
lower_index_array,
@@ -841,6 +845,7 @@ def from_json(cls, data: IndexTransformJSON) -> IndexTransform:
841845
if "index_array" in om:
842846
where = f"output[{i}]"
843847
arr = lower_index_array(om["index_array"], f"{where}.index_array")
848+
check_index_array_bounds(arr, om["index_array_bounds"], where)
844849
# ndsel leaves index-array rank unvalidated, so an external
845850
# producer may send an array of lower rank that broadcasts
846851
# against the domain. Widen it here, on the way in, so every
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
"""Index-array bounds are validated against raw coordinates during loading."""
2+
3+
from __future__ import annotations
4+
5+
from typing import TYPE_CHECKING, Any
6+
7+
import numpy as np
8+
import pytest
9+
10+
from zarr_indexing import ConstantMap, IndexTransform, normalize_ndsel
11+
from zarr_indexing.messages import NdselError
12+
from zarr_indexing.output_map import output_index_map_from_json
13+
14+
if TYPE_CHECKING:
15+
from zarr_indexing.json import IndexTransformJSON, OutputIndexMapJSON
16+
17+
18+
def load(values: Any, bounds: Any, entry: str, offset: int, stride: int) -> Any:
19+
output: OutputIndexMapJSON = {"index_array": values, "offset": offset, "stride": stride}
20+
if bounds is not None:
21+
output["index_array_bounds"] = bounds
22+
if entry == "output_map":
23+
return output_index_map_from_json(output)
24+
body: IndexTransformJSON = {"input_shape": list(np.asarray(values).shape), "output": [output]}
25+
return IndexTransform.from_json(body)
26+
27+
28+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
29+
@pytest.mark.parametrize(("offset", "stride"), [(0, 1), (10, -2), (7, 0)])
30+
@pytest.mark.parametrize(
31+
("values", "bounds"),
32+
[
33+
([-3, 0, 4], [-3, 4]),
34+
([-3, 0, 4], ["-inf", 4]),
35+
([-3, 0, 4], [-3, "+inf"]),
36+
([-3, 0, 4], ["-inf", "+inf"]),
37+
([-3, 0, 4], None),
38+
([2], [2, 2]),
39+
([], [2, 2]),
40+
([], ["+inf", "+inf"]),
41+
([], ["-inf", "-inf"]),
42+
([[0, 2], [2, 0]], [0, 2]),
43+
],
44+
)
45+
def test_valid_bounds_preserve_values_and_roundtrip(
46+
values: Any, bounds: Any, entry: str, offset: int, stride: int
47+
) -> None:
48+
model = np.asarray(values, dtype=np.intp)
49+
loaded = load(values, bounds, entry, offset, stride)
50+
expected = offset + stride * model
51+
if entry == "transform":
52+
points = np.array(list(np.ndindex(model.shape)), dtype=np.intp).reshape(-1, model.ndim)
53+
restored = IndexTransform.from_json(loaded.to_json())
54+
np.testing.assert_array_equal(loaded.apply_many(points).reshape(model.shape), expected)
55+
np.testing.assert_array_equal(restored.apply_many(points).reshape(model.shape), expected)
56+
# Further selection and affine adjustment operate on validated coordinates.
57+
if model.ndim == 1 and model.size > 1:
58+
selected = loaded.oindex[[model.size - 1, 0]].translate((3,))
59+
np.testing.assert_array_equal(
60+
selected.apply_many([[0], [1]]).ravel(), expected[[-1, 0]] + 3
61+
)
62+
else:
63+
restored = output_index_map_from_json(loaded.to_json())
64+
if isinstance(restored, ConstantMap):
65+
np.testing.assert_array_equal(np.full(model.shape, restored.offset), expected)
66+
else:
67+
assert restored == loaded
68+
np.testing.assert_array_equal(loaded.offset + loaded.stride * loaded.index_array, expected)
69+
if bounds is not None:
70+
canonical = normalize_ndsel(
71+
{
72+
"kind": "transform",
73+
"input_shape": list(model.shape),
74+
"output": [{"index_array": values, "index_array_bounds": bounds}],
75+
}
76+
)
77+
assert canonical["output"][0]["index_array_bounds"] == bounds
78+
79+
80+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
81+
@pytest.mark.parametrize(("offset", "stride"), [(0, 1), (10, -2), (7, 0)])
82+
@pytest.mark.parametrize(
83+
("values", "bounds"),
84+
[
85+
([-1, 0], [0, 2]),
86+
([0, 3], [0, 2]),
87+
([3], [2, 2]),
88+
([-1], [0, "+inf"]),
89+
([3], ["-inf", 2]),
90+
([0], ["+inf", "+inf"]),
91+
([0], ["-inf", "-inf"]),
92+
([[0, 1], [1, 3]], [0, 2]),
93+
],
94+
)
95+
def test_out_of_bounds_raw_values_are_rejected(
96+
values: Any, bounds: Any, entry: str, offset: int, stride: int
97+
) -> None:
98+
with pytest.raises(NdselError, match="index_array_bounds") as exc:
99+
load(values, bounds, entry, offset, stride)
100+
assert exc.value.reason == "invalid_json"
101+
102+
103+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
104+
@pytest.mark.parametrize("values", [[], [1]])
105+
@pytest.mark.parametrize("bounds", [[], [0], [0, 1, 2], [False, 2], [0.5, 2], ["bad", 2]])
106+
def test_malformed_bounds_are_rejected(values: Any, bounds: Any, entry: str) -> None:
107+
with pytest.raises(NdselError) as exc:
108+
load(values, bounds, entry, 0, 1)
109+
assert exc.value.reason == "invalid_json"
110+
111+
112+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
113+
@pytest.mark.parametrize("values", [[], [1]])
114+
def test_reversed_bounds_are_rejected(values: Any, entry: str) -> None:
115+
with pytest.raises(NdselError) as exc:
116+
load(values, [2, 0], entry, 0, 1)
117+
assert exc.value.reason == "bounds_out_of_order"
118+
119+
120+
@pytest.mark.parametrize("entry", ["transform", "output_map"])
121+
@pytest.mark.parametrize("value", [int(np.iinfo(np.intp).min), int(np.iinfo(np.intp).max)])
122+
def test_integer_extreme_bounds_are_exact(entry: str, value: int) -> None:
123+
result = load([value], [value, value], entry, 0, 1)
124+
if entry == "transform":
125+
assert result.apply((0,)) == (value,)
126+
else:
127+
assert int(result.index_array[0]) == value
128+
bounds = [value + 1, "+inf"] if value < 0 else ["-inf", value - 1]
129+
with pytest.raises(NdselError, match="index_array_bounds"):
130+
load([value], bounds, entry, 0, 1)

0 commit comments

Comments
 (0)