Skip to content

Commit f6d91e6

Browse files
d-v-bmkitticlaude
committed
refactor(indexing): make lexicographic_order_iter the lazy primitive
Per review from @mkitti: invert the relationship between the lazy iterator and the eagerly-collected tuple. `lexicographic_order_iter` is now a genuine lazy generator over the chunk-grid coordinates, and `_lexicographic_order_keys` collects it into a cached tuple — the eager version is "collect the lazy one", not the other way around. Previously lexicographic_order_iter returned iter() over the cached tuple, so any consumer that only needed a prefix still paid to materialize the entire grid. _is_total_shard does exactly that — an early-exit `all(coord in set for coord in ...)` — and on a cold cache for a 32^3 shard whose first coordinate is absent this dropped from ~15.8ms to ~24us (the lazy generator builds one coordinate and bails). The hot path is unchanged: the two dict.fromkeys sites consume the full grid and use the cached `_lexicographic_order_keys` tuple directly (~0.9ms at 32^3), so the regression fix this branch delivers is intact. This also resolves @chuckwondo's point — the iterator is now actually lazy rather than a thin wrapper over eager data. Co-authored-by: Mark Kittisopikul <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 500104f commit f6d91e6

1 file changed

Lines changed: 14 additions & 10 deletions

File tree

‎src/zarr/core/indexing.py‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1602,18 +1602,22 @@ def _lexicographic_order(chunk_shape: tuple[int, ...]) -> npt.NDArray[np.intp]:
16021602
return order
16031603

16041604

1605-
@lru_cache(maxsize=16)
1606-
def _lexicographic_order_keys(chunk_shape: tuple[int, ...]) -> tuple[tuple[int, ...], ...]:
1607-
return tuple(tuple(int(x) for x in row) for row in _lexicographic_order(chunk_shape))
1605+
def lexicographic_order_iter(chunk_shape: tuple[int, ...]) -> Iterator[tuple[int, ...]]:
1606+
# Lazily yield the chunk grid coordinates in lexicographic (row-major / C)
1607+
# order. This is the primitive: callers that only need a prefix (e.g. an
1608+
# early-exit `all(...)`) pay only for the coordinates they consume, instead
1609+
# of materializing the whole grid. Callers that need every coordinate
1610+
# repeatedly for the same shape should use `_lexicographic_order_keys`, which
1611+
# caches the fully collected tuple.
1612+
return (tuple(int(x) for x in row) for row in _lexicographic_order(chunk_shape))
16081613

16091614

1610-
def lexicographic_order_iter(chunk_shape: tuple[int, ...]) -> Iterator[tuple[int, ...]]:
1611-
# Iterate the chunk grid in lexicographic (row-major / C) order. The full
1612-
# coordinate tuple is built once and cached on `chunk_shape` (see
1613-
# _lexicographic_order_keys), so repeated calls for the same shape — e.g. one
1614-
# per shard write — reuse it rather than rebuilding tens of thousands of
1615-
# tuples each time.
1616-
return iter(_lexicographic_order_keys(chunk_shape))
1615+
@lru_cache(maxsize=16)
1616+
def _lexicographic_order_keys(chunk_shape: tuple[int, ...]) -> tuple[tuple[int, ...], ...]:
1617+
# Eagerly collect `lexicographic_order_iter` into a tuple, cached per shape so
1618+
# the per-shard chunk grid is built once and reused across writes (the hot
1619+
# path feeds this straight into `dict.fromkeys`).
1620+
return tuple(lexicographic_order_iter(chunk_shape))
16171621

16181622

16191623
def c_order_iter(chunks_per_shard: tuple[int, ...]) -> Iterator[tuple[int, ...]]:

0 commit comments

Comments
 (0)