Sync gpu-codecs with main, rename the gpu extra to cuda, fix nvCOMP Zstd sync and options - #4

…nd add 10m timeout to individual tests (zarr-developers#4140)

* chore(deps): bump the actions group across 1 directory with 8 updates (zarr-developers#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* test: bind moto server to an ephemeral port to fix xdist deadlock

The session-scoped moto_server fixture bound a fixed port (5555). Under
pytest-xdist, session-scoped fixtures run once per worker, so concurrent
workers raced to bind the same port. The losers blocked forever inside
ThreadedMotoServer.start(): its werkzeug thread dies on "Address already
in use" before setting the ready event that start() waits on, and the
wait has no timeout. Any multi-worker run whose s3 tests landed on 2+
workers deadlocked near the end of the suite at 0% CPU. CI never saw
this because it runs the suite single-process.

Bind port 0 instead and yield the actual endpoint from the fixture.
Since the endpoint is now only known at runtime, test_fsspec.py's
module-level endpoint_url constant becomes an endpoint_url fixture
(replacing the s3_base alias), and its consumers take it as a
parameter.

With this fix the full suite completes in ~1 minute under -n auto on a
10-core machine, where it previously hung indefinitely.

Assisted-by: ClaudeCode:claude-fable-5

* doc: add changelog fragment for moto ephemeral-port fix

Assisted-by: ClaudeCode:claude-fable-5

* test: enable pytest strict mode and fix duplicate parametrization ids

Adopt pytest 9's `strict = true` config option, which enables
strict_config, strict_markers, strict_xfail, strict_parametrization_ids,
and future strictness options. This replaces the --strict-config and
--strict-markers addopts flags (which pytest silently ignored when
passed via addopts before 9.1, pytest#14442) and the separate
xfail_strict setting. minversion is bumped to 9 accordingly.

strict_parametrization_ids surfaced seven parametrizations whose ids
collided, which pytest previously deduplicated silently:

- test_common.py paired float("inf") with np.inf (and -inf/nan), but
  np.inf IS a Python float, so three cases per test ran twice and the
  intended numpy-scalar coverage did not exist. Use np.float64 scalars
  with explicit ids, restoring the intended coverage.
- test_int.py deliberately includes np.dtype("i")/np.dtype("I"),
  which compare equal to one of the explicit-endian dtypes and
  stringify identically. The class-fixture hook in test_dtype/conftest
  now suffixes colliding ids with their index instead of relying on
  pytest's silent deduplication.
- test_group.py test_consistent_signatures id'd both create_hierarchy
  pairs identically; ids now include module and qualname.
- test_array.py ("a" vs b"a"), test_vlen.py (str vs "str"), and
  test_metadata/test_v2.py (3 vs "3") get explicit ids for the
  colliding member.

Assisted-by: ClaudeCode:claude-fable-5

* doc: add changelog fragment for pytest strict mode

Assisted-by: ClaudeCode:claude-fable-5

* test: arm faulthandler watchdog to turn deadlocks into loud failures

Set faulthandler_timeout=600 with faulthandler_exit_on_timeout (new in
pytest 9): if a single test exceeds 10 minutes, every thread's
traceback is dumped and the run is killed, instead of hanging
indefinitely with no diagnostics. Motivated by the moto fixed-port
deadlock, which hung local -n auto runs at ~98% for hours and required
manual kill -ABRT forensics to diagnose; with this watchdog the same
bug would have failed in 10 minutes with the offending stack in the
log. The ceiling is ~30x the slowest legitimate test (~20s locally) to
stay clear of slow CI runners, coverage overhead, and nightly stateful
hypothesis runs.

Assisted-by: ClaudeCode:claude-fable-5

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#3885)

* feat: define `PreparedWrite` and `SupportsChunkPacking` data structures

`PreparedWrite` models a set of per-chunk changes that would be applied to a stored chunk. `SupportsChunkPacking`
is a protocol for array -> bytes codecs that can use `PreparedWrite` objects to update an existing chunk.

* feat: new codec pipeline that uses sync path

* feat: complete second codecpipeline

* fix: handle rectilinear chunks

* fixup

* feat: SupportsSetRange protocol + sync byte-range writes

Adds a SupportsSetRange protocol to zarr.abc.store for stores that
allow overwriting a byte range within an existing value. Implementations
are added for LocalStore (using file-handle seek+write) and MemoryStore
(in-memory bytearray slice assignment).

This is the prerequisite for the partial-shard write fast path in
ShardingCodec, which can patch individual inner-chunk slots without
rewriting the entire shard blob when the inner codec chain is fixed-size.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: add sync codec methods to V2 and numcodecs codecs

V2Codec, BytesCodec, BloscCodec, etc. previously only implemented the
async _decode_single / _encode_single methods. Add their sync
counterparts (_decode_sync / _encode_sync) so that the upcoming
SyncCodecPipeline can dispatch through them without spinning up an
event loop.

For codecs that wrap external compressors (numcodecs.Zstd, numcodecs.Blosc,
the V2 fallback chain), the sync versions just call the underlying
compressor's blocking API directly instead of routing through
asyncio.to_thread.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: SyncCodecPipeline — synchronous codec pipeline with per-chunk parallelism

Adds SyncCodecPipeline alongside BatchedCodecPipeline. The new pipeline
runs codecs through their sync entry points (_decode_sync / _encode_sync)
and dispatches per-chunk work to a module-level thread pool sized by
the codec_pipeline.max_workers config (default = os.cpu_count()).

Each chunk's full lifecycle (fetch + decode + scatter for reads;
get-existing + merge + encode + set/delete for writes) runs as one
pool task — overlapping IO of one chunk with compute of another.
Scatter into the shared output buffer is thread-safe because chunks
have non-overlapping output selections.

The async wrappers (read/write) detect SupportsGetSync/SupportsSetSync
stores and dispatch to the sync fast path, passing the configured
max_workers. Other stores fall through to the async path, which still
uses asyncio.concurrent_map at async.concurrency.

Notes on perf:
- Default (None → cpu_count) is tuned for chunks ≥ ~512 KB.
- Small chunks (≤ 64 KB) regress 1.5-3x because pool dispatch overhead
  (~30-50 µs/task) dominates per-chunk work. Workaround:
  zarr.config.set({"codec_pipeline.max_workers": 1}).
- For large chunks on local/memory stores, IO+compute parallelism
  yields 1.7-2.5x over BatchedCodecPipeline on direct-API reads and
  ~2.5x on roundtrip.

ChunkTransform encapsulates the sync codec chain. It caches resolved
ArraySpecs across calls with the same chunk_spec — combined with the
constant-ArraySpec optimization in indexing, hot-path overhead is
minimized.

Includes test scaffolding for the new pipeline (test_sync_codec_pipeline)
and config plumbing for the max_workers key.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: partial-shard write support in ShardingCodec

Adds _encode_partial_sync and _decode_partial_sync to ShardingCodec.
For fixed-size inner codec chains and stores that implement
SupportsSetRange, partial writes patch individual inner-chunk slots
in-place instead of rewriting the whole shard:

  - Reads existing shard index (one byte-range get).
  - For each affected inner chunk: decodes the slot, merges the new
    region, re-encodes.
  - Writes each modified slot at its deterministic byte offset, then
    rewrites just the index.

For variable-size inner codecs (e.g. with compression) or stores that
don't support byte-range writes, falls through to a full-shard rewrite
matching BatchedCodecPipeline semantics.

The partial-decode path computes a ReadPlan from the shard index and
issues one byte-range get per overlapping chunk, decoding only what
the read selection touches.

Both paths are dispatched from SyncCodecPipeline via the existing
supports_partial_decode / supports_partial_encode protocol checks.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test: codec invariants + pipeline parity matrix

Two new test files:

  test_codec_invariants — asserts contract-level properties that every
  codec / shard / buffer combination must satisfy: round-trip exactness,
  prototype propagation, fill-value handling, all-empty shard handling.

  test_pipeline_parity — exhaustive matrix asserting that
  SyncCodecPipeline and BatchedCodecPipeline produce semantically
  identical results across codec configs, layouts (including
  nested sharding), write sequences, and write_empty_chunks settings.
  Three checks per cell:
    1. Same array contents on read.
    2. Same set of store keys after writes.
    3. Each pipeline reads the other's output identically (catches
       layout-divergence bugs).

These tests pinned the design throughout the SyncCodecPipeline +
partial-shard development.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: gitignore local agent/planning notes

Adds .gitignore entries for .claude/, CLAUDE.md, and docs/superpowers/
so local IDE/agent planning artifacts don't get committed by accident.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: remove unused PreparedWrite and SupportsChunkCodec

Both were exported from zarr.abc.codec.__all__ but never referenced
by either codec pipeline or any test. Artifacts of an earlier design
iteration superseded by the current SyncCodecPipeline.

Also remove now-unused imports of `dataclass` and `ChunkProjection`
that were only needed by the deleted symbols.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: remove stale phased-pipeline test files

Both tests/test_phased_codec_pipeline.py and tests/test_pipeline_benchmark.py
import PhasedCodecPipeline, which no longer exists in src/. Each failed at
collection. The benchmarking intent of test_pipeline_benchmark.py is replaced
by extensions to tests/benchmarks/test_e2e.py later in this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: rename SyncCodecPipeline to FusedCodecPipeline

The new name describes what the pipeline does (fuses fetch+decode+scatter
into one task per chunk) rather than the implementation detail of using
sync codec entry points. The name also stays accurate when this pipeline
gains a remote-store / async fast path in a future change.

Mechanical rename across the class, register_pipeline call, dotted-path
strings used by zarr.config, isinstance checks, parametrize values, and
docstrings. tests/test_sync_pipeline.py renamed to tests/test_fused_pipeline.py.

Nothing on this branch is released, so no deprecation alias is needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: lift _merge_chunk_array to module level

The BatchedCodecPipeline and FusedCodecPipeline classes had identical
copies of _merge_chunk_array (one method, one staticmethod). Extract
once as a module-level free function and call from both. No new base
class or mixin is introduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: extract _async_read_fallback to module level

Both BatchedCodecPipeline.read_batch (non-partial-decode branch) and
FusedCodecPipeline.read (async fallback) duplicate the same sequence:
concurrent_map(get) -> pipeline.decode -> scatter into out. Lift to a
module-level free function and call from both.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: extract _async_write_fallback to module level

Both BatchedCodecPipeline.write_batch (non-partial-encode branch) and
FusedCodecPipeline.write (async fallback) duplicate the same sequence:
read existing bytes -> decode -> merge -> encode -> set/delete. Lift to
a module-level free function and call from both.

After this change, neither pipeline class carries _merge_chunk_array,
nor the duplicated read/write fallback bodies. Each class is reduced to
its constructor, fast-path methods, and thin async dispatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(bench): parametrize test_e2e over both codec pipelines

Adds a `pipeline` fixture with values ["batched", "fused"] that swaps
codec_pipeline.path for the duration of each benchmark. Both
test_write_array and test_read_array now produce one benchmark cell
per (compression x layout x store x pipeline). CodSpeed will report
comparable numbers for both pipelines on the same workloads.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(bench): parametrize test_e2e over a synthetic latency dimension

Adds `latency in {0, 0.001, 0.05, 0.2}` and a bench_store fixture that
wraps the underlying memory store in zarr.testing.store.LatencyStore
when latency > 0. Local-store cells skip nonzero latency — adding
synthetic latency on top of a real filesystem double-counts and is not
the intended measurement.

Combined with the pipeline parameter, the matrix now produces
comparable benchmark numbers for {Batched, Fused} x {0, 1ms, 50ms, 200ms}
on memory-shaped operation. The numbers are signal under one simple
model of remote latency, not absolute predictions of S3 behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: restore deleted comments in V2Codec._decode_sync

Commit 7f45aba9 (which converted _decode_single -> _decode_sync) dropped
two explanatory comment blocks from the dtype-handling branches in
V2Codec.decode. Both comments document non-obvious WHY:

- The TypeError catch is for chunks whose stored dtype doesn't match
  the array spec dtype (e.g. string dtype vs object array).
- The elif branch fires when filters were tampered with: an object array
  needs an object codec in the filter chain to be read correctly.

These were removed as drive-by cleanup during the sync-method rename
without intent to delete the substance. Restoring verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: explain non-obvious behaviors in sharding sync methods

Add docstring substance and a couple of inline notes to the new sync
methods on ShardingCodec that landed on this branch. Concretely:

- _decode_sync / _encode_sync: explain how each relates to the async
  counterpart and the partial-* variants, and why inner chunks are
  iterated in Morton order on the encode path.
- _encode_shard_dict_sync: explain the two-pass offset shift in the
  index-at-start branch (offsets are written relative to the data
  section, then bumped by len(index_bytes)) and the MAX_UINT_64
  empty-chunk sentinel that must not be touched.
- _encode_partial_sync byte-range path: explain WHY morton-rank
  determines byte offset deterministically (fixed-size inner chunks =
  every slot at a stable offset regardless of which others are present);
  this is the load-bearing invariant for the byte-range fast path.
- _decode_partial_sync: docstring now lists the two sub-paths
  (full-shard fetch vs. index-then-byte-ranges) and the reason the
  full-shard branch exists (one round trip beats N+1 small ones).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: convert RST inline literals to Markdown-style backticks

Docstrings added on this branch used RST-style ``literal`` markup
(double-backticks). Convert to Markdown-style `literal` (single
backticks) so the docstrings render correctly in Markdown-aware
viewers without needing a separate RST-to-Markdown step.

Two cases worth calling out:

- src/zarr/core/codec_pipeline.py and src/zarr/codecs/sharding.py:
  every ``literal`` in these files came in on this branch, so the
  conversion is global within those files.
- src/zarr/abc/store.py and src/zarr/core/array.py: only docstrings
  added on this branch are converted; pre-existing RST-style
  literals from main are left alone (out of scope).

Also converted one .. note:: directive in
src/zarr/core/array.py (the regular_chunk_array_spec helper) to
a Markdown blockquote, since that directive was added on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* perf: memoize encoded inner chunk for scalar complete-shard writes (#177)

In ShardingCodec._encode_partial_sync's full-shard-rewrite loop, a scalar
broadcast value produces byte-for-byte identical results for every complete
inner chunk (same fill, same empty-check, same encoded bytes). Compute that
outcome once and reuse it across all complete chunks instead of re-merging,
re-checking write_empty_chunks, and re-encoding tens of thousands of identical
chunks. Incomplete edge chunks still merge against their own data individually.

Target case (fused, memory, chunks=100/shards=1M, no compression):
write 92.26ms -> 21.59ms (4.3x). Pipeline parity (byte-identical to batched)
and 956 tests pass under the fused pipeline; adversarial partial-overwrite/
edge/compression/2D/aliasing checks pass.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* perf+fix: bulk whole-shard read + repair _ShardIndex construction post-merge

Two things, both scoped to the sync sharding read path:

1. Fix: main's #3975 made _ShardIndex a 2-field NamedTuple (chunks_per_shard,
   offsets_and_lengths), but the Fused sync methods still constructed it with one
   arg, erroring on every Fused sharded read. Pass chunks_per_shard through in
   _decode_shard_index_sync and the byte-range write path.

2. Perf: _decode_full_shard_bulk + _ShardIndex.is_dense. A whole-shard read of a
   dense, fixed-size, uncompressed shard is reconstructed by reshaping/scattering
   the data section in bulk, replacing the per-chunk decode/index/projection loop
   (~78% of a full read). Chunk positions are read from the stored index, so it is
   correct for any subchunk_write_order. Falls through to the per-chunk path for
   compression/filters, non-dense shards, and any read whose output shape != the
   shard shape (strided/partial/fancy).

Full read (memory, 10000 chunks/shard, uint8): ~291ms -> ~21ms (13.9x vs Batched).
Verified: 0 new test failures vs the merge baseline; full reads correct across
dtypes and 2D; partial/strided/gzip fall through. (Pre-existing Fused x
subchunk_write_order gaps remain, tracked separately.)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: FusedCodecPipeline honors subchunk_write_order + coalesced partial reads

Three integration gaps surfaced when the Fused pipeline met main's new
subchunk_write_order (#3826), partial-read coalescing (#3004), and _ShardIndex
refactor. Under Fused these caused 25 sharding/parity failures (data was
correct in the partial-read cases; the failures were write-order layout +
IO-pattern divergence). Fixes:

1. Write order: _encode_shard_dict_sync laid out chunks in hardcoded morton
   order, ignoring subchunk_write_order. Now iterates
   _subchunk_order_iter(self.subchunk_write_order), matching the async
   _encode_shard_dict. Fixes lexicographic/colexicographic/unordered storage.

2. Coalesced sync partial reads: add Store.get_ranges_sync (a synchronous,
   coalescing counterpart of get_ranges, reusing coalesce_ranges) and
   ShardingCodec._load_partial_shard_maybe_sync; route _decode_partial_sync's
   partial branch through it. Sync stores now get #3004's byte-range coalescing
   without an event loop (fewer, merged reads).

3. Non-sync fallback: FusedCodecPipeline.read now routes non-sync stores (e.g.
   ZipStore) through the async partial-decode path when the AB codec supports
   it, instead of _async_read_fallback's whole-shard get(). Matches Batched's
   IO behavior; avoids over-reading whole shards on partial reads.

Tests: the #3004 partial-read tests are made pipeline-aware (assert the active
method family: get/get_ranges vs get_sync/get_ranges_sync, gated on store sync
support). 573 sharding+parity+pipeline+indexing and 657 codec tests pass under
BOTH pipelines (was 25 failing under Fused).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: address roborev review (job 222) — Fused sharding correctness

HIGH (sharding.py, byte-range write fast path): derived each chunk's physical
slot from self.subchunk_write_order instead of hardcoded morton order, and
excluded 'unordered' (no recoverable rank -> falls through to the index-driven
full-rewrite path). A partial write into a dense shard first written with a
non-default order no longer corrupts data via wrong byte offsets.

HIGH (sharding.py, _decode_full_shard_bulk): build the read-view dtype from the
BytesCodec's endian (as BytesCodec._decode_sync does), not the dtype's native
endianness. A big-endian shard read on a little-endian host (or vice versa) now
decodes correctly instead of silently reinterpreting bytes.

MEDIUM (sharding.py, _decode_full_shard_bulk): the bulk fast path now requires
the inner chain to be exactly one BytesCodec, excluding crc-bearing shards. The
bulk path can't verify per-chunk checksums, so crc shards fall through to the
per-chunk path and keep their corruption detection.

LOW (codec_pipeline.py, ChunkTransform._resolve_specs): key the resolved-spec
cache on the frozen, hashable ArraySpec value instead of (shape, id()), which
could collide after id reuse.

LOW (codec_pipeline.py, _get_pool): don't shutdown(wait=False) the old pool on
grow — a concurrent in-flight pool.map could hit 'cannot schedule new futures
after shutdown'. The orphaned pool drains and is GC'd.

Tests: extended test_pipeline_parity with big-endian + crc32c codec configs and
a dedicated subchunk_write_order x index_location parity test (asserts identical
contents always, identical bytes for deterministic orders). Verified each new
test fails when its corresponding fix is reverted. 1219 tests pass under both
pipelines; mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs: correct FusedCodecPipeline framing — sync scheduling, not IO/compute separation

The class docstring claimed it 'separates IO from compute', then immediately
said the ShardingCodec does IO internally — self-contradictory and misleading.
The actual win is replacing per-chunk ASYNC scheduling with synchronous,
batched/coalesced execution; the sharding codec still owns its storage IO
(the zarrs model, unlike tensorstore's storage-free codecs). Rewrite the
docstring to state this plainly and note that a storage-free codec is a
possible future direction, not what this pipeline does. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor: stop hard-coding assumptions about the 'unordered' write order

After merging #4011 (which made 'unordered' deterministic and warns callers not
to rely on its layout), drop the two places my earlier fixes special-cased it by
name:

- Byte-range write fast path: remove the 'subchunk_write_order != unordered'
  gate. The rank map is derived from _subchunk_order_iter(self.subchunk_write_
  order), which is the single source of truth for physical layout — correct for
  every order without a name check. _subchunk_order_iter is the only place that
  knows a given order's layout.
- Parity test: assert byte-equality across pipelines for ALL orders, not just
  'deterministic' ones. The check verifies the two pipelines AGREE (they share
  _subchunk_order_iter), which holds whatever an order resolves to; it makes no
  assumption about what 'unordered' means.

540 parity+sharding and 862 codec/indexing tests pass under both pipelines; mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat: make FusedCodecPipeline the default codec pipeline

Flip codec_pipeline.path default from BatchedCodecPipeline to FusedCodecPipeline.
Fused runs codec compute synchronously/in bulk and gives large speedups on
sharded workloads (up to ~24x write / ~14x read on many-chunks-per-shard, more
with compression) and no regressions on compute-bound cases; it falls back to
the async path for non-sync stores. Batched remains selectable via config.

Test fallout from the flip (all behavior, not stale-assertion churn):
- test_config_defaults_set: expected default path updated.
- test_config_codec_implementation: the mock codec now also overrides
  _encode_sync, so it records a call regardless of which pipeline is default
  (Fused uses the sync entry point).
- StoreExpectingTestBuffer (zarr.testing.buffer): added set_sync/get_sync that
  mirror the async buffer-type guards, so the 'all buffers are TestBuffer'
  invariant is checked on the sync write path too. Verified Fused correctly
  threads a custom BufferPrototype (sharded writes store TestBuffer instances) —
  the test simply wasn't exercising the sync path before.

Full suite: 6346 passed, 0 failed under the new default.

NOTE: changelog fragment filename is a PLACEHOLDER — rename
changes/PLACEHOLDER-fused-default.feature.md to changes/<PR#>.feature.md once the
PR number is known (towncrier keys fragments by issue/PR number).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: ShardingCodec inner pipeline follows the configured default, not hard-coded Batched

The codec_pipeline property hard-coded BatchedCodecPipeline.from_codecs(). main
resolves it against the registry via get_pipeline_class() (#2179); the branch
carried an older hard-coded version and the main-merge kept the branch side.
With FusedCodecPipeline now the default this left the inner sub-chunk pipeline
stuck on Batched while the outer array used Fused — an inconsistency, and stale
relative to main. Restore get_pipeline_class().from_codecs(), matching the rest
of this module (which already uses get_pipeline_class elsewhere).

Verified: sharding + parity + pipeline (596) and codecs+array+indexing+properties
(2161) pass; nested sharding roundtrips correctly under both pipelines; no
functional BatchedCodecPipeline references remain in sharding.py. mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: Fused async decode/encode must evolve codec specs (HIGH-2) + shared CodecPipelineTests

HIGH-2: FusedCodecPipeline.decode()/encode() (the async fallback for non-sync
stores) reused one flat chunk_spec across every codec stage instead of evolving
it per codec via resolve_metadata. Spec-changing array->array codecs broke:
TransposeCodec crashed on read (could not broadcast (2,2) into (2,4));
cast_value/scale_offset would silently corrupt. Reachable on the DEFAULT
pipeline for every non-sync store (S3/GCS/fsspec/zip).

Fix, without re-duplicating spec logic (the duplication caused the bug):
- Extract resolve_aa_specs(): single source of truth for per-stage spec
  evolution (forward-thread resolve_metadata over the AA codecs). Pure metadata.
- Add AsyncChunkTransform: per-chunk ASYNC mirror of ChunkTransform, driving the
  codecs' async _decode_single/_encode_single with the correct per-stage spec.
  No mini-batch concept (that stays a BatchedCodecPipeline concern).
- ChunkTransform._resolve_specs delegates to resolve_aa_specs.
- Fused.decode()/encode() loop per chunk through AsyncChunkTransform.

Also harden the sharding byte-range WRITE fast path: take chunk offsets from the
stored shard index, not from the live subchunk_write_order (which is not
recoverable on reopen by design).

New tests/test_codec_pipeline_suite.py: xUnit CodecPipelineTests base run as
TestBatchedPipeline and TestFusedPipeline over a sync (MemoryStore) AND a
non-sync (LatencyStore) store axis. Reproduces HIGH-2 automatically. 140 pass;
mypy clean; original ZipStore+transpose crash now roundtrips.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: dedupe codec-pipeline tests against the shared CodecPipelineTests suite

The shared suite runs every pipeline-agnostic behavior test against BOTH
pipelines x both store paths, so per-file copies of the same behavior are
redundant. Remove confirmed duplicates; keep tests that exercise something the
suite does not.

- Strengthen the suite's write_empty_chunks tests to also assert chunk-key
  presence/absence (absorbing the old _no_store / _persists coverage).
- test_codec_pipeline.py: drop the 8 behavior duplicates now in the suite. KEEP
  test_read_returns_get_results (low-level pipeline.read GetResult API),
  test_write_empty_chunks_false_no_store (store-key shape), and
  test_codec_pipeline_threads_dtype_through_evolve (#3937 regression).
- test_fused_pipeline.py: drop the array-level streaming read/write tests and
  test_partial_shard_write_roundtrip_correctness (array behavior, suite-covered).
  KEEP all pipeline-API / Fused-internal tests (construction, evolve, low-level
  write/read(_sync) roundtrips, sync-write/async-read interop, ChunkTransform
  encode/decode, set_range, inner_codecs_fixed_size, byte-range fast path).

740 pass across suite + codec_pipeline + fused + sync + invariants + parity +
sharding; ruff + mypy clean. No coverage removed without a verified equivalent.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: unify the create/write/read suite tests into one Scenario-parametrized test

The bulk of CodecPipelineTests followed one shape: create an array, apply some
writes, optionally assert which chunk keys exist, then assert reads come back
correct. Capture those variables in a frozen Scenario dataclass (array_kwargs,
writes, reads, keys_present/absent) and drive them all through a single
parametrized test_scenario. Correctness is checked against a numpy reference the
scenario derives from its own writes, so cases don't hand-maintain expected
values. 18 scenarios cover the same matrix (layouts, gzip, transpose
spec-evolution, nested sharding, partial-shard overwrite, write_empty key
presence/absence) x both pipelines x sync/async stores.

Kept as separate focused tests the two cases that don't fit the shape:
test_read_missing_chunks_false_raises (asserts an exception) and
test_partial_write_after_reopen_is_correct (has an extra reopen step).

Verified the parametrized form keeps its regression-guard value: reverting the
HIGH-2 spec-evolution fix still fails test_scenario[async-transpose]. 670 pass
across pipeline + sharding suites; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: prune test_fused_pipeline.py to its irreducible Fused-specific core

The Fused test file had accumulated tests that either duplicated the
pipeline-agnostic CodecPipelineTests suite or were misfiled. Triage:

- async roundtrip / missing-chunk-fill / partial-shard-write dups: removed;
  the shared test_scenario covers these across both pipelines x sync/async
  stores. Added float32 and zstd Scenarios first so the dtype/codec coverage
  the dups carried transfers to the shared matrix (no net coverage loss).
- store set_range / SupportsSetRange tests: already covered (more thoroughly,
  parametrized) in tests/test_store/test_memory.py; removed as dups.
- ShardingCodec._inner_codecs_fixed_size tests: moved to
  tests/test_codecs/test_sharding_unit.py where the sharding internals live.

What stays is genuinely Fused-only and cannot be pipeline-agnostic: the
synchronous API (write_sync / read_sync / _sync_transform) which Batched has
no equivalent of, and the byte-range fast-path assertions (set_range_sync
fires / falls back) which test a Fused-only optimization.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: dissolve test_codec_invariants.py, redistributing by subject

The "invariants" file grouped tests by their shared motivation (a design
doc) rather than by what they test, which is the wrong axis -- it mixed
pipeline-agnostic behavior, Fused-only internals, and a per-codec property
into one file. Sorted each test into the home its subject implies:

Pipeline-agnostic behavior -> CodecPipelineTests (runs on BOTH pipelines x
sync/async stores via the existing fixtures):
- S2 empty-chunk skipping under default config -> a Scenario (keys_absent).
- S2 shard deleted after overwrite-to-fill -> a base-class method (it needs
  a mid-sequence key assertion the Scenario shape can't express).
- C3 no isinstance(ShardingCodec) branching in read/write -> a base-class
  method that resolves the subclass's configured pipeline and source-scans it.

Fused-only (byte-range fast path / ChunkTransform internals) ->
test_fused_pipeline.py:
- S3 fast path skipped when write_empty_chunks=False (the unique complement
  of the existing uses-set-range test; the write_empty_chunks=True case was a
  dup and is dropped).
- B1 byte-range path copies read-only LocalStore buffers before mutating.
- C2 ChunkTransform passes each codec the runtime chunk_spec prototype.

Per-codec contract -> tests/test_codecs/test_codecs.py:
- C1 resolve_metadata only mutates shape (prototype/dtype/fill_value/config
  stable across the chain) -- a property of individual codecs, no pipeline.

Dropped as a pure duplicate (already in test_store/test_memory.py):
- test_supports_set_range_is_runtime_checkable.

No coverage lost: every kept test moved, and the two genuinely-shared
behaviors now run on both pipelines instead of only whichever was default.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: drop redundant read-parity matrix, move partial-read coverage to shared suite

test_pipeline_read_parity checked Fused vs Batched partial reads against
*each other*. The shared CodecPipelineTests suite already reads partial/strided
selections from sharded arrays against a numpy reference on BOTH pipelines --
which is strictly stronger (it would catch both pipelines diverging from the
spec in the same way, which a pipeline-vs-pipeline check cannot).

The one sliver read-parity covered that the shared suite didn't was scalar
single-element reads from a sharded array (the sharding codec's partial-decode
path). Added two Scenarios (sharded-scalar-reads-1d / -2d) to capture it.
Verified they exercise the partial-decode path on both pipelines: the default
Fused pipeline routes a scalar sharded read through _decode_partial_sync, the
Batched pipeline through _decode_partial_single -- so both variants are now
checked against numpy, not just against each other.

Kept in test_pipeline_parity.py the two checks the per-pipeline suite cannot
express, because its two subclasses run in isolation and never see each other's
output:
- test_pipeline_parity: cross-read interop (write under A, read whole under B)
  + cross-pipeline store-key-set equality.
- test_pipeline_parity_subchunk_write_order: byte-identical shard output across
  pipelines for every subchunk_write_order x index_location.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: rename test_sync_codec_pipeline -> test_chunk_transform; drop cross-file dup

The file named test_sync_codec_pipeline.py tested no pipeline -- it is the unit
test suite for ChunkTransform (the per-chunk synchronous codec chain that
FusedCodecPipeline uses internally). "sync codec pipeline" was an earlier name
for the Fused pipeline; the filename had outlived it. Renamed to
test_chunk_transform.py (git mv preserves history) and added a module docstring
naming what it actually covers.

Also removed test_sync_transform_encode_decode_roundtrip from
test_fused_pipeline.py: it was a weaker cross-file duplicate of this file's
test_encode_decode_roundtrip (which covers the same encode->decode->compare over
five codec chains rather than just bytes-only). Its one extra assertion -- that
evolve_from_array_spec populates _sync_transform -- is already covered by
test_evolve_from_array_spec in the Fused file.

test_codec_pipeline.py left as-is: all three tests are correctly placed and
cover things the Scenario suite can't (the low-level pipeline.read GetResult
API, a plain dict store, and the #3937 cast_value dtype-threading regression).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat: remove byte-range-write support pending store-interface decision

The byte-range-write machinery works, but the right store interface for it is
still undecided, so it is removed from this PR and will return once that lands.

Removed:
- SupportsSetRange protocol (abc/store.py) and its __all__ export.
- MemoryStore.set_range / set_range_sync / _set_range_impl and the
  SupportsSetRange base (storage/_memory.py).
- LocalStore.set_range / set_range_sync, the _put_range helper, and the
  SupportsSetRange base (storage/_local.py).
- The sharding codec's byte-range-write fast path in _encode_partial_sync;
  partial shard writes now always take the full-shard-rewrite path (identical
  to BatchedCodecPipeline, verified by the pipeline-parity suite). Also dropped
  the now-dead _chunk_byte_offset helper it relied on.
- changes/3907.feature.md (the byte-range-writes changelog note). The
  byte-range-READ changelog (3004) is unrelated and kept.

Byte-range READS (ByteRequest, get(byte_range=), get_ranges coalescing,
the read-side bulk shard decode) are untouched -- this only removes writes.

The known-good tests that exercise byte-range writes are commented out (not
deleted) in test_store/test_memory.py, test_store/test_local.py, and
test_fused_pipeline.py, to restore once the store design is settled.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: remove dead _get_default_chunk_spec helper

PR-added module-level helper in array.py with zero callers — an ArraySpec-reuse
optimization that was never wired up. Plain function, no protocol role, safe to
drop. Verified: no references anywhere in src/ or tests/, and the full
array/sharding/pipeline suites stay green.

Note: ShardingCodec._encode_sync, though never *called*, is NOT dead — it is a
required member of the runtime_checkable SupportsSyncCodec protocol. Removing it
drops ShardingCodec from SupportsSyncCodec and breaks the sync read-fallback
routing (16 test failures), so it stays.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: correct ShardingCodec._encode_sync docstring re: write order

The docstring claimed _encode_sync "iterates inner chunks in Morton order —
that's the canonical layout the shard index expects", which is wrong and a
latent footgun: it implies the method imposes a morton physical layout. It does
not. The morton iteration only populates an intermediate dict whose key order is
immaterial; the on-disk layout is decided downstream by the subchunk_write_order
loop in _encode_shard_dict_sync (same as the async _encode_single sibling).

Also clarified that this method IS reached — via nested sharding, where an inner
ShardingCodec is encoded through the outer codec's ChunkTransform. (It is not
called for top-level sharded writes, which route through _encode_partial_sync.)

Verified empirically: routing through nested _encode_sync, all three
subchunk_write_order values roundtrip correctly AND morton vs lexicographic
produce physically different bytes — i.e. the order is honored, not ignored.
Behavior unchanged; docstring only.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: remove unused ShardingCodec._load_shard_index wrapper

PR-added thin wrapper (`_load_shard_index_maybe(...) or _ShardIndex.create_empty(...)`)
with zero invocations anywhere in src/ or tests/. Unlike _encode_sync, this is
genuinely removable: confirmed it is NOT a member of any runtime_checkable
protocol or ABC (no reference in src/zarr/abc/, not a base-class override) and is
reached by no dynamic dispatch (no getattr / string reference). main has no
_load_shard_index* methods at all, so it was introduced and left unused by this
PR. The _maybe and _maybe_sync variants it wrapped remain and are used.

Verified: full sharding + nested-sharding + parity + pipeline suites stay green,
ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: drop stale set_range_sync mention from FusedCodecPipeline docstring

The FusedCodecPipeline class docstring still described sharded writes as using
"byte-range writes via set_range_sync" — but byte-range-write support was removed
from this PR (set_range_sync / SupportsSetRange are gone). Sharded writes now take
the codec's synchronous full-shard-rewrite path. Docstring only; no behavior change.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: use plain single backticks in docstrings, not RST double-backticks

This branch's docstrings/comments had introduced RST-style ``double-backtick``
inline literals, which this project does not use (plain single backticks only —
no RST roles or double-backticks). Converted the 25 occurrences across the
sharding codec, codec_pipeline, and fsspec store docstrings/comments to single
backticks. Style only; no behavior change.

Also confirmed (via git blame, this-branch lines only) there are no remaining
references to removed/outdated designs: the byte-range-write (set_range) mentions
and the "separating IO from compute" framing were already corrected earlier in
this branch.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat: default codec_pipeline.max_workers to 1 (sequential), threading opt-in

Pairs with the FusedCodecPipeline default: keep the new pipeline, but do NOT
enable threading by default. `max_workers=None` (auto -> cpu_count) spawned a
thread pool on every read/write, which is a behavior change with real downstream
risk — it runs custom stores/codecs concurrently (thread-safety) and can
oversubscribe many-core nodes whose workloads already parallelize at the
dask/MPI layer. The default is now 1 (fully sequential: the pool is never
created when max_workers <= 1). Parallelism is opt-in via
`codec_pipeline.max_workers` (positive int, or None for auto).

Updates _resolve_max_workers docstring and the config-defaults test accordingly.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* perf: vectorize shard_dict build in _encode_partial_sync (fix write regression)

CodSpeed flagged test_sharded_morton_write_single_chunk regressing ~38-39%
(writing one 1x1x1 chunk into a 32^3 = 32768-chunk shard). Both main and this
branch do a full shard rewrite for a partial write, so the rewrite itself is not
the regression — and it is NOT the removed byte-range fast path (that path was
gated out here anyway: write_empty_chunks defaults to False -> skip_empty=True).

The cause: the sync _encode_partial_sync rebuilt the in-memory shard_dict with a
per-coordinate __getitem__ loop over all 32768 chunks (O(n_chunks) Python
overhead + try/except per chunk), whereas main's async _encode_partial_single
builds the same dict with a single vectorized index lookup via
_ShardReader.to_dict_vectorized. Switched the sync path to to_dict_vectorized
(a plain, non-async method; _shard_reader_from_bytes_sync already returns a
_ShardReader), matching the async path.

The dict's key order is immaterial (the physical layout is decided downstream by
the subchunk_write_order loop in _encode_shard_dict_sync), so the merge loop —
which looks up by coordinate, not order — is unaffected.

Local micro-benchmark (32^3 shard, single 1x1x1 chunk write): 59.4 -> 40.0
ms/write (~1.5x), matching the CodSpeed delta. Correctness: full sharding +
pipeline-parity suites pass (581), so Fused still matches Batched byte-for-byte.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix: thread spec forward in FusedCodecPipeline.evolve_from_array_spec

The deps=optional CI job (where cast_value_rs is installed) failed
test_codec_pipeline_threads_dtype_through_evolve and several test_cast_value
tests with "Invalid endianness: None" / "endian needs to be specified for
multi-byte data types".

Root cause: FusedCodecPipeline.evolve_from_array_spec evolved EVERY codec against
the same original array_spec:

    evolved = tuple(c.evolve_from_array_spec(array_spec=array_spec) for c in self.codecs)

When an array->array codec widens the dtype (e.g. cast_value int8 -> int16), the
BytesCodec serializer was still evolved against the single-byte SOURCE dtype, so
it stripped its `endian` to None (bytes.py treats single-byte dtypes as having no
endianness) and then failed at decode time on the multi-byte data.
BatchedCodecPipeline.evolve_from_array_spec already threads the spec forward
(spec = evolved_codec.resolve_metadata(spec)); the Fused version did not. Fixed by
mirroring the Batched threading.

Also added test_evolve_threads_spec_preserving_serializer_endian: a
dependency-free regression test (uses a minimal dtype-widening AA codec stub, no
cast_value_rs) that runs on BOTH pipelines via the pipeline_class fixture. It
fails on [sync] without this fix and passes with it — closing the gap where the
only coverage required an optional dep and thus ran in no default env.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: extract shared pipeline logic into freestanding functions

The endian bug fixed in the previous commit existed because the two codec
pipelines duplicated the same conceptual logic and one copy drifted: Batched's
evolve_from_array_spec threaded the spec forward correctly, Fused's did not.
Duplicated logic that can silently diverge is a standing bug source, so extract
the drift-prone pieces into single sources of truth that both pipelines delegate
to (mirroring the existing resolve_aa_specs precedent):

- evolve_codecs(codecs, array_spec): the construction-time spec-threading loop.
  Both BatchedCodecPipeline and FusedCodecPipeline.evolve_from_array_spec now
  call it. There is now exactly one place this logic lives, so it cannot drift.
- pipeline_supports_partial_decode / _encode(ab, *, aa, bb, require_no_aa_bb):
  the partial-decode/encode predicate. Both pipelines delegate. The two pass
  DIFFERENT require_no_aa_bb values (Batched True, Fused False) — that divergence
  is pre-existing and deliberately preserved here (not silently unified); it is
  now explicit at the call sites and documented in one function instead of being
  buried in two slightly-different inline isinstance checks.

Behavior-preserving: each pipeline computes exactly what it did before. Left the
trivial fan-out loops (validate, compute_encoded_size) as-is — deduping those
would require changing the CodecPipeline ABC contract (abstract -> concrete) for
near-zero drift benefit. Full pipeline/sharding/parity suites + the dtype-evolve
regression test pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: cover the max_workers>1 thread-pool path and concurrent decode

The pool dispatch in read_sync/write_sync (codec_pipeline.max_workers > 1) had
zero functional test coverage — only config-default assertions existed — even
though threading is the opt-in we point users at. Adds:

- an end-to-end multi-chunk read/write roundtrip with max_workers=4 (verified
  the pool dispatch actually fires, not the sequential branch);
- worker-exception propagation tests for both write_sync (list-consumed
  pool.map) and read_sync (tuple-consumed pool.map): a store error raised in a
  pool worker must surface to the caller;
- a concurrent-decode test: transpose filter (so ChunkTransform._resolve_specs
  cache traffic actually occurs — with no AA codecs the cache is bypassed),
  pool workers decoding concurrently, plus an outer thread pool issuing
  overlapping reads. Pins correctness under concurrency around the shared
  transform's mutable cache.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: add zarr v2 scenarios to the shared codec-pipeline suite

zarr_format=2 appeared in none of the pipeline test files: v2 arrays were only
exercised implicitly through whichever pipeline is the global default. v2 goes
through the V2Codec wrapper (numcodecs filters + compressor) — a different
codec path than the v3 AA/AB/BB chain, with its own _encode_sync/_decode_sync
under FusedCodecPipeline — so it deserves explicit coverage on BOTH pipelines
and BOTH store kinds (sync fast path + async fallback).

Adds v2-roundtrip (uncompressed) and v2-gzip-roundtrip (numcodecs.GZip — the v2
compressor spelling; v3 codec configs are rejected for v2 arrays) to SCENARIOS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* perf: encode the shard index once, via a shared layout helper

Both _encode_shard_dict_sync and the async _encode_shard_dict encoded the shard
index TWICE when index_location=start: encode to learn the length, shift the
present chunks' offsets by it, then re-encode with corrected offsets. The index
size is knowable without encoding — _shard_index_size() is already the byte-
exact contract every index read path relies on (reads slice exactly that many
bytes) — so the layout can use absolute offsets from the start and the index is
encoded once. Saves a full index encode (including its crc32c over the offsets
array) per shard write with index_location=start.

The layout loop was also duplicated between the sync and async versions — the
same drift surface that produced the evolve_from_array_spec endian bug. Both now
delegate to a shared pure _build_shard_layout (offset math lives once) and
_assemble_shard. A runtime guard verifies the encoded index length matches
_shard_index_size rather than silently corrupting offsets if someone ever
configures variable-size index codecs.

Verified: 606 tests pass including the pipeline-parity byte-identical shard
assertions across index_location=start/end and every subchunk_write_order, and
the sharded reopen tests — the on-disk layout is unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: pin read_missing_chunks=False semantics on sharded arrays

read_missing_chunks exists to help consumers distinguish a transport error from
a truly missing chunk. That distinction is a STORE-KEY-level concept: a missing
shard key raises ChunkNotFoundError. It does not cleanly apply to inner
subchunks of a shard that was fetched successfully — there is no transport
ambiguity there; the shard index simply records the subchunk as absent — so
those fill with the fill value rather than raising.

Both pipelines already implement exactly this (verified empirically), but
nothing pinned it, so the asymmetry vs unsharded arrays read as a bug in review.
This adds a shared-suite test (both pipelines x sync/async stores) asserting
both sides: missing shard key raises; missing inner subchunk of an existing
shard fills.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* Ig/fused additions (#180)

* chore: add parallelism TODOs

* perf: non-sharded reads

* chore: code duplication

* fix: arguments dont spread themselves!

* fix: bring in suggested guard

* perf: don't block on pool ops

* chore: docstring + materialize early

---------

Co-authored-by: ilan-gold <[email protected]>

* perf: sync IO for sharding byte getters; evolve the nested inner pipeline

Addresses the open question on this PR about sync/async byte getters,
benchmark-guided as discussed.

_ShardingByteGetter/_ShardingByteSetter are in-memory dict wrappers but
presented only an async API, so the nested codec_pipeline.read over inner
chunks fell to the async fallback: one concurrent_map coroutine per inner chunk
for a dict lookup (~2.1 us/chunk pure asyncio overhead, ~8.8 ms per 4096-chunk
shard). On top of that, the nested pipeline (ShardingCodec.codec_pipeline) is
built by from_codecs and never evolved, so its sync transform was always None —
inner chunks also paid per-chunk AsyncChunkTransform coroutines, and the
decode_sync/encode_sync fallback improvements in this PR could not reach them.

Changes:
- SyncByteGetter / SyncByteSetter runtime protocols in zarr.abc.store
  (resurrecting the design from the original perf/prepared-write experiments,
  same names and shape). StorePath matches structurally but is still gated on
  its STORE's sync support; the protocols gate non-StorePath byte getters.
- _ShardingByteGetter/Setter implement get_sync/set_sync/delete_sync; the async
  methods delegate to the sync ones (single implementation).
- ShardingCodec._get_inner_pipeline(shard_spec): the nested pipeline evolved
  against the inner chunk spec (threads specs through the inner chain AND
  builds the sync transform). The four nested read/write call sites use it.
- FusedCodecPipeline.read/write gates accept non-StorePath SyncByteGetter/
  SyncByteSetter, so nested inner-chunk IO takes read_sync/write_sync.
- _decode_shard_index/_encode_shard_index delegate to their sync twins (pure
  compute; kills a per-shard AsyncChunkTransform round-trip and a sync/async
  duplication).

Benchmark (4096 inner chunks per shard, LatencyStore@0 i.e. the async-fallback
path = sharded data on remote stores), vs this PR's head:

  uncompressed read: 44.2 -> 28.9 ms (1.53x)
  gzip         read: 182.0 -> 50.9 ms (3.6x)
  writes: unchanged (~34 / ~70 ms) — already optimized by this PR's
  encode_sync fallback (whole-shard sync encode, no byte setters involved).

Adds a regression test asserting sharded fallback reads route inner chunks
through the sync fast path (read_sync engaged, zero AsyncChunkTransform calls);
verified it fails if the gate is removed. Full sharding + parity + pipeline
suites pass (619).

Assisted-by: ClaudeCode:claude-fable-5

* fix: root-cause the cross-file test flake (pytest-asyncio loop leak); cache inner pipeline

The pipeline test suites have failed intermittently all along on an arbitrary
test that passes in isolation. Root cause (allocation site verified with
PYTHONTRACEMALLOC): pytest-asyncio implicitly creates an event loop in
_get_event_loop_no_warn during fixture setup/teardown and never closes it. When
GC reclaims that loop — or its self-pipe socketpair — mid-test, pytest's
unraisable hook converts the ResourceWarning into a failure of whichever
unrelated test happens to be running. The sync-bytegetter change increased
per-shard-op allocation churn enough to make this near-deterministic, which is
how it was finally traced.

Two changes:
- pyproject filterwarnings: narrowly ignore the two unraisable shapes
  (BaseEventLoop.__del__, AF_UNIX socketpair), mirroring the existing
  s3fs/aiobotocore entry. Not zarr's loops.
- ShardingCodec._get_inner_pipeline is now memoized per (pipeline class,
  shard_spec) — evolving built a ChunkTransform on every shard operation. The
  pipeline class is part of the key so codec_pipeline.path config changes are
  still honored.

Battery that previously failed ~every run now passes 3x consecutively (635).

Assisted-by: ClaudeCode:claude-fable-5

* refactor: dedupe sharding sync/async mirrors; fix un-threaded inner-chain evolve

Continues the anti-skew work: where the sync and async sharding paths
implemented the same logic twice, extract a single source of truth so the
copies cannot drift (the mechanism behind the pipeline-level endian bug).

- _get_inner_chunk_transform / _get_index_chunk_transform now evolve their
  codec chains via evolve_codecs (spec THREADED forward). Both previously
  evolved every codec against the same unthreaded spec — the exact bug shape
  that stripped BytesCodec.endian at the pipeline level, latent here for any
  spec-changing inner codec. Both are also now actually memoized (the inner
  transform's docstring claimed a cache that did not exist; transforms were
  rebuilt per call).
- New regression test (dependency-free dtype-widening stub codec) asserting
  the inner serializer keeps its endian; verified it fails on the unthreaded
  version.
- _shard_index_byte_range(): the index-location byte-range arithmetic existed
  verbatim in both _load_shard_index_maybe and its _sync twin; now one helper.
- _pair_chunks_with_byte_ranges(): the chunk-coord/byte-range pairing loop
  existed verbatim in both _load_partial_shard_maybe and its _sync twin; now
  one helper.

Deliberately NOT unified: the small hand-rolled loops remaining in
_decode_sync/_encode_sync vs their async twins. Post sync-bytegetter work the
async versions are thin delegations to the (shared, evolved) nested pipeline,
so the heavy machinery — evolve, transforms, layout, index codecs — is already
single-sourced; force-merging the residual loops would couple different
missing-chunk/concurrency semantics for little drift-surface gain. The
pipeline-parity suite guards their behavioral equivalence.

Full battery passes twice (636); ruff + mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: canonical chunk write-state functions; fix complete-chunk merge copy

Prototype of the "chunk state algebra" direction: the write-side state logic
(maybe-read existing -> merge -> empty-normalize -> encode-or-elide) existed in
four places with divergent inline conventions. It is now three canonical
functions in codec_pipeline.py:

- chunk_is_empty(): THE write_empty_chunks normalization rule (all-fill chunk
  normalizes to missing), previously five scattered inline all_equal checks.
- encode_or_elide_chunk(): normalize-empty + encode; None = must not be stored.
- merge_and_encode_chunk(): the full single-chunk write transition. Used by the
  fused _write_one and both branches of the sharding _encode_partial_sync loop
  (including the scalar-broadcast memoization, which now memoizes the canonical
  function's result). _encode_sync uses encode_or_elide_chunk.

Unification found a real perf bug: _merge_chunk_array's complete-chunk early
return required value.shape == chunk_spec.shape, which never holds for
multi-chunk writes — so every complete chunk of every multi-chunk fused write
paid a create+fill+copy. (Sharding's hand-rolled loop bypassed this with a
view, which is itself how the two copies had drifted.) The guard now returns
value[out_selection] whenever it is exactly chunk-shaped. Measured, two A/B
passes: unsharded full write (1000 chunks) ~1.5x faster; bulk partial shard
write (900 complete inner chunks) ~1.4x faster; sharded single-chunk write
unchanged. Callers pass existing=None for complete chunks so fully-overwritten
data is never decoded.

Also removed the last divergent missing-chunk conventions: _decode_sync's
try/except KeyError is now .get() -> None (None is the single "missing"
convention), and the dead skip_empty/fill_value prologues are gone.

1945 tests pass including pipeline-parity byte-identical assertions; ruff +
mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: canonical chunk read functions (decode_and_scatter / scatter_chunk)

The read twin of merge_and_encode_chunk: the fill-on-missing scatter logic
existed in five places (fused _read_one, partial-decode _read_one, the async
fallback scatter loop, and the two sharding decode tails) with two different
missing conventions (None vs try/except KeyError) and three fill spellings
(precomputed batch fill, raw shard_spec.fill_value, inline or-default).

Now two canonical functions in codec_pipeline.py:
- scatter_chunk(): scatter an already-selected region; None = missing ->
  scatter fill, return a "missing" GetResult. POLICY-FREE on purpose: whether
  missing is an error (read_missing_chunks=False) is decided at the array
  layer from the top-level statuses — which is exactly what makes missing
  INNER chunks of a present shard fill rather than raise (the sharding codec
  discards the nested statuses). That semantic, previously implicit in which
  loop happened to run, is now written down where the rule lives.
- decode_and_scatter_chunk(): decode (None = missing) -> select -> scatter.

All five sites converted; the last try/except KeyError missing-convention is
gone (Mapping.get works for both _ShardReader and plain dicts). Read
benchmarks A/B neutral-to-marginally-better on all paths (unsharded full,
sharded full/partial, half-missing fill).

786 tests pass including pipeline-parity and indexing; ruff + mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* test: property tests asserting fast paths equal general paths

Add tests/test_fastpath_equivalence.py with four hypothesis properties,
one per fast path on the branch:

- _merge_chunk_array complete-chunk view == general merge path (and
  independent of existing chunk content)
- ShardingCodec._decode_full_shard_bulk == _decode_sync for dense
  uncompressed shards across dtypes, endianness, subchunk write order,
  and index location (asserts the bulk path actually applies, so the
  test cannot pass vacuously)
- scalar writes leave the store byte-identical to equivalent broadcast
  array writes (pins the sharded scalar-broadcast memoization)
- Store.get_ranges_sync coalesced reads == one get_sync per range, for
  arbitrary gap/coalesce limits and Range/Offset/Suffix/None requests

Each was verified to catch its bug class by temporary fault injection:
dropping the bulk decode's endian handling and shifting the coalesce
re-slice offset by one both produced shrunk falsifying examples.

Assisted-by: ClaudeCode:claude-fable-5

* fix: address open review findings across sharding, pipeline, and tests

Code fixes:
- ShardingCodec.evolve_from_array_spec now threads the spec through the
  inner chain via evolve_codecs. The unthreaded evolve survived on the
  real array-creation path after the transform builders were fixed,
  baking an endian-stripped BytesCodec into the evolved instance behind
  any dtype-changing inner codec. Regression test goes through
  evolve_from_array_spec and then builds the inner transform.
- Async _decode_shard_index/_encode_shard_index fall back to the async
  pipeline when an index codec is not sync-capable, instead of failing
  every path for third-party async-only index codecs.
- Removed the redundant hand-rolled dict caches inside the chunk
  transform builders; the instance-local lru_cache wrappers are the
  single memoization mechanism. _shard_index_size is now lru_cached too.
- The encoded-index-size guard lives once, in _assemble_shard, instead
  of duplicated in the sync and async encoders.
- _get_inner_pipeline cache key includes codec_pipeline.batch_size,
  which from_codecs captures at construction.
- Coordinate arrays built via np.indices instead of
  np.array(list(np.ndindex(...))) in the partial-write loaders.
- _merge_chunk_array docstring states the view-aliasing contract; the
  guard comment sits on the check it annotates.
- The socketpair unraisable filter covers family=(1|2) (Windows
  emulates socketpair with AF_INET), and its comment owns the tradeoff
  that the patterns cannot scope to pytest-asyncio.

Test hardening:
- Pool tests assert the pool branch actually fires (_resolve_max_workers
  + a _get_pool spy) instead of silently degrading to the sequential
  branch on a config regression; the concurrent-read test re-opens the
  array each round so readers race a cold spec cache.
- New direct test pins the SyncByteSetter write gate in
  FusedCodecPipeline.write (verified to fail with the gate removed).
- New test pins the merge fast path's view-aliasing + source-unmutated
  contract end-to-end on both pipelines.
- read_missing_chunks=False sharded test asserts both halves of the
  asymmetry against the same partially-written array.
- v2 scenario with a numcodecs Delta filter covers the V2Codec filter
  branch; _chunk_keys recognizes v2 metadata keys.

Assisted-by: ClaudeCode:claude-fable-5

* doc: add changelog entries for PR #3885

Rename the fused-default placeholder to the PR-numbered 3885.feature.md and
add a second feature entry for the new SyncByteGetter/SyncByteSetter protocols
and Store.get_ranges_sync.

Assisted-by: ClaudeCode:claude-opus-4.8

* test: cover async fallback paths orphaned by the Fused default

Making FusedCodecPipeline the default left the async (Batched) mirror
paths and the new sync-IO error paths exercised only narrowly, dropping
project coverage. Add targeted tests for the reachable gaps:

- AsyncChunkTransform.decode_chunk/encode_chunk == ChunkTransform across
  aa/ab/bb codec combinations (the async per-chunk chain the default
  sync path never runs), plus FusedCodecPipeline.encode/decode
  None-chunk passthrough.
- ShardingCodec._decode_single/_encode_single whole-shard round-trip and
  all-empty branches. The codec advertises partial decode/encode, so the
  pipeline always picks the partial methods; these whole-shard async
  methods are reached only via the direct ArrayBytesCodec API.
- Async-only index codec fallback in _decode_shard_index/
  _encode_shard_index (#269), via a test-only async-only ArrayBytesCodec
  stub that is not SupportsSyncCodec.
- Store.get_ranges_sync happy path, missing-key BaseExceptionGroup, and
  the non-sync-store TypeError.

Local merged coverage on the touched files: codec_pipeline.py
85.4->92.2%, sharding.py 92.4->96.7%, abc/store.py 93.8->95.3%.

Assisted-by: ClaudeCode:claude-opus-4.8

* refactor: move reused functions around chunks into standalone file

* refactor: use more helpful name and remove comment

* refactor: move more

* refactor: move more

* Ig/as completed read (#194)

* perf: use `as_completed`

* refactor: unfiy branches

* refactor: `concurrent_iter` only handles running

* add note

* add note about blocking

* chore: latency distribution in store

* refactor: as_completed reading

* perf: as_completed writes

* chore: remove dead code

* fix: use concurrent_map

* fix: remove unnecessary list

* fix: revert use of concurrent_map

---------

Co-authored-by: ilan-gold <[email protected]>

* fix: bulk shard decode must not serve reordering reads in natural order

`ShardingCodec._decode_full_shard_bulk_if_uncompressed` gated the
vectorized whole-shard fast path on `indexer.shape == shard_spec.shape`.
For a `CoordinateIndexer` (vindex / integer-array oindex), `.shape` is the
flattened point count, which can equal the shard shape by coincidence
(trivially in 1-D). Such a selection then passed the gate and the bulk
path returned the shard in natural order, silently dropping the
reordering — data corruption on uncompressed, crc-free shards.

Gate instead on `sel_shape`: a gather indexer exposes it, a contiguous
full read (BasicIndexer, or a non-gathering OrthogonalIndexer from
`arr[:]`) does not. `isinstance(indexer, BasicIndexer)` would be too
strict — `arr[:]` produces an OrthogonalIndexer and must keep the fast
path.

Regression guard: test_reordering_read_on_uncompressed_shard_honors_selection
exercises the end-to-end read path (where the gate lives), which the
BasicIndexer-only bulk-decode parity test could not reach. The `arrays()`
hypothesis strategy now also samples the uncompressed single-BytesCodec
sharding config (via `sharding_inner_codecs`) so the fast path is covered
under randomized indexing going forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: make ChunkTransform spec cache torn-read-safe under threading

A `ChunkTransform` is shared across thread-pool workers (read_sync /
write_sync with max_workers > 1). Its `_resolve_specs` cache stored the
key and the resolved specs in three separate fields, written
non-atomically: a worker could observe a freshly-set key paired with the
previous (or None) specs, returning the wrong codec chain for a chunk_spec
— silent corruption with mixed specs, or a tripped assert.

Collapse the cache into a single `(key, aa_specs, ab_spec)` tuple field
replaced with one atomic attribute write. Under the GIL a reader now sees
either the complete old entry or the complete new one, never a torn mix;
worst case is a recompute, never a wrong result.

Regression guard: test_shared_transform_decode_alternating_specs pins the
single-slot eviction/refill correctness that underpins the atomicity (it
fails if specs go stale on eviction).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: default Codec.is_fixed_size so sharded vlen/numcodecs reads don't crash

`ShardingCodec._inner_codecs_fixed_size` reads `c.is_fixed_size` on every
inner codec to gate the bulk-decode fast path. `is_fixed_size` was
declared on the Codec ABC as a bare annotation with no default, so codecs
that never set it — VLenUTF8Codec, VLenBytesCodec, the numcodecs wrappers
— raised AttributeError, crashing every sharded read whose inner chain
included such a codec under the default (Fused) pipeline.

Give the ABC a conservative default `is_fixed_size = False`. It stays a
class attribute (not a dataclass field; fixed-size codecs still override
with True), and treating an unknown codec as not-fixed-size only disables
the size-dependent fast path, never correctness.

Regression guard: test_sharding_vlen_inner_codec_roundtrip (Fused +
Batched). Uses StringDType to force the VLenUTF8 inner chain — a
fixed-width <U dtype would use BytesCodec and not reproduce.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor: drop unreachable None branch in _async_write_fallback

`_merge_chunk_array` always returns a real NDBuffer, so the merged-chunk
list never contains None and the `if chunk_array is None` branch (with its
`# type: ignore[unreachable]`) was dead. Replace the loop with a
compreh…

…_multiprocessing (zarr-developers#4143)

Python 3.15 makes os.fork() emit a DeprecationWarning when the process
has multiple threads (python/cpython#84559). zarr's sync event-loop
thread is always running by the time test_multiprocessing forks, and the
suite runs with filterwarnings = ["error"], so the fork parametrizations
fail on 3.15. Fork-safety despite those threads is exactly what this
test pins down (zarr registers an os.register_at_fork handler), so keep
the coverage and ignore the warning with a targeted per-param filter.

Fixes zarr-developers#4086

Assisted-by: ClaudeCode:claude-fable-5

* fix: skip object store prefix markers in list_dir

Signed-off-by: ABrandes <[email protected]>

* Address list_dir prefix marker review

Signed-off-by: ABrandes <[email protected]>

* Format prefix marker follow-up

Signed-off-by: ABrandes <[email protected]>

* Test S3 prefix marker handling

Signed-off-by: ABrandes <[email protected]>

* Filter boto3 deprecation in S3 test

Signed-off-by: ABrandes <[email protected]>

* Address final prefix marker review

Signed-off-by: ABrandes <[email protected]>

---------

Signed-off-by: ABrandes <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>

…rr-developers#4142)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* chore(deps): bump the actions group across 1 directory with 8 updates (zarr-developers#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: byte-order handling for structured dtypes in the bytes codec (zarr-developers#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* docs: 3.3.0 release notes

Add missing changelog fragments for zarr-developers#3955 (datetime64/timedelta64 V3
metadata types), zarr-developers#3966 (writes to 0-dimensional sharded arrays), and the
public alias renames that accompanied the zarr-developers#3963/zarr-developers#3968 enum deprecations,
then build the 3.3.0 release notes with towncrier, consuming all
fragments accumulated since v3.2.1.

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <[email protected]>

* chore: CI worfklows for missing docs and RST docstrings

* Add linter for docs

* Add link checking workflow

* fix links

* fix: apply suggestions from code review

* refactor: readability improvements

* chore: switch lychee action to opening issue on failure

---------

Co-authored-by: Davis Bennett <[email protected]>

…4153)

* chore: cleanup indentation and guidance for release

* Make explanation a comment

…zarr-developers#4152)

Bumps the actions group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `6.0.3` | `7.0.0` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `8.2.0` | `8.3.2` |
| [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.18.1` | `4.18.5` |
| [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics) | `4.2.8` | `5.0.0` |
| [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) | `2.8.0` | `2.9.0` |
| [actions/labeler](https://github.com/actions/labeler) | `6.1.0` | `6.2.0` |



Updates `actions/checkout` from 6.0.3 to 7.0.0
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v6.0.3...9c091bb)

Updates `astral-sh/setup-uv` from 8.2.0 to 8.3.2
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@fac544c...11f9893)

Updates `CodSpeedHQ/action` from 4.18.1 to 4.18.5
- [Release notes](https://github.com/codspeedhq/action/releases)
- [Changelog](https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md)
- [Commits](CodSpeedHQ/action@a4a36bb...f99becd)

Updates `github-community-projects/issue-metrics` from 4.2.8 to 5.0.0
- [Release notes](https://github.com/github-community-projects/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@44173f9...df8c49d)

Updates `lycheeverse/lychee-action` from 2.8.0 to 2.9.0
- [Release notes](https://github.com/lycheeverse/lychee-action/releases)
- [Commits](lycheeverse/lychee-action@8646ba3...e747777)

Updates `actions/labeler` from 6.1.0 to 6.2.0
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@f27b608...b8dd2d9)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/labeler
  dependency-version: 6.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: astral-sh/setup-uv
  dependency-version: 8.3.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: CodSpeedHQ/action
  dependency-version: 4.18.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github-community-projects/issue-metrics
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: lycheeverse/lychee-action
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: byte-order handling for structured dtypes in the bytes codec (zarr-developers#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* fix(store): copy buffers on write in MemoryStore

Encoding an uncompressed chunk hands the store a zero-copy view of the
caller's array, and MemoryStore keeps whatever it is given alive in a
dict rather than serializing it. Mutating the source array after a write
therefore rewrote chunks already committed to the store, silently.

Stores that serialize on write (LocalStore, ZipStore, remote stores) are
unaffected, so they keep the full benefit of zarr-developers#3885. Only MemoryStore pays
the copy, and only where it was aliasing to begin with: an uncompressed
34 MB write goes from ~23 ms to ~39 ms, while compressed writes are
unchanged.

Copying at the store boundary rather than narrowing the fast path in
_merge_chunk_array also fixes the single-chunk case, which aliased in
v3.2.1 too.

Assisted-by: ClaudeCode:claude-opus-4.8

* docs: add changelog entry for MemoryStore buffer copy

Assisted-by: ClaudeCode:claude-opus-4.8

* docs: correct changelog

…arr-developers#4165)

* fix: byte-order handling for structured dtypes in the bytes codec (zarr-developers#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* fix(store): FsspecStore.close() no longer closes the filesystem

FsspecStore.close() closed the underlying filesystem's session, on the
premise that a store built by from_url "owns" the filesystem it created.
That premise does not hold: fsspec caches and shares filesystem
instances across callers (its instance cache keys on storage options,
not path), and users can hand one filesystem to many stores directly.
Closing one store therefore killed the session that sibling stores were
still using, and left the dead filesystem in fsspec's cache for later
callers.

Determining whether a filesystem is actually shared requires reaching
into fsspec's private instance cache (_cache, _fs_token, cachable) and
walking wrapper chains for caching/proxy filesystems — an implementation
detail that leaks upward and that we would have to keep in sync with
fsspec forever, getting it subtly wrong in between. The wrapper case
alone (simplecache::/dir://) already slipped through a cache-membership
check.

The filesystem's lifecycle is simply not the store's to manage. This
removes the ownership model added in the unreleased zarr-developersgh-4003: no _owns_fs,
no _close_fs, no ownership transfer in with_read_only, and close() just
marks the store not-open. The only thing given up is suppressing an
"Unclosed client session" ResourceWarning, which was true anyway — the
session belongs to a cached filesystem that outlives the store.

Since zarr-developersgh-4003 never shipped (latest release is v3.2.1), its changelog
fragment is removed rather than superseded.

Assisted-by: ClaudeCode:claude-opus-4.8

* test: skip with_read_only fs test when AsyncFileSystemWrapper is absent

test_with_read_only_shares_filesystem replaced an ownership test that
carried a guard for fsspec < 2024.12.0, and the guard was dropped in the
rewrite. The test still opens a file:// URL, which needs
AsyncFileSystemWrapper, so it failed the min_deps job.

Assisted-by: ClaudeCode:claude-opus-4.8

* docs: correct changelog claim about zarr-developersgh-4003 release status

The fragment said zarr-developersgh-4003 was unreleased with no net change for
released versions. Its text is already in the staged 3.3.0 release
notes, so the revert is a real behavior change for anyone relying on
close() releasing the session.

Assisted-by: ClaudeCode:claude-opus-4.8

* docs: remove changelog entry for unreleased versions

)

Bumps [setuptools](https://github.com/pypa/setuptools) from 82.0.1 to 83.0.0.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v82.0.1...v83.0.0)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-version: 83.0.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.2.0 to 12.3.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@12.2.0...12.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.3.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

)

Bumps the actions group with 3 updates: [actions/labeler](https://github.com/actions/labeler), [actions/attest](https://github.com/actions/attest) and [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action).


Updates `actions/labeler` from 6.2.0 to 7.0.0
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@b8dd2d9...bf12e9b)

Updates `actions/attest` from 4.1.1 to 4.2.0
- [Release notes](https://github.com/actions/attest/releases)
- [Changelog](https://github.com/actions/attest/blob/main/RELEASE.md)
- [Commits](actions/attest@a1948c3...f7c74d2)

Updates `zizmorcore/zizmor-action` from 0.5.7 to 0.6.0
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@192e21d...6599ee8)

---
updated-dependencies:
- dependency-name: actions/labeler
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/attest
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#4187)

Allows constructing a ZipStore from any seekable binary reader, enabling
zip archives on remote storage:

- io objects (BytesIO, fsspec file objects) are used directly
- minimal readers that are not io.IOBase instances and whose read() may
  return buffer-protocol objects rather than bytes (e.g.
  obstore.ReadableFile) are adapted via a small io.RawIOBase wrapper
  when opened for reading

clear()/move() raise NotImplementedError for file-object-backed stores.

Co-authored-by: Claude Fable 5 <[email protected]>

…p` and `Group` (zarr-developers#4128)

* chore(deps): bump the actions group across 1 directory with 8 updates (zarr-developers#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add type-safe get_array and get_group methods to AsyncGroup and Group

Add `get_array` and `get_group` methods that return the child node at
a given path or raise: `ArrayNotFoundError`/`GroupNotFoundError` when
no node exists, and `ContainsGroupError`/`ContainsArrayError` when the
node is the wrong kind. Paths mirror `getitem` semantics, so nested
paths like "subgroup/subarray" work on both the plain and
consolidated-metadata lookup routes.

Existing tests that fetched a child via `getitem` and then manually
narrowed the type with `isinstance` asserts or a walrus expression now
use the new methods instead.

Assisted-by: ClaudeCode:claude-fable-5

* fix: pass pre-formatted messages to error constructors in get_array/get_group

The multi-argument template form of BaseZarrError.__init__ is
documented as deprecated; build the message string at the raise site
instead, matching every other call site in the codebase.

Assisted-by: ClaudeCode:claude-fable-5

* docs: add changelog entry for get_array/get_group

Assisted-by: ClaudeCode:claude-fable-5

* docs: demonstrate get_array/get_group in the groups user guide

Assisted-by: ClaudeCode:claude-fable-5

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump the actions group across 1 directory with 8 updates (zarr-developers#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(zarr-metadata): add model._validation — structural validators

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): add model._array — array metadata models

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): port array model test suite

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): add group metadata models

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): add consolidated metadata model tests

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): export model layer from package front door

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): changelog entry for the model layer

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): harden model validation and error reporting

Findings from an API-ergonomics exercise (a fresh agent consuming
defective metadata documents):

- ValidationProblem gains a machine-readable kind (missing_key /
  invalid_type / invalid_value / invalid_json), ending message
  string-matching in consumers.
- The v2 array validator now enforces what its types declare (dtype,
  order, compressor, filters, dimension_separator), and all four
  document validators check the fixed zarr_format / node_type literals.
- All ingestion failures surface as MetadataValidationError: missing
  store keys and undecodable bytes in from_key_value (previously
  KeyError / JSONDecodeError) and constructor invariants (previously
  bare ValueError).
- ZarrMetadataV3 is renamed NamedConfigModelV3: it models a name +
  configuration pair, and the old name read as a whole-document type.
- Discoverability: the validate_*/is_*/parse_* contract is documented
  on zarr_metadata.model itself; update() documents that it does not
  re-validate; the v2 to_json/to_key_value attributes split is
  documented on both.

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): annotate metadata fields with role alias MetadataFieldModelV3

Model fields and consumer signatures should convey the logical meaning
of the type (a metadata-document field), not the form it takes when
JSON-serialized (a named configuration). MetadataFieldModelV3 is today
exactly NamedConfigModelV3; if a future spec revision adds a field form
that cannot normalize to name + configuration, the alias widens to a
union and annotation sites do not move. Mirrors the raw-layer split
between NamedConfigV3 (shape) and MetadataV3 (field union).

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): assert required-key coverage via the typed constant

test_v3_to_json_includes_required_fields hand-enumerated keys with
chained asserts, restating what ARRAY_METADATA_REQUIRED_KEYS_V3 already
defines. Now: one coverage assert driven by the constant (tracks the
TypedDict automatically) and one whole-document equality for the values.

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): single whole-document comparison for v3 to_json

The subset assert against ARRAY_METADATA_REQUIRED_KEYS_V3 was redundant:
equality with a literal that spells out the full document already covers
every required key. One dict, one assert.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): close validation holes found by adversarial review

Invalid documents that previously passed validation:

- shape/chunks containing JSON booleans (bool is an int subclass in
  Python but not an integer in a metadata document) or negative values
- dimension_names whose length does not match shape
- attributes and configuration values that are not JSON-serializable —
  now checked recursively like fill_value, so an int-keyed dict cannot
  be silently rewritten by json.dumps on round-trip and a set() cannot
  escape as a TypeError from to_key_value
- consolidated_metadata envelopes: the group validator now deep-validates
  the envelope and its entries via the shared
  validate_consolidated_metadata_v3, which ConsolidatedMetadataModelV3
  .from_json also uses, so is_group_metadata_v3 never vouches for a
  document the model constructor would reject

Three pre-existing test fixtures paired dimension_names=('x',) with the
default scalar shape () and were themselves spec-invalid; they now use a
matching 1-d shape.

Deliberately unchanged, pending a design decision: unknown extension
fields with must_understand: true still pass (which layer owns the
spec's refusal duty), and empty v2 dtype records / empty codec names
still pass (domain territory).

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): expose must_understand_fields on the v3 models

The v3 core spec: 'An implementation MUST fail to open Zarr groups or
arrays if any metadata fields are present which (a) the implementation
does not recognize and (b) are not explicitly set to
"must_understand": false' — and fields are implicitly must-understand
unless waived.

The model layer cannot discharge this itself: recognition is
reader-specific (consolidated_metadata is itself an extension field one
reader understands and another does not), and a document carrying a
must-understand extension is still a valid document. So the models
partition by obligation: must_understand_fields is the subset of
extra_fields not explicitly waived, and a compliant reader fails to
open when must_understand_fields.keys() - recognized is non-empty.
The design spec pins that duty on the part-2 resolve layer, matching
what zarr-python's parse_extra_fields enforces today.

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): executable example of pydantic integration

Delegate wholesale rather than letting pydantic introspect the dataclass:
InstanceOf (is-instance core schema) + BeforeValidator(from_json) +
PlainSerializer(to_json, return_type=dict). Field-by-field validation is
impossible anyway (the models' annotation-only imports live behind
TYPE_CHECKING, so pydantic raises class-not-fully-defined) and would
diverge from the library's structural validation via coercion if it
weren't. MetadataValidationError subclasses ValueError, so failed parses
surface as pydantic ValidationError with the loc-annotated messages.

pydantic is already in the package's test dependency group.

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): document pydantic's native dataclass path and why not

Correcting the previous commit's too-strong claim: pydantic CAN
introspect the model dataclass — TypeAdapter(...).rebuild() with the
TYPE_CHECKING-only names supplied as _types_namespace resolves the
schema, and __post_init__ invariants still run. A new test exercises
that path and pins why it is not the recommended integration: it
validates the model shape, not the document (bare-string data_type
rejected — no from_json normalization), and pydantic's lax coercion
silently re-opens holes the library validators close (shape=[True, -5]
coerces to (1, -5); a wrong dimension_names count passes).

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): engine-backed pydantic BaseModel example (pydantic-zarr pattern)

For consumers that want a first-class BaseModel — JSON schema generation
and generics for typed attributes, as in pydantic-zarr's ArraySpec — the
example adds a third pattern: pydantic-native fields as the user-facing
surface, with the library as the engine. A mode='before' validator
canonicalizes every input via from_json(...).to_json(), so structural
validation and normalization run before pydantic parses fields (the
[True, -5] coercion divergence cannot occur), and to_metadata_model /
to_document bridge both ways through the document form. One translation
noted at the bridge: the document spells 'no dimension names' as key
absence, the pydantic side as None.

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): pin that a null dimension_names field is invalid

Spec: 'If specified, must be an array of strings or null objects...
If dimension_names is not specified, all dimensions are unnamed.' The
null object is a permitted element (an unnamed dimension), never the
field value; key absence is the only spelling of 'not specified'. Pins
the validator's existing rejection so it is not later 'fixed' to accept
null-as-absence, and documents that in-memory None maps to key absence
on serialization.

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): optional pydantic integration as zarr_metadata.pydantic

Gamed out three shapes with prototypes before choosing:
- dunders on the core classes (works, verified pydantic 2.0-2.13, but
  puts a framework protocol in the dependency-free layer);
- pydantic-aware SUBCLASSES in a namespace (rejected on empirical
  failures: identity split breaks equality, core instances are rejected
  by subclass-typed fields, and nested construction produces core-class
  children unless every cross-reference is overridden);
- Annotated field types over the CORE classes in an opt-in module
  (chosen): instances are the core classes so interop is free, pydantic
  imports eagerly at the module (loud failure when absent), core stays
  framework-free, and pydantic-protocol risk is quarantined to one
  clearly-labeled module.

The module exports one field type per model. Validation delegates to
from_json (structural validation and normalization cannot be bypassed by
pydantic coercion), instances pass through, serialization emits the
canonical document, and WithJsonSchema describes the accepted document
form so model_json_schema works. Tests cover all seven field types,
core-instance interop, error quality, JSON schema, roundtrip, and that
importing zarr_metadata does not import pydantic.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): create_default derives the chunk grid from shape

create_default(shape=(100, 100)) silently kept the scalar default's 0-d
chunk grid (chunk_shape: ()), producing a structurally-valid but
semantically inconsistent document — a footgun for every test fixture
built on it. When shape is overridden and the grid is not, the default
is now one regular chunk covering the array (v3 chunk_shape == shape,
v2 chunks == shape); an explicit chunk_grid/chunks override still wins.
update() stays a dumb dataclasses.replace, per its documented contract.

One existing whole-document test literal carried exactly this
inconsistency (shape (10,) with chunk_shape ()) and was updated.

Assisted-by: ClaudeCode:claude-fable-5

* test(zarr-metadata): pin zero-length-dimension case of the derived chunk grid

The spec's constraint is conditional ('non-zero when the corresponding
dimensions of the arrays have non-zero length'), so chunk_shape == shape
is sound for every shape, including empty dimensions.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): document that create_default's derivation is one-way

Overriding shape without a grid derives the grid; the reverse does not
hold. A user-supplied chunk_grid is an extension point taken verbatim —
deriving shape from it would require interpreting grid configurations,
which the model layer never does and cannot do for unrecognized grid
names. Pinned by test so the asymmetry reads as a decision, not an
oversight; the v2 model documents the same one-way rule for chunks for
cross-version consistency.

Assisted-by: ClaudeCode:claude-fable-5

* refactor(zarr-metadata): eliminate every type-ignore comment

Audited all 24 (15 src, 9 tests); each was either obsolete, replaceable
by a sound cast, or avoidable by better-typed code:

- Two fill_value arg-type ignores were factually obsolete: their
  justifying comment said 'fill_value: object in upstream TypedDict',
  but 0.3.0 narrowed it to JSONValue.
- Eight pre-existing call-arg/reportInvalidTypeForm ignores on the PEP
  728 TypedDicts and the recursive JSONValue alias were mypy-dialect
  suppressions that the checker of record (pyright strict with
  enableExperimentalFeatures) never needed; mypy has never checked this
  package.
- The two extra_fields comprehensions are a genuine checker limitation
  (a key filter cannot narrow a PEP 728 TypedDict's item-value union),
  now expressed as casts whose comments state the soundness claim
  instead of suppressing the diagnostic.
- pydantic.py's generic coercer factory takes the parse callable
  explicitly instead of calling from_json through type[_M].
- NamedConfigModelV3.from_json casts the validated configuration (sound
  since configuration values are now deep-validated as JSON).
- Tests: _build_v2/_build_v3 gained real Unpack[...Partial] signatures;
  raw-document pydantic inputs go through model_validate (the idiomatic
  entry point for untyped data) instead of ignoring constructor
  signatures; the frozen-dataclass test uses setattr for its
  intentional runtime error.

src and tests/model now carry zero type-ignore comments.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): absent v2 dimension_separator means '.', not '/'

roborev job 426 (branch review) found that ArrayMetadataModelV2
normalized an ABSENT dimension_separator key to '/', inherited verbatim
from the zng prototype. The v2 convention's default is '.': a consumer
deriving chunk keys from the model against a real-world v2 array
written with the default separator would have looked for '0/0' instead
of '0.0'. No test caught it because every fixture started from
create_default(), which always carries an explicit separator.

Absence is normalized to an explicit '.' -- a semantics-preserving
spelling normalization consistent with the model's existing canonical
forms (bare-string metadata fields, missing configuration). The field
is deliberately NOT modeled as Optional: the document grammar has no
null spelling for this key, and a None in the model invites writing
'dimension_separator': null into documents. Pinned by three tests,
including explicit-null rejection.

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): UNSET sentinel for absent optional document keys

Establishes the models' None/absence invariant: None in a model always
corresponds to a JSON null in the document (a v2 compressor/filters
value, an unnamed dimension inside dimension_names), and UNSET always
means the key is absent. The two are never interchangeable.

Applied to the two fields that used None as an absence marker:
dimension_names (ArrayMetadataModelV3) and consolidated_metadata
(GroupMetadataModelV3). For dimension_names this also preserves a
semantic distinction d-v-b identified: an absent field ("there are no
dimension names") and an explicit all-null array ("every dimension has
a name, which is null") are different documents; both spellings now
round-trip faithfully and compare unequal.

Normalizing absence to the all-null form was considered and rejected:
the spellings' interpretations coincide but interpretation-equivalence
is the resolve layer's business, and collapsing document-level
distinctions on that basis is the layer violation this package exists
to avoid. Verified that current zarr-python never writes
"consolidated_metadata": null (GroupMetadata.to_dict pops the key), so
None there was purely an absence marker, not a document spelling.

UnsetType is a single-member enum (identity-checkable, repr "UNSET",
deliberately truthy so `if not x` cannot silently treat it as absent);
UNSET and UnsetType are exported from zarr_metadata.model and the
package front door.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): accept and preserve the wild consolidated_metadata null

Historical zarr-python versions wrote "consolidated_metadata": null into
group documents for groups without consolidated metadata, so real stores
contain the spelling; the validator was rejecting those documents
("expected a mapping"). Per the None/UNSET invariant, the field is now
honestly three-state: UNSET (key absent), None (the document's literal
null, preserved on round-trip), or a ConsolidatedMetadataModelV3.
Interpreting null as absence is the consumer's call, not a document
rewrite by this layer.

Also records an implementation constraint on the sentinel itself:
typing_extensions.Sentinel (PEP 661) is the intended spelling, but
pyright 1.1.411 degrades a Sentinel to Unknown in dataclass FIELD
annotations (function signatures work), verified by probe both with and
without enableExperimentalFeatures. Using it would reintroduce
suppressions at every use site under the strict gate, so UNSET stays a
single-member enum, with the Sentinel switch documented in
_sentinel.py for when pyright catches up.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): repair consolidated_metadata null to absence, not preserve it

d-v-b: the bugged spelling should not be preserved or honored. The
three-state field reverts to two states (model | UNSET): a document
carrying "consolidated_metadata": null — written by a historical
zarr-python bug — remains readable (the validator accepts it so real
stores open), but the spelling gets no model representation: it is read
as absence and never written back. This is the one deliberate exception
to faithful round-tripping, pinned as such: from_json(null_doc) equals
from_json(absent_doc), and to_json omits the key.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): pin the Sentinel blocker to pyright regression #11115

Investigated: the Unknown-degradation of typing_extensions.Sentinel is a
confirmed upstream pyright regression, not by-design. Introduced in
1.1.405 (verified: 1.1.404 is clean on the same probe, 1.1.411 fails),
affects reads of any class-body attribute annotation (dataclass or
plain class), does not affect function signatures or module variables,
and Final on the sentinel does not help. Tracked as
microsoft/pyright#11115 (open, bug+regression); #11467 closed as its
duplicate. The enum sentinel stays until the fix lands.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): sentinel switch is blocked by mypy too, not just pyright

Pinning a working pyright (<= 1.1.404) in CI was considered and does
not suffice: the pin controls one of four checker surfaces. Contributor
IDEs (Pylance bundles current pyright) and downstream consumers'
pyright read the py.typed inline annotations with their own versions,
and decisively, mypy 2.1.0 has no PEP 661 support at all — a sentinel
in type position is a hard [valid-type] error, which would degrade
these fields to Any for mypy consumers, including zarr-python itself.
The enum is currently the only spelling with exact types on every
surface; switch when pyright#11115 is fixed AND mypy implements PEP 661.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): PEP 661 is Final (Python 3.15), not a draft

Corrects the sentinel implementation note: PEP 661 was accepted
2026-04-23 and ships as stdlib sentinel in Python 3.15. The two checker
gaps blocking the Sentinel spelling (pyright regression #11115, mypy
not yet implementing the PEP) are therefore temporary gaps against a
Final standard, and the enum is a stopgap with a defined end state.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): ty fully supports typed sentinels; pyright/mypy are the laggards

ty 0.0.56 types the Sentinel spelling perfectly in dataclass fields:
exact T | UNSET unions, both-direction is/is-not narrowing, and
wrong-typed constructor arguments rejected (verified with reveal_type,
so it is real inference, not silent Any). The checker matrix for
sentinel-in-type-position is therefore ty full / pyright regressed
(#11115) / mypy not implemented — recorded so the switch decision has
current calibration.

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): adopt the PEP 661 sentinel for UNSET

d-v-b's call: PEP 661 is Final, ty already types the sentinel spelling
exactly, mypy support is in review (python/mypy#21647) and treated as
imminent, and pyright has a known-good version — so use the standard
sentinel today rather than carrying the enum stopgap.

- UNSET is now typing_extensions.Sentinel("UNSET"), used directly in
  type expressions (tuple[str | None, ...] | UNSET); the UnsetType
  companion enum is gone from the API.
- typing_extensions floor bumped to 4.14 (where Sentinel arrived).
- CI pins pyright==1.1.404, the last version before the class-attribute
  sentinel regression (microsoft/pyright#11115); pyproject documents the
  same pin for local runs. 0 errors on the pin; ty checks the sentinel
  fields clean (its 2 remaining diagnostics are its incomplete PEP 728
  extra_items write support, unrelated).
- Known short-term cost, accepted deliberately: mypy-checked consumers
  need cast/type-ignore at narrowing sites until mypy#21647 merges, and
  contributors' Pylance may show phantom Unknowns until the pyright fix
  ships. Recorded in _sentinel.py and the changelog.
- The pydantic native-introspection test reverts to documenting that
  introspection is unsupported (pydantic 2.13 cannot schema a Sentinel);
  the delegation patterns are unaffected.

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): .zattrs presence is part of the store, not an artifact

Resolves the last flagged round-trip question from the initial port:
to_key_value on the v2 models always emitted a .zattrs key, so a store
that never had one gained a file on round-trip. Per d-v-b's ruling,
attributes on ArrayMetadataModelV2/GroupMetadataModelV2 is now
`dict[str, JSONValue] | UNSET`: UNSET means no .zattrs file (and no
attributes key in the merged document form) and emits nothing, while
any dict — including an explicit empty {} — means the file exists and
is emitted. The two spellings stay distinct through round-trips, per
the None/UNSET invariant; create_default defaults to UNSET (a fresh
minimal node has no .zattrs).

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): require typing_extensions>=4.16 so UNSET pickles by reference

Models hold the UNSET sentinel as field values (dimension_names,
attributes), so any object graph containing a model must survive pickling
and deep-copying. A sentinel's contract is identity — state-based pickling
would produce impostor objects that fail every `is UNSET` check — which is
why typing_extensions <= 4.15 refused to pickle sentinels at all.

typing_extensions 4.16 implements Sentinel.__reduce__ as pickling by
reference (a lookup of the sentinel's name on its defining module), the same
mechanism enum members use, so the singleton identity survives the round
trip. Bump the floor and pin the behavior with tests: identity across
pickle/copy/deepcopy, models holding UNSET round-tripping, and a guard that
a non-importable sentinel still fails loudly rather than pickling by state.

Co-Authored-By: Claude Fable 5 <[email protected]>

* refactor(zarr-metadata): format-version-first naming; dataclasses primary, JSON suffix for documents

Applies the naming decisions from the PR discussion: ZarrV2/ZarrV3 moves to
the front of every type name so a format version cannot be misread as a
class revision, and the model dataclasses take the bare entity names
(ZarrV3ArrayMetadata, ZarrV3GroupMetadata, ZarrV3ConsolidatedMetadata,
ZarrV3NamedConfig, role alias ZarrV3MetadataField) while the TypedDict
document forms carry a JSON suffix (ZarrV3ArrayMetadataJSON, ...,
ZarrV3MetadataFieldJSON, ZarrV3NamedConfigJSON).

The zarr_metadata.pydantic field types take the bare entity names, matching
the model classes they validate into; the module now references the model
module qualified to keep those names free. Raw-layer names released in 0.3.0
are renamed without aliases (pre-1.0), documented in changes/4119.removal.md.
Validation problem messages name documents in plain English instead of type
names. snake_case function names (validate_array_metadata_v3, ...) and
SCREAMING_SNAKE constants are deliberately untouched: the revision ambiguity
the rename fixes does not arise for them, and renaming them is a separate
decision.

Co-Authored-By: Claude Fable 5 <[email protected]>

* fix(zarr-metadata): harden model validation

Assisted-by: Codex:gpt-5

* docs(zarr-metadata): define v3 conformance boundary

Assisted-by: Codex:gpt-5

* fix(zarr-metadata): preserve v3 extension obligations

Assisted-by: Codex:gpt-5

* fix(zarr-metadata): enforce v3 core extension rules

Assisted-by: Codex:gpt-5

* fix(zarr-metadata): align v3 additional field types

Assisted-by: Codex:gpt-5

* chore(zarr-metadata): finalize v3 model conformance

Assisted-by: Codex:gpt-5

* docs: remove llm docs

* chore(zarr-metadata): correct changelog PR number

Assisted-by: Codex:gpt-5

* fix(metadata): enforce canonical document boundaries

Assisted-by: Codex:GPT-5

* docs(metadata): record review fix design

Assisted-by: Codex:gpt-5

* fix(metadata): align validation and schemas

Assisted-by: Codex:gpt-5

* fix(metadata): tighten validation boundaries

Assisted-by: Codex:gpt-5

* fix(metadata): repair package CI

Assisted-by: Codex:gpt-5

* chore: remove stray design notes from docs

Assisted-by: ClaudeCode:claude-fable-5

* fix(zarr-metadata): deep-copy mutable state in to_json output

to_json previously returned documents holding direct references to the
model's internal dicts (attributes, named-config configurations, extra
fields, v2 compressor/filters, consolidated entries), so mutating a
serialized document silently mutated the frozen model. Every value that
can hold a mutable container is now deep-copied on the way out, with
parametrized tests proving mutation independence for all seven models.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): describe the widened package scope

The README tagline, intro, and scope section (and the PyPI description)
still presented the package as type definitions only. Restructure them
around the two layers plus optional integration, extend the contribution
scope to models and structural validation, and state the runtime-behavior
boundary explicitly.

Assisted-by: ClaudeCode:claude-fable-5

* style(zarr-metadata): move None to the end of the JSONValue union

The unpinned ruff in CI now enforces RUF036.

Assisted-by: ClaudeCode:claude-fable-5

* feat(zarr-metadata): export store-key aliases and type to_key_value with them

The six store-key Literal aliases were private and unused: only reachable
via underscore modules, and absent from every signature. Export them from
zarr_metadata.model beside their constants (matching the package's
name/constant pairing everywhere else), and key each to_key_value return
mapping by them so the store keys a model can emit are visible in its
signature. from_key_value keeps Mapping[str, bytes] input on purpose — it
accepts whole store mappings. A pair test guards export and value drift,
and the removal note now states the version-placement and JSON-suffix
conventions explicitly.

Assisted-by: ClaudeCode:claude-fable-5

* refactor(zarr-metadata): make every public type name parse against a naming grammar

Three grammars now cover the public surface, enforced by a conformance
test that walks every public module's __all__:

- core document/model names: ZarrV{2,3} + entity + optional role suffix
  (JSON / JSONPartial / Partial / StoreKey)
- extension-entity names: registered entity + exactly one role suffix
  (CodecMetadata, DataTypeName, FillValue, ...)
- a closed standalone-vocabulary allowlist for role-less scalar and
  diagnostic types, with a staleness guard

The three .z-file document types were the only names that fit no grammar
and are renamed: ZArrayMetadata -> ZarrV2ZArrayJSON, ZGroupMetadata ->
ZarrV2ZGroupJSON, ZAttrsMetadata -> ZarrV2ZAttrsJSON. The leading V2 of
V2ChunkKeyEncodingMetadata is that encoding's registered entity name, not
a format version; its module docstring now says so.

Assisted-by: ClaudeCode:claude-fable-5

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <[email protected]>

* fix: let open_like create arrays by default

* test: cover open_like read-only mode

---------

Co-authored-by: Davis Bennett <[email protected]>

…developers#4194)

* fix: byte-order handling for structured dtypes in the bytes codec (zarr-developers#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* fix(codec_pipeline): keep FusedCodecPipeline compute off the event-loop thread

FusedCodecPipeline.read/write ran their synchronous fast path inline on
the coroutine servicing the request — i.e. on the global zarr_io event
loop thread. Single-chunk batches decoded inline on the loop and
multi-chunk batches blocked the loop in pool.map, so every sync-API call
from every user thread serialized behind each other's codec compute.
The blocked window scales with codec cost, which is why users reported
the fused pipeline as "slower for zstd-compressed data" under
multi-threaded (dask-style, one chunk per call) access: at 8 reader
threads on 4 MiB zstd chunks it was 3.4x slower than
BatchedCodecPipeline, and throughput did not scale with threads at all
(336 -> 439 ms from 1 to 8 threads, versus 669 -> 121 ms for batched).

Offload the synchronous batch to a worker thread with asyncio.to_thread:
one hop per batch, not per chunk, preserving the fused pipeline's win
over per-chunk async scheduling while keeping the loop free. After the
fix the same workload scales 625 -> 109 ms from 1 to 8 threads, beating
batched at every thread count; single-threaded performance is unchanged
(the hop costs ~75 us per batch).

The regression test asserts deterministically (no timing) that codec
compute never runs on a thread with a running event loop, covering
single- and multi-chunk reads and writes through the sync API. A new
benchmark covers the many-threads/one-chunk-per-call access pattern.

Assisted-by: ClaudeCode:claude-fable-5

* docs: rename change note to upstream PR number (4194)

towncrier's issue_format links to zarr-developers/zarr-python issues, so
247 (the fork PR number) would render a link to an unrelated old issue.

Assisted-by: ClaudeCode:claude-fable-5

* test: guard event-loop test against vacuity if the sync fast path stops triggering

The test asserts a negative (compute never ran on the loop thread). If a
refactor made the sync fast path stop triggering, the traced ChunkTransform
methods would never be called (the async fallback uses AsyncChunkTransform)
and the test would pass while guarding nothing. Assert the traced hooks
actually ran.

Assisted-by: ClaudeCode:claude-fable-5

… checking fill_value for byte-identicality, fixes zarr-developers#3054. (zarr-developers#4183)

* fix(ArraySpec): proper and robust equality semantics for ArraySpec by checking fill_value for byte-identicality, fixes zarr-developers#3054.

* fix: added extra case for unequal types

* Update tests/test_array_spec.py

* addressed reviewers comments

* test: add end-to-end regression test for structured-dtype fills in sharded arrays, and changelog entry

The new test exercises the sharding codec's chunk-spec caches with an
unhashable np.void fill value (zarr-developers#3054), which the ArraySpec test suite
only covers at the unit level.

Assisted-by: ClaudeCode:claude-fable-5

---------

Co-authored-by: Davis Bennett <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>

…zarr-developers#4200)

* fix: byte-order handling for structured dtypes in the bytes codec (zarr-developers#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* fix: make benchmark page-cache clearing opt-in, never prompt for sudo

clear_cache() in tests/benchmarks/test_e2e.py ran sudo unconditionally in
every benchmark's setup, so a plain `pytest` run blocked on a password
prompt (zarr-developers#4199). It is now a no-op unless ZARR_BENCHMARK_CLEAR_CACHE is
set, uses `sudo -n` so it can never block interactively, and the broken
Darwin invocation ("&&" passed as an argument to sync) is fixed. The
benchmark CI jobs set the variable to keep clearing caches.

Closes zarr-developers#4199

Assisted-by: ClaudeCode:claude-fable-5

…opers#4172)

* perf(indexing): sorted 1-D fast path

* clean up comments

* name changelog file

* rename changes file

* add guard for uint and add test case for it

* apply suggestions

* add test_coordinate_indexer_1d_last_chunk_boundary_does_not_overflow  and it's fix

* add path whenever the requests are sparse

* first == last edge case and do the cost check inside the active path not before (duh)

* update the changelog file

* rewording

---------

Co-authored-by: Davis Bennett <[email protected]>

…ith async-only inner codecs (zarr-developers#4179)

ShardingCodec structurally satisfies SupportsSyncCodec, but its sync
methods delegate to the configured inner and index codec chains — so a
shard whose inner or index chain contains a codec implementing only the
async codec interface passed the fused pipeline's sync gate and then
raised TypeError mid-IO in ChunkTransform construction.

Sync capability is now answered by _codec_supports_sync, which combines
the structural protocol check with a per-instance _sync_capable opt-out
(absent means capable). ShardingCodec reports False when any codec in
its inner or index chain is not sync-capable (recursively, so a nested
shard propagates its opt-out outward), which makes ChunkTransform
construction raise at pipeline evolve time and the pipeline decline the
sync fast path — such arrays route through the async paths, matching
BatchedCodecPipeline. Fully sync-capable chains keep the fast path.

Closes zarr-developers#4178

Assisted-by: ClaudeCode:claude-fable-5

…partial methods (zarr-developers#4201)

The partial dispatch in FusedCodecPipeline.read_sync/write_sync asserted
the private _decode_partial_sync/_encode_partial_sync methods, which only
ShardingCodec implements. A codec advertising the public partial mixins
(ArrayBytesCodecPartialDecodeMixin/-EncodeMixin) with only the documented
async partial methods died with a bare AssertionError — or, under
python -O, an AttributeError mid-IO. The asserts are now capability
gates: codecs without the sync partial methods take the full-chunk sync
path instead.

The related crash for sharded arrays with async-only inner codecs is
fixed separately in zarr-developers#4179.

Assisted-by: ClaudeCode:claude-fable-5

zarr-developers#4202)

FusedCodecPipeline.supports_partial_decode/supports_partial_encode passed
require_no_aa_bb=False, unlike BatchedCodecPipeline (True). With an outer
array-array or bytes-bytes codec around a sharding serializer (e.g.
compressors=[GzipCodec()], or filters=[TransposeCodec()]), the fused
pipeline's partial read/write branches called ShardingCodec's partial
sync methods directly on the raw stored value, skipping those outer
codecs entirely. That wrote non-conforming bytes for an outer BB codec
(unreadable by BatchedCodecPipeline or any conforming reader) and
silently produced wrong data for an outer AA codec.

Pass require_no_aa_bb=True in both fused properties so these chains fall
through to the full-chunk fused path instead, matching batched behavior.

Adds cross-pipeline parity coverage (full and partial read/write) for
sharding with an outer compressor and with an outer transpose filter, and
removes the "known limitation" exclusion that previously kept the
sharding+compressor case out of the nested-sharding parity matrix.

Assisted-by: ClaudeCode:claude-sonnet-5

…x density check (zarr-developers#4203)

The FusedCodecPipeline's vectorized whole-shard decode accepted any
indexer without `sel_shape` whose output shape matched the shard shape.
An OrthogonalIndexer from `arr[perm, :]` / `arr.oindex[...]` satisfies
that, so reordering or duplicating fancy-index reads on uncompressed,
crc-free sharded arrays silently returned the shard in natural order.
The bulk path now requires an identity full read: one whole-dimension,
step-1 slice per dimension (structural, not a BasicIndexer type check,
since `arr[:]` arrives as an OrthogonalIndexer).

Also:
- decline structured dtypes in the bulk path, which lacks the Struct
  byte-order branch of BytesCodec._decode_sync (latent until zarr-developers#3054 is
  fixed);
- `_ShardIndex.is_dense` now requires offsets to exactly tile the data
  section instead of merely being unique, so corrupt indexes with
  overlapping or out-of-range offsets (e.g. pointing into an
  index_location='start' index region) cannot be served as array data.

Assisted-by: ClaudeCode:claude-fable-5

…ile (zarr-developers#4208)

* docs(zarr-metadata): add API reference to the docs site

Add mkdocstrings pages for every public zarr_metadata module (model,
pydantic, v2, and v3 with its chunk_grid, chunk_key_encoding, codec, and
data_type subpackages) under a new zarr-metadata group in the API
Reference nav. griffe documents the package statically from
packages/zarr-metadata/src, so the docs build environment does not need
the package installed. Point the package's Documentation URL at the
rendered reference instead of the README.

Assisted-by: ClaudeCode:claude-fable-5

* chore(zarr-metadata): add justfile with package-scoped dev recipes

Recipes mirror the zarr-metadata CI jobs (pytest, ruff, pyright pinned to
the version CI uses, on CI's python) plus changelog-draft and docs-serve
conveniences. Recipes run from the package directory regardless of where
just is invoked, and remain reachable from the repo root as
'just packages/zarr-metadata/<recipe>'; a future root justfile can
namespace them with a 'mod' declaration.

Assisted-by: ClaudeCode:claude-fable-5

* chore(zarr-metadata): make docs-serve robust to a busy port

With no argument, docs-serve now binds port 8000 if free and otherwise
falls back to an ephemeral free port. An explicitly requested port is
used as-is so a conflict fails loudly.

Assisted-by: ClaudeCode:claude-fable-5

* chore(zarr-metadata): point docs-serve at the package docs, fix cleanup

Print the zarr-metadata API reference URL once the server accepts
connections, since mkdocs's own 'Serving on' line points at the
zarr-python site root. Run the server in its own process group so
stopping the recipe kills the whole uv->mkdocs tree instead of leaving
an orphaned server holding the port.

Assisted-by: ClaudeCode:claude-fable-5

* docs(zarr-metadata): make the package docs a standalone site

Move the zarr-metadata API reference out of the zarr-python site into a
self-contained mkdocs site under packages/zarr-metadata (own mkdocs.yml,
landing page, and .readthedocs.yaml for a dedicated RTD project), so the
package presents as a separate project with docs versioned by its own
zarr_metadata-v* release tags rather than zarr-python's. The zarr-python
API Reference nav now links out to the standalone site instead of
embedding the pages.

The package gains a pinned docs dependency group, a docs build job in
its CI workflow, and docs-check / docs-serve justfile recipes targeting
the package site.

Assisted-by: ClaudeCode:claude-fable-5

* update index.md

* ci(zarr-metadata): delegate workflow steps to the justfile

The workflow duplicated every command the justfile defines; jobs now run
'just test/lint/typecheck/docs-check' so the justfile is the single
source of truth for the package's verbs. CI keeps only its own concerns:
the python matrix sync for pytest, and uv caching. The pyright job's
python/sync steps are dropped because the typecheck recipe pins the
interpreter and pyright version itself.

Assisted-by: ClaudeCode:claude-fable-5

…pers#4051)

* docs: add a deprecation policy to the contributing guide

Closes zarr-developers#2924.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: neutral tone

* Add missing component from NEP23

* docs: align the deprecation policy with the versioning section and the code

Review follow-ups on the deprecation policy:

- Explain who each warning class reaches: DeprecationWarning is hidden
  outside __main__ and test runners, FutureWarning is shown to everyone,
  which a behavior change needs.
- Drop the "release first appeared in" requirement from the message; the
  release type is decided at release time under EffVer.
- Widen the recently-introduced-API exception to match the versioning
  section, and make the experimental promotion re-export explicitly exempt.
- Extend the minimum period to behavior changes, name raised exceptions as
  observable behavior, and note that a removal in a minor release is not
  SemVer (review threads).
- Rename the changelog fragment to the PR number.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <[email protected]>

* use more sentences

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Davis Vann Bennett <[email protected]>

Bumps [cairosvg](https://github.com/Kozea/CairoSVG) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/Kozea/CairoSVG/releases)
- [Changelog](https://github.com/Kozea/CairoSVG/blob/main/NEWS.rst)
- [Commits](Kozea/CairoSVG@2.9.0...2.9.1)

---
updated-dependencies:
- dependency-name: cairosvg
  dependency-version: 2.9.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.16.6 → v0.16.10](astral-sh/ruff-pre-commit@v0.16.6...v0.16.10)
- [github.com/DavidAnson/markdownlint-cli2: v0.23.2 → v0.23.3](DavidAnson/markdownlint-cli2@v0.23.2...v0.23.3)
- [github.com/numpy/numpydoc: v1.11.0rc0 → v1.11.0](numpy/numpydoc@v1.11.0rc0...v1.11.0)
- [github.com/zizmorcore/zizmor-pre-commit: v1.30.0 → v1.30.1](zizmorcore/zizmor-pre-commit@v1.30.0...v1.30.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

* fix: bound memory used by resize chunk cleanup

Co-Authored-By: OpenAI Codex <[email protected]>

* fix: delete resize chunks with a worker pool instead of lockstep batches

Batches sized to async.concurrency were awaited one after another, so each
group drained to zero in-flight deletions while waiting on its slowest
request. A fixed pool of workers pulling from the shared lazy coordinate
iterator keeps `concurrency` deletions in flight with the same memory bound.

Also rename the changelog fragment to the PR number.

Assisted-by: ClaudeCode:claude-opus-5-5

* fix: cancel remaining resize deletions after the first failure

With gather, a failed deletion raised out of resize while the other
workers kept deleting in the background, racing any retry. A TaskGroup
cancels them; the first exception is unwrapped from the ExceptionGroup
so resize keeps raising the store's own error.

Assisted-by: ClaudeCode:claude-opus-5-5

* fix(array): move the resize deletion pool into concurrent_foreach and size it to the job

Review follow-ups on the bounded resize cleanup:

- Add `zarr.core.common.concurrent_foreach`, a lazy bounded-concurrency
  sibling of `concurrent_map`: `limit` workers share one iterator, the
  first failure stops further items and cancels in-flight calls, and the
  failure is re-raised as itself. Simultaneous failures are kept as the
  `__cause__` group instead of being dropped, and `BaseExceptionGroup` is
  handled so non-Exception failures are unwrapped too.
- A call that fails before its first await no longer fans out to every
  worker, so a read-only store raises once.
- `_resize` computes the number of chunks to delete in O(ndim), skips the
  pool when it is zero (which subsumes the only-growing guard), and sizes
  the pool to `min(limit, n_delete)`.
- The changelog fragment states the new cancel-on-first-failure semantics.
- The pool-saturation test asserts the bound instead of the exact asyncio
  interleaving; helper semantics are tested in tests/test_common.py.

Assisted-by: ClaudeCode:claude-fable-5-1

---------

Co-authored-by: 2160039878-cyber <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Davis Vann Bennett <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>

Bumps [multidict](https://github.com/aio-libs/multidict) from 6.7.1 to 6.9.1.
- [Release notes](https://github.com/aio-libs/multidict/releases)
- [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst)
- [Commits](aio-libs/multidict@v6.7.1...v6.9.1)

---
updated-dependencies:
- dependency-name: multidict
  dependency-version: 6.9.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

Bumps the uv group with 1 update in the /packages/zarr-http-server directory: [multidict](https://github.com/aio-libs/multidict).


Updates `multidict` from 6.7.1 to 6.9.1
- [Release notes](https://github.com/aio-libs/multidict/releases)
- [Changelog](https://github.com/aio-libs/multidict/blob/master/CHANGES.rst)
- [Commits](aio-libs/multidict@v6.7.1...v6.9.1)

---
updated-dependencies:
- dependency-name: multidict
  dependency-version: 6.9.1
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.1.8 to 3.1.9.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.8...3.1.9)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-version: 3.1.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

…opers#4487)

Bumps the actions group with 1 update in the / directory: [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics).


Updates `github-community-projects/issue-metrics` from 5.0.2 to 5.0.3
- [Release notes](https://github.com/github-community-projects/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@a7dc2fb...6715286)

---
updated-dependencies:
- dependency-name: github-community-projects/issue-metrics
  dependency-version: 5.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

…#4483)

* fix: reset the codec-pipeline thread pool after fork

A forked child inherits codec_pipeline._pool pointing at threads that
do not exist in the child, plus the parent's idle-semaphore permits.
Submissions up to the permit count are consumed without spawning a
thread, so small FusedCodecPipeline reads hang forever.

Add an after_in_child hook that drops the inherited pool, size, and
lock. The lock is recreated because it may have been held by a thread
that no longer exists in the child.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: add changelog entry for zarr-developers#4483

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: barlowa124 <[email protected]>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <[email protected]>

* DOC: remove result="ansi" causing empty code blocks in user guide

Closes zarr-developers#3506.

Made-with: Cursor

* DOC: fix store migration guide: remove broken row, add LRUStoreCache note

Removes truncated table row and adds migration note for LRUStoreCache.
Closes zarr-developers#2699. Closes zarr-developers#2857.

Made-with: Cursor

* docs: point LRUStoreCache migration note at CacheStore

Move `LRUStoreCache` into the renamed-stores table, since it was never
deprecated in 2.x, and point it at the experimental
`zarr.experimental.cache_store.CacheStore`. Replace the nonexistent
`fsspec.implementations.caching` module with `fsspec.implementations.cached`,
and add the changelog fragment.

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Co-authored-by: Leo Ji <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>
Co-authored-by: Max Jones <[email protected]>

# Conflicts:
#	docs/user-guide/gpu.md
#	pyproject.toml
#	src/zarr/codecs/__init__.py
#	src/zarr/codecs/gpu.py
#	src/zarr/core/buffer/gpu.py
#	tests/test_codecs/test_codecs.py
#	tests/test_codecs/test_nvcomp.py

# Conflicts:
#	src/zarr/core/buffer/gpu.py
#	tests/test_api.py

…in CI

…of cast

Answers review question on why the cast was needed: get_codec_class returns
type[Codec]; the registry lookup exists so enable_gpu() can swap in
NvcompZstdCodec. A runtime isinstance check replaces the static cast.

…n gpu.NDBuffer (S101)

…Buffer object-dtype error with main

Per review discussion on zarr-developers#2863 (JeppeKlitgaard, d-v-b, TomAugspurger): the
extra installs CUDA-specific packages (cupy-cuda12x, nvidia-nvcomp-cu12), so
name it after CUDA rather than GPUs in general, leaving room for other
accelerator backends. The 'gpu' extra is removed (extras can't be deprecated).

enable_gpu() and zarr.buffer.gpu are unchanged: they predate this PR and are
public API.

nvCOMP only allows NO_COMPUTE_NO_VERIFY with a RAW bitstream, so mapping
checksum to ChecksumPolicy made checksum=True fail on both encode and
decode. The CUDA event is still recorded before synchronizing.