You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e02437c
Browse filesBrowse the repository at this point in the historyBrowse files
fix: set_range capability detection, bounds check, and managed path prefix
Layered on top of the per-key locking from #178. Addresses branch-review findings
not covered by that change:
- GpuMemoryStore must not satisfy SupportsSetRange: the inherited MemoryStore
implementation mutates a host copy of the GPU buffer and would silently lose the
write. MemoryStore/LocalStore now satisfy the protocol *structurally* (dropped the
nominal `SupportsSetRange` base), so GpuMemoryStore disclaims set_range/set_range_sync
by setting them to None and isinstance(gpu_store, SupportsSetRange) is now False.
- Uniform out-of-bounds handling: add _check_set_range_bounds (storage/_utils),
enforcing start + len(value) <= len(existing) in both MemoryStore and LocalStore so
a too-large write raises a clear ValueError instead of LocalStore silently extending
the file or MemoryStore raising an opaque numpy error. Matches zarrs, which validates
ranges rather than extending.
- ManagedMemoryStore.set_range/set_range_sync now apply the store's path prefix like
its other key-bearing methods, so a path-prefixed store targets the correct key.
- Expand the SupportsSetRange docstring with the caller-owns-consistency concurrency
contract and record the opt-in-protocol-vs-universal-method design decision; update
the changelog.
Adds out-of-bounds tests (MemoryStore, LocalStore), a GpuMemoryStore capability test,
and a path-prefixed ManagedMemoryStore set_range test. Verified on GPU hardware.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add protocols for stores that support byte-range-writes. This is necessary to support in-place writes of sharded arrays.
1
+
Add the `SupportsSetRange` protocol for stores that support writing to a byte range within an existing value, implemented by `LocalStore` and `MemoryStore`. This is necessary to support in-place writes of sharded arrays (e.g. writing a single subchunk without rewriting the entire shard).
2
+
3
+
Byte-range writes are exposed as an opt-in protocol rather than a method on the `Store` ABC. Only a few stores can perform them natively, and most cannot. A universal method with a read-modify-write fallback (as in the Rust `zarrs` crate) would let every store participate, but for the motivating use case that fallback would silently rewrite an entire shard, defeating the purpose. The opt-in protocol keeps the cost model honest and keeps `set_range` out of the signatures of stores that will never support it; any fallback strategy is left to the caller (the sharding codec). Stores satisfy the protocol structurally, so `GpuMemoryStore` (which has no use case for in-place GPU byte-range writes) disclaims it and is correctly reported as unsupported by `isinstance`.
4
+
5
+
It is entirely the caller's responsibility to ensure consistency: concurrent writes to overlapping ranges are order-dependent, `set_range` racing against `set`/`delete` is a race, and writes are not guaranteed to be atomic with respect to a process crash. A write that does not fit within the existing value raises `ValueError` consistently across implementations.
0 commit comments