Repository navigation
Conversation
NUMPY_TIME_UNITS gains "generic", NumPy's unit-less unit, which NumPy parses in brackets (<M8[generic]) though its own dtype.str omits it, so the v2 reader accepts the bracketed generic spellings zarr-python writes (<M8[generic], <m8[2generic]). The v3 unit tuple already carried generic and now reuses the shared tuple unchanged. A unit is still required: the unitless <M8 stays refused, as the v2 spec demands. Refs zarr-developers#4580 Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
…velopers#4521) * Remove zarr.tree and the Array.compressor properties Both were deprecated in 3.0.0 and were scheduled for removal in 3.2.0 by zarr-developers#3325. The rest of that PR shipped in 3.2.0 through zarr-developers#3900-zarr-developers#3903. Removed: - zarr.tree, zarr.api.synchronous.tree, zarr.api.asynchronous.tree. Use Group.tree / AsyncGroup.tree. - Array.compressor and AsyncArray.compressor. Use Array.compressors. The warning-only test for zarr.tree is gone, and the v2 compressor assertions in tests/test_v2.py now use `compressors`. The zarr.storage.default_compressor shim from the same zarr-developers#3325 list is left alone; zarr-developers#4505 removes it. --------- Co-authored-by: Davis Bennett <[email protected]>
…arr-developers#4504) * fix: reset the codec-pipeline thread pool after fork A forked child inherits codec_pipeline._pool pointing at threads that do not exist in the child, plus the parent's idle-semaphore permits. Submissions up to the permit count are consumed without spawning a thread, so small FusedCodecPipeline reads hang forever. Add an after_in_child hook that drops the inherited pool, size, and lock. The lock is recreated because it may have been held by a thread that no longer exists in the child. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: add changelog entry for zarr-developers#4483 Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(codec_pipeline): pin pool size in the fork regression test Move the zarr-developers#4478 regression test next to the other FusedCodecPipeline pool tests and pin `codec_pipeline.max_workers` to 4 through the shared `_FUSED_POOL_CONFIG`. With the default, a 1-CPU host resolves `max_workers` to 1, never creates the pool, and passes the test without exercising the bug. The test now asserts that the parent created the pool before forking, and it runs on macOS, where the fork start method is available. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: barlowa124 <[email protected]> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <[email protected]>
* feat: read only the needed rows of uncompressed chunks BytesCodec now implements partial decoding. An uncompressed chunk is stored in C order, so each row along its first axis is a contiguous run of bytes; for a selection that does not touch every row, the rows from the first to the last one it touches are fetched with a single range request and the selection is applied to them. The pipeline already uses partial decoding when the array-to-bytes codec supports it and there are no array-to-array or bytes-to-bytes codecs, so this applies to uncompressed arrays only. Reading 100 rows of a single-chunk 20 MB array now fetches the bytes of those rows instead of the whole chunk. Co-Authored-By: Claude Opus 5.5 <[email protected]> * Fix boolean masks and stores that ignore ranges in BytesCodec partial reads A boolean mask on the first axis, as from arr.oindex[mask], was trimmed to start at the first selected row but not to end at the last, so its length no longer matched the rows fetched and numpy raised an IndexError. The row window and the shifted selection are now computed together in _row_window, which trims the mask at both ends. This also gives mypy the narrowing it needs. FsspecStore over HTTP returns the whole object when a server ignores the Range header. The partial read then failed to reshape a chunk that the full read would have decoded, so a read that worked before this branch raised an error. A response longer than the requested range is now sliced to it. Co-Authored-By: Claude Opus 5.5 <[email protected]> * Add sync partial decode to BytesCodec and check the length of ranged reads Add _decode_partial_sync so FusedCodecPipeline reads only the needed rows of an uncompressed chunk, sharing the decode step with the async method. Decide what a store sent for a range request by its exact length: the requested range, the whole chunk, or everything from the range start. Any other length raises instead of returning wrong data. Run the partial read tests on both codec pipelines and add a changelog entry. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Davis Bennett <[email protected]>
…4522) * Deprecate the ignored Zarr-Python 2 keyword arguments `synchronizer`, `chunk_store`, `cache_metadata`, `cache_attrs`, `object_codec`, `read_only` and `meta_array` are accepted by `create`, `group` and `open_group` (and by everything that wraps them, such as `zeros`) but have never done anything in Zarr-Python 3. Passing one used to emit a `ZarrRuntimeWarning` saying "not yet implemented", which read as a promise to implement it. Emit a `ZarrDeprecationWarning` instead, saying the argument is ignored and will be removed in a release after April 2027, and say the same in the docstrings and the v3 migration guide. Calls that do not pass these arguments are unaffected. First step of the plan for zarr-developers#3976. --------- Co-authored-by: Davis Bennett <[email protected]>
…ers#4612) Python shows a DeprecationWarning by default only when it is attributed to __main__, so warnings that pointed at zarr internals were hidden from users running scripts: - `ArrayV2Metadata.chunk_grid` is a cached_property, which adds a frame; stacklevel 2 pointed at functools. Now 3. - Setting `zarr.storage.default_compressor` used stacklevel 1, which pointed at zarr itself. Now 2.
…velopers#4610) * feat(zarr-metadata)!: write store bytes through a JSONWriter `to_key_value` takes a writer instead of an `indent` keyword. A `JSONWriter` owns how a document becomes JSON bytes: it holds the settings (`indent`, `sort_keys`, `ensure_ascii`, `trailing_newline`) and its `encode` method does the writing, so the separators, the bare non-finite tokens and the surrogate escapes are no longer split off in a private helper. `to_key_value` also accepts a mapping of the settings, `JSONWriterSchema`, so callers need not import the class: `to_key_value({"indent": 2})`. The defaults are those of `json.dumps`, so a bare `to_key_value()` writes the same bytes as before. A setting `json.dumps` would misread raises, and so does a mapping key that is not a setting. Rebased onto main as one commit, replacing the JSONWriterConfig design. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragment for zarr-developers#4610 Assisted-by: ClaudeCode:claude-opus-5-5
…developers#4608) * docs(zarr-metadata): say that to_json returns arrays as tuples The guide said `to_json` returns plain containers, while another section said it returns tuples. Each model's `to_json` docstring now says that every JSON array is a tuple, as the package's JSON types spell one, so the result does not compare equal to `json.loads` output, and points to comparing models instead. Prompted by refkv's report. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): number the changelog fragment for #441 Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> * docs(zarr-metadata): number the changelog fragment for zarr-developers#4608 Assisted-by: ClaudeCode:claude-opus-5-5 --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
…initions (zarr-developers#4609) * feat(zarr-metadata): export base64_fill_value_rules for data type definitions Export the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it as `fill_value_rules` or yield from it. `bytes` and `variable_length_bytes` now share it. The parsing and validation guide shows a data type that defines `fill_value_rules`, built with `union`. Rebased onto main as one commit; the earlier history carried the merged ZARR_PYTHON_CONTEXT branch. Assisted-by: ClaudeCode:claude-opus-5-5 * docs(zarr-metadata): number the changelog fragment for zarr-developers#4609 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Owner
Author
|
🤖 AI text below 🤖 Closed as superseded: this branch merged upstream as zarr-developers#4614. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Accepted the generic time unit in Zarr format 2 time dtypes:
NUMPY_TIME_UNITSgains"generic", NumPy's unit-less unit, which NumPy parses in brackets (<M8[generic]) though its owndtype.stromits it, so the v2 reader accepts the bracketed generic spellings zarr-python writes —<M8[generic], and the scaled form<m8[2generic]. A unit is still required: the unitless<M8stays refused, as the v2 spec demands.The v3 unit tuple (
NUMPY_TIME_UNIT) appended"generic"to the shared tuple and now reuses it unchanged, so v3 behavior is identical. The canonical spelling preservesgenericas written, like every unit other than theμsalias.This is the zarr-metadata half of zarr-developers#4580; the zarr-python half (writing the bracketed form instead of the unitless typestr) is #421. Closing the seam for zarr-python's document model requires a zarr-metadata release carrying this change.
🤖 Generated with Claude Code