Skip to content

fix(zarr-metadata): accept the generic time unit in v2 time dtypes - #422

Closed
d-v-b wants to merge 15 commits into
mainfrom
fix/zarr-metadata-generic-time-unit
Closed

d-v-b wants to merge 15 commits into
mainfrom
fix/zarr-metadata-generic-time-unit

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Accepted the generic time unit in Zarr format 2 time dtypes: NUMPY_TIME_UNITS gains "generic", NumPy's unit-less unit, which NumPy parses in brackets (<M8[generic]) though its own dtype.str omits it, so the v2 reader accepts the bracketed generic spellings zarr-python writes — <M8[generic], and the scaled form <m8[2generic]. A unit is still required: the unitless <M8 stays refused, as the v2 spec demands.

The v3 unit tuple (NUMPY_TIME_UNIT) appended "generic" to the shared tuple and now reuses it unchanged, so v3 behavior is identical. The canonical spelling preserves generic as written, like every unit other than the μs alias.

This is the zarr-metadata half of zarr-developers#4580; the zarr-python half (writing the bracketed form instead of the unitless typestr) is #421. Closing the seam for zarr-python's document model requires a zarr-metadata release carrying this change.

🤖 Generated with Claude Code

d-v-b added 2 commits October 29, 2026 15:17
NUMPY_TIME_UNITS gains "generic", NumPy's unit-less unit, which NumPy
parses in brackets (<M8[generic]) though its own dtype.str omits it, so
the v2 reader accepts the bracketed generic spellings zarr-python
writes (<M8[generic], <m8[2generic]). The v3 unit tuple already carried
generic and now reuses the shared tuple unchanged. A unit is still
required: the unitless <M8 stays refused, as the v2 spec demands.

Refs zarr-developers#4580

Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
maxrjones and others added 8 commits November 9, 2026 09:13
…velopers#4521)

* Remove zarr.tree and the Array.compressor properties

Both were deprecated in 3.0.0 and were scheduled for removal in 3.2.0
by zarr-developers#3325. The rest of that PR shipped in 3.2.0 through zarr-developers#3900-zarr-developers#3903.

Removed:
- zarr.tree, zarr.api.synchronous.tree, zarr.api.asynchronous.tree.
  Use Group.tree / AsyncGroup.tree.
- Array.compressor and AsyncArray.compressor. Use Array.compressors.

The warning-only test for zarr.tree is gone, and the v2 compressor
assertions in tests/test_v2.py now use `compressors`.

The zarr.storage.default_compressor shim from the same zarr-developers#3325 list is
left alone; zarr-developers#4505 removes it.

---------

Co-authored-by: Davis Bennett <[email protected]>
…arr-developers#4504)

* fix: reset the codec-pipeline thread pool after fork

A forked child inherits codec_pipeline._pool pointing at threads that
do not exist in the child, plus the parent's idle-semaphore permits.
Submissions up to the permit count are consumed without spawning a
thread, so small FusedCodecPipeline reads hang forever.

Add an after_in_child hook that drops the inherited pool, size, and
lock. The lock is recreated because it may have been held by a thread
that no longer exists in the child.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* docs: add changelog entry for zarr-developers#4483

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(codec_pipeline): pin pool size in the fork regression test

Move the zarr-developers#4478 regression test next to the other FusedCodecPipeline
pool tests and pin `codec_pipeline.max_workers` to 4 through the shared
`_FUSED_POOL_CONFIG`. With the default, a 1-CPU host resolves
`max_workers` to 1, never creates the pool, and passes the test without
exercising the bug. The test now asserts that the parent created the
pool before forking, and it runs on macOS, where the fork start method
is available.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: barlowa124 <[email protected]>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <[email protected]>
* feat: read only the needed rows of uncompressed chunks

BytesCodec now implements partial decoding. An uncompressed chunk is
stored in C order, so each row along its first axis is a contiguous run
of bytes; for a selection that does not touch every row, the rows from
the first to the last one it touches are fetched with a single range
request and the selection is applied to them. The pipeline already
uses partial decoding when the array-to-bytes codec supports it and
there are no array-to-array or bytes-to-bytes codecs, so this applies to
uncompressed arrays only. Reading 100 rows of a single-chunk 20 MB
array now fetches the bytes of those rows instead of the whole chunk.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Fix boolean masks and stores that ignore ranges in BytesCodec partial reads

A boolean mask on the first axis, as from arr.oindex[mask], was trimmed to
start at the first selected row but not to end at the last, so its length
no longer matched the rows fetched and numpy raised an IndexError. The row
window and the shifted selection are now computed together in _row_window,
which trims the mask at both ends. This also gives mypy the narrowing it
needs.

FsspecStore over HTTP returns the whole object when a server ignores the
Range header. The partial read then failed to reshape a chunk that the full
read would have decoded, so a read that worked before this branch raised an
error. A response longer than the requested range is now sliced to it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Add sync partial decode to BytesCodec and check the length of ranged reads

Add _decode_partial_sync so FusedCodecPipeline reads only the needed rows
of an uncompressed chunk, sharing the decode step with the async method.

Decide what a store sent for a range request by its exact length: the
requested range, the whole chunk, or everything from the range start.
Any other length raises instead of returning wrong data.

Run the partial read tests on both codec pipelines and add a changelog
entry.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>
…4522)

* Deprecate the ignored Zarr-Python 2 keyword arguments

`synchronizer`, `chunk_store`, `cache_metadata`, `cache_attrs`,
`object_codec`, `read_only` and `meta_array` are accepted by `create`,
`group` and `open_group` (and by everything that wraps them, such as
`zeros`) but have never done anything in Zarr-Python 3. Passing one
used to emit a `ZarrRuntimeWarning` saying "not yet implemented", which
read as a promise to implement it.

Emit a `ZarrDeprecationWarning` instead, saying the argument is ignored
and will be removed in a release after April 2027, and say the same in
the docstrings and the v3 migration guide. Calls that do not pass these
arguments are unaffected.

First step of the plan for zarr-developers#3976.

---------

Co-authored-by: Davis Bennett <[email protected]>
…ers#4612)

Python shows a DeprecationWarning by default only when it is attributed
to __main__, so warnings that pointed at zarr internals were hidden
from users running scripts:

- `ArrayV2Metadata.chunk_grid` is a cached_property, which adds a frame;
  stacklevel 2 pointed at functools. Now 3.
- Setting `zarr.storage.default_compressor` used stacklevel 1, which
  pointed at zarr itself. Now 2.
…velopers#4610)

* feat(zarr-metadata)!: write store bytes through a JSONWriter

`to_key_value` takes a writer instead of an `indent` keyword. A
`JSONWriter` owns how a document becomes JSON bytes: it holds the
settings (`indent`, `sort_keys`, `ensure_ascii`, `trailing_newline`)
and its `encode` method does the writing, so the separators, the bare
non-finite tokens and the surrogate escapes are no longer split off in
a private helper. `to_key_value` also accepts a mapping of the settings,
`JSONWriterSchema`, so callers need not import the class:
`to_key_value({"indent": 2})`.

The defaults are those of `json.dumps`, so a bare `to_key_value()` writes
the same bytes as before. A setting `json.dumps` would misread raises,
and so does a mapping key that is not a setting.

Rebased onto main as one commit, replacing the JSONWriterConfig design.

Assisted-by: ClaudeCode:claude-opus-5-5

* docs(zarr-metadata): number the changelog fragment for zarr-developers#4610

Assisted-by: ClaudeCode:claude-opus-5-5
…developers#4608)

* docs(zarr-metadata): say that to_json returns arrays as tuples

The guide said `to_json` returns plain containers, while another section
said it returns tuples. Each model's `to_json` docstring now says that
every JSON array is a tuple, as the package's JSON types spell one, so
the result does not compare equal to `json.loads` output, and points to
comparing models instead. Prompted by refkv's report.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(zarr-metadata): number the changelog fragment for #441

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(zarr-metadata): number the changelog fragment for zarr-developers#4608

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
…initions (zarr-developers#4609)

* feat(zarr-metadata): export base64_fill_value_rules for data type definitions

Export the rule that rejects a fill value string that is not
standard-alphabet base64, so a data type defined outside the package can
pass it as `fill_value_rules` or yield from it. `bytes` and
`variable_length_bytes` now share it. The parsing and validation guide
shows a data type that defines `fill_value_rules`, built with `union`.

Rebased onto main as one commit; the earlier history carried the merged
ZARR_PYTHON_CONTEXT branch.

Assisted-by: ClaudeCode:claude-opus-5-5

* docs(zarr-metadata): number the changelog fragment for zarr-developers#4609

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b

d-v-b commented Nov 11, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Closed as superseded: this branch merged upstream as zarr-developers#4614.

@d-v-b d-v-b closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants