Repository navigation
docs: split tutorial into multiple user guide sections - #2589
Conversation
dstansby
left a comment
There was a problem hiding this comment.
Liking the new sections, makes it much easier to read than the old tutorial 👍
A couple of major questions:
The "copying/migrating data" tutorial section wasn't migrated from the old tutorial, is this intentional?
For IPython, what was the motivation for switching to this from doctests? As it stands it has a few major drawbacks:
- Trying to copy/paste a block of code doesn't work, because it selects the
Out[...]text too:

- The expected output isn't embedded any more. This gave us another layer of regression tests when we ran the tutorial as a doc test, and on top of making sure the code runs, made sure the output was as expected.
- I find having a line break between each line of code makes it much less readable - old vs new:


Is it possible to fix these with new IPython plugin?
Finally, what's the plan for all the TODOs (especially the numcodecs and Sharding bits, and configuring blosc)? To do them in another PR before 3.0? If so would be good to open issues for these so they're not lost and released as TODOs in 3.0
Co-authored-by: David Stansby <[email protected]>
|
Thanks for the review @dstansby!
We have not implemented
For one, we weren't using the doc tests! Beyond that, this isn't something I feel super strong about. We've had a lot of success using the Ipython directive in the Xarray project. As it stands it has a few major drawbacks:
Not true if you use the copy button, example below: z1[:] = 42
z1[0, :] = np.arange(10000)
z1[:, 0] = np.arange(10000)I think there is probably a way to further configure the style.
We can insert assertions in hidden cells as desired. This would achieve the same thing.
Likely yes. Can we iterate on the styling after this first PR goes in?
I'm waiting #2463. I have a branch combining these but I'll need to start it over now that I've split up the docs refactor. So yes, let's do them in another PR. |
This isn't a solution if you want to select two lines of code from a longer block.
But with a much higher maintenance burden I think? We would have to manually concoct a hidden assertion for every code block in this case, whereas with the doctests we can just generate all the output automatically. So given there don't seem to be any benefits to using the IPython plugin, but there still seem to be drawbacks, I think we should keep existing code block style for now. re. the TODOs, I'm a bit tired to do it now but tomorrow I can open issues to keep track of them (if you haven't already). |
|
My view is that the most important thing is to have documentation that actually runs! I'm happy to hand this off to someone with sufficient experience with doctests (maybe @dstansby) but that isn't me. If no one is volunteering to do it right now, I'm going to press forward with this solution and we can revisit when someone else has time/energy for another doctest effort. I'll also note that these aren't mutually exclusive concepts (reading up on the ipython extension is probably worthwhile). |
|
In order to keep this moving, lets drop the switch to using the IPython plugin, since that shouldn't block a v3 release. |
|
If you can update the code snippets today, feel free to |
* docs: add new top level about page * fixup * fixup * fixup
|
👍 I can take a look. I think it's a reasonable ask to revert the change though, and not merge before it's reverted. |
* docs: add docs on extending zarr 3 * Apply suggestions from code review Co-authored-by: David Stansby <[email protected]> * move note up * remove test.py (#2612) * Note that whole directories can be deleted in LocalStore (#2606) * fix: run-coverage command now tracks src directory (#2615) * fix doc build * Update docs/user-guide/extending.rst --------- Co-authored-by: Norman Rzepka <[email protected]> Co-authored-by: David Stansby <[email protected]> Co-authored-by: Davis Bennett <[email protected]>
To be honest, I think it is more important to merge this PR quickly because of the dependent PRs. I feel this is blocking our progress right now. We can change away from IPython at any later point in time. |
+1 to this, lets get this PR in and sort out doctest / ipython later |
|
I concur. Working doc examples are very valuable! |
|
I'm sorry, but I really want to insist we don't merge this with IPython in it, because it's a change that has demonstratable downsides. On a point of principle, just opening a large PR shouldn't come with an assumption that it's going to get merged without engaging with review. On a practical point, I spent about an hour sorting this out at #2623 today. So 🙏 can folks review and merge #2623 into this PR first. |
|
Let's deescalate this a bit. If we can't get the doctests pr passing with ci today, we can merge this as is (with ipython) and go back to doctests as soon as it's ready. I'm hearing general agreement that this PR is directionally aligned with where we want to go. Let's continue to iterate but in a non blocking way. @dstansby - I know this isn't your ideal but I think it's what is best for the project overall. |
* Use doctests for arrays.rst * Use doctests for attributes.rst * Use doctests for config.rst * Use doctests for consolidated metadata * Use doctests for groups.rst * Use doctests for preformance.rst * Use doctests for storage.rst * Remove ipython config for docs * Fix performance doctest output * Enable doctests
|
Quick update, I've:
Some of the doctests need fixing because of #2463 being merged, but hopefully the new CI job and hatch env make that a lot easier for someone else to do. |
dstansby
left a comment
There was a problem hiding this comment.
Doctests need fixing - otherwise I think this is 👍
Working on it. |
dstansby
left a comment
There was a problem hiding this comment.
LGTM 🚢 - a couple of optional comments. I haven't reviewed any of the actual content this time.
|
|
||
| @pytest.mark.parametrize("root_name", [None, "root"]) | ||
| def test_tree(root_name: Any) -> None: | ||
| os.environ["OVERRIDE_COLOR_SYSTEM"] = "truecolor" |
There was a problem hiding this comment.
Just out of interest, why's this needed?
There was a problem hiding this comment.
test_tree.py expects the tree to be rendered with the control sequences for text formatting (e.g. bold font). However, by default rich doesn't include that when running within pytest, because it detects that it is not running in a real terminal as the output is captured by pytest. This overrides the behavior in rich. This issue only surfaced now, because I added rich to the test dependencies and therefore test_tree.py wasn't skipped anymore.
https://github.com/zarr-developers/zarr-python/blob/main/tests/test_tree.py#L31-L54
* chore(deps-dev): bump the python-dependencies group across 1 directory with 13 updates Bumps the python-dependencies group with 13 updates in the / directory: | Package | From | To | | --- | --- | --- | | [obstore](https://github.com/geospatial-jeff/pyasyncio-benchmark) | `0.12.0` | `0.12.1` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.16.2` | `7.17.2` | | [pytest](https://github.com/pytest-dev/pytest) | `9.1.1` | `9.2.0` | | [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) | `1.4.0` | `1.4.1.post1` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.170.5` | `6.170.14` | | [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) | `5.3.0` | `5.3.1` | | [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed) | `5.0.3` | `5.0.4` | | [uv](https://github.com/astral-sh/uv) | `0.13.5` | `0.13.7` | | [moto[s3,server]](https://github.com/getmoto/moto) | `5.2.4` | `5.2.5` | | [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.7` | `9.8.0` | | [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.5` | `1.12.6` | | [ruff](https://github.com/astral-sh/ruff) | `0.17.1` | `0.17.3` | | [mypy](https://github.com/python/mypy) | `2.4.0` | `2.4.1` | Updates `obstore` from 0.12.0 to 0.12.1 - [Commits](https://github.com/geospatial-jeff/pyasyncio-benchmark/commits) Updates `coverage` from 7.16.2 to 7.17.2 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.16.2...7.17.2) Updates `pytest` from 9.1.1 to 9.2.0 - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.1.1...9.2.0) Updates `pytest-asyncio` from 1.4.0 to 1.4.1.post1 - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Commits](pytest-dev/pytest-asyncio@v1.4.0...v1.4.1.post1) Updates `hypothesis` from 6.170.5 to 6.170.14 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.170.5...v6.170.14) Updates `pytest-benchmark` from 5.3.0 to 5.3.1 - [Release notes](https://github.com/ionelmc/pytest-benchmark/releases) - [Changelog](https://github.com/ionelmc/pytest-benchmark/blob/master/CHANGELOG.rst) - [Commits](ionelmc/pytest-benchmark@v5.3.0...v5.3.1) Updates `pytest-codspeed` from 5.0.3 to 5.0.4 - [Release notes](https://github.com/CodSpeedHQ/pytest-codspeed/releases) - [Changelog](https://github.com/CodSpeedHQ/pytest-codspeed/blob/master/CHANGELOG.md) - [Commits](CodSpeedHQ/pytest-codspeed@v5.0.3...v5.0.4) Updates `uv` from 0.13.5 to 0.13.7 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md) - [Commits](astral-sh/uv@0.13.5...0.13.7) Updates `moto[s3,server]` from 5.2.4 to 5.2.5 - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.2.4...5.2.5) Updates `mkdocs-material[imaging]` from 9.7.7 to 9.8.0 - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](squidfunk/mkdocs-material@9.7.7...9.8.0) Updates `markdown-exec[ansi]` from 1.12.5 to 1.12.6 - [Release notes](https://github.com/pawamoy/markdown-exec/releases) - [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md) - [Commits](pawamoy/markdown-exec@1.12.5...1.12.6) Updates `ruff` from 0.17.1 to 0.17.3 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.17.1...0.17.3) Updates `mypy` from 2.4.0 to 2.4.1 - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v2.4.0...v2.4.1) --- updated-dependencies: - dependency-name: obstore dependency-version: 0.12.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: coverage dependency-version: 7.17.2 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: pytest dependency-version: 9.2.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: pytest-asyncio dependency-version: 1.4.1.post1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.170.14 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: pytest-benchmark dependency-version: 5.3.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: pytest-codspeed dependency-version: 5.0.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.13.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: moto[s3,server] dependency-version: 5.2.5 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocs-material[imaging] dependency-version: 9.8.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: markdown-exec[ansi] dependency-version: 1.12.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.17.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mypy dependency-version: 2.4.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> * test: keep the suite running with pytest 9.2 and hypothesis 6.170.7+ pytest 9.2 applies the configured filterwarnings while importing plugins (pytest#12697), so the "error" filter turns the ast.Num DeprecationWarning that astor (a dependency of pytest-accept) triggers at import on Python < 3.14 into a startup failure (exit code 3). Ignore that one astor-scoped deprecation, which pytest <= 9.1 never routed through the filters. pytest 9.2 also requires packaging>=24, which conflicts with the min_deps pin packaging==22.* (the declared runtime floor packaging>=22.0). Keep that floor tested by overriding pytest to <9.2 in the min_deps env only, via UV_OVERRIDE and ci/min_deps-overrides.txt; every other env uses pytest 9.2.0. hypothesis 6.170.7 annotates run_state_machine_as_test as taking type[RuleBasedStateMachine] (its docstring still accepts any zero-arg factory), so mypy now reports arg-type instead of no-untyped-call at the four call sites that pass a factory function. Update the ignore codes; no test logic changes. * test: drop the unused pytest-accept plugin pytest-accept was added in #2589 for a `hatch run doctest:fix` script that ran `pytest --accept` over the RST doctests. That script and those doctests are gone, and nothing in the justfile or CI passes `--accept`. The plugin's only remaining effect was importing astor (last released in 2019) at every pytest startup, whose `ast.Num` lookup pytest 9.2 turns into a startup error under our "error" warning filter. Removing the plugin removes astor, so the astor-scoped filter added for pytest 9.2 is no longer needed. Assisted-by: ClaudeCode:claude-opus-5-5 * build(deps): require packaging>=24.0 pytest 9.2 requires packaging>=24, so the min_deps env could no longer pin packaging==22.*. Instead of holding min_deps on pytest<9.2 through a UV_OVERRIDE file, raise the runtime floor to packaging>=24.0 (released March 2024) and test that floor in min_deps. zarr uses only packaging.version.Version and parse, which behave the same on 22 and 24. min_deps now runs the same pytest as every other env, and the override file and env var are gone. Assisted-by: ClaudeCode:claude-opus-5-5 * docs: add a changelog entry for the packaging>=24.0 floor Assisted-by: ClaudeCode:claude-opus-5-5 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: invisiblefix-bot <[email protected]>
This PR updates Zarr's documentation to include a new multi-page user-guide, in-place of the prior single page tutorial.
Notes for reviewers:
closes #1767