Skip to content

docs: split tutorial into multiple user guide sections - #2589

Merged
dstansby merged 24 commits into
mainfrom
docs/3.0-user-guide
Jan 3, 2025
Merged

dstansby merged 24 commits into
mainfrom
docs/3.0-user-guide

Conversation

@jhamman

@jhamman jhamman commented Dec 29, 2024 •

Copy link
Copy Markdown
Member

This PR updates Zarr's documentation to include a new multi-page user-guide, in-place of the prior single page tutorial.

Notes for reviewers:

  • This was split off from Doc updates for 3.0 release #2568
  • Most content remains unchanged from the tutorial, despite the large diff line count. Some new sections were added in 2568. The only net new section added was config.rst.
  • Examples now use the Ipython sphinx directive

closes #1767

@dstansby dstansby left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Liking the new sections, makes it much easier to read than the old tutorial 👍

A couple of major questions:

The "copying/migrating data" tutorial section wasn't migrated from the old tutorial, is this intentional?

For IPython, what was the motivation for switching to this from doctests? As it stands it has a few major drawbacks:

  1. Trying to copy/paste a block of code doesn't work, because it selects the Out[...] text too:
    Screenshot 2024-12-30 at 10 07 54
  2. The expected output isn't embedded any more. This gave us another layer of regression tests when we ran the tutorial as a doc test, and on top of making sure the code runs, made sure the output was as expected.
  3. I find having a line break between each line of code makes it much less readable - old vs new:
    Screenshot 2024-12-30 at 10 52 17
    Screenshot 2024-12-30 at 10 51 24

Is it possible to fix these with new IPython plugin?

Finally, what's the plan for all the TODOs (especially the numcodecs and Sharding bits, and configuring blosc)? To do them in another PR before 3.0? If so would be good to open issues for these so they're not lost and released as TODOs in 3.0

Comment thread docs/index.rst Resolved
Comment thread docs/index.rst Resolved
Comment thread docs/user-guide/arrays.rst Resolved
Comment thread docs/user-guide/arrays.rst Resolved
Comment thread docs/user-guide/arrays.rst Resolved
Comment thread docs/user-guide/performance.rst Resolved
Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Resolved
Comment thread docs/user-guide/index.rst Resolved
Comment thread docs/user-guide/config.rst Resolved
@jhamman

jhamman commented Jan 1, 2025

Copy link
Copy Markdown
Member Author

Thanks for the review @dstansby!

Liking the new sections, makes it much easier to read than the old tutorial 👍

A couple of major questions:

The "copying/migrating data" tutorial section wasn't migrated from the old tutorial, is this intentional?

We have not implemented copy, copy_all, and copy_store yet. I'll add these to the list of TODOs in the migration guide if they are not there already.

For IPython, what was the motivation for switching to this from doctests?

For one, we weren't using the doc tests! Beyond that, this isn't something I feel super strong about. We've had a lot of success using the Ipython directive in the Xarray project.

As it stands it has a few major drawbacks:

  1. Trying to copy/paste a block of code doesn't work, because it selects the Out[...] text too:

Not true if you use the copy button, example below:

z1[:] = 42

z1[0, :] = np.arange(10000)

z1[:, 0] = np.arange(10000)

I think there is probably a way to further configure the style.

  1. The expected output isn't embedded any more. This gave us another layer of regression tests when we ran the tutorial as a doc test, and on top of making sure the code runs, made sure the output was as expected.

We can insert assertions in hidden cells as desired. This would achieve the same thing.

  1. I find having a line break between each line of code makes it much less readable - old vs new:

Is it possible to fix these with new IPython plugin?

Likely yes. Can we iterate on the styling after this first PR goes in?

Finally, what's the plan for all the TODOs (especially the numcodecs and Sharding bits, and configuring blosc)? To do them in another PR before 3.0? If so would be good to open issues for these so they're not lost and released as TODOs in 3.0

I'm waiting #2463. I have a branch combining these but I'll need to start it over now that I've split up the docs refactor. So yes, let's do them in another PR.

@dstansby

dstansby commented Jan 1, 2025 •

Copy link
Copy Markdown
Contributor

Trying to copy/paste a block of code doesn't work, because it selects the Out[...] text too:

Not true if you use the copy button

This isn't a solution if you want to select two lines of code from a longer block.

This gave us another layer of regression tests when we ran the tutorial as a doc test, and on top of making sure the code runs, made sure the output was as expected.

We can insert assertions in hidden cells as desired. This would achieve the same thing.

But with a much higher maintenance burden I think? We would have to manually concoct a hidden assertion for every code block in this case, whereas with the doctests we can just generate all the output automatically.

So given there don't seem to be any benefits to using the IPython plugin, but there still seem to be drawbacks, I think we should keep existing code block style for now.

re. the TODOs, I'm a bit tired to do it now but tomorrow I can open issues to keep track of them (if you haven't already).

@jhamman

jhamman commented Jan 1, 2025 •

Copy link
Copy Markdown
Member Author

My view is that the most important thing is to have documentation that actually runs! I'm happy to hand this off to someone with sufficient experience with doctests (maybe @dstansby) but that isn't me. If no one is volunteering to do it right now, I'm going to press forward with this solution and we can revisit when someone else has time/energy for another doctest effort. I'll also note that these aren't mutually exclusive concepts (reading up on the ipython extension is probably worthwhile).

@dstansby

dstansby commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

In order to keep this moving, lets drop the switch to using the IPython plugin, since that shouldn't block a v3 release.

@dstansby dstansby left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking as request changes to revert the IPython change. @jhamman I'd be happy to change the code snippets in code back, but don't want to push to your PR without permission - let me know either way.

@jhamman

jhamman commented Jan 2, 2025 •

Copy link
Copy Markdown
Member Author

If you can update the code snippets today, feel free to push to this branch open a PR (this branch is in the zarr-python repo). I'm all checked in. Like I said above, I'm not planning on reverting the change myself though.

* docs: add new top level about page

* fixup

* fixup

* fixup
@dstansby

dstansby commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

👍 I can take a look. I think it's a reasonable ask to revert the change though, and not merge before it's reverted.

@dstansby dstansby mentioned this pull request Jan 2, 2025
* docs: add docs on extending zarr 3

* Apply suggestions from code review

Co-authored-by: David Stansby <[email protected]>

* move note up

* remove test.py (#2612)

* Note that whole directories can be deleted in LocalStore (#2606)

* fix: run-coverage command now tracks src directory (#2615)

* fix doc build

* Update docs/user-guide/extending.rst

---------

Co-authored-by: Norman Rzepka <[email protected]>
Co-authored-by: David Stansby <[email protected]>
Co-authored-by: Davis Bennett <[email protected]>
@normanrz

normanrz commented Jan 2, 2025 •

Copy link
Copy Markdown
Member

I think it's a reasonable ask to revert the change though, and not merge before it's reverted.

To be honest, I think it is more important to merge this PR quickly because of the dependent PRs. I feel this is blocking our progress right now. We can change away from IPython at any later point in time.

@d-v-b

d-v-b commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

I think it's a reasonable ask to revert the change though, and not merge before it's reverted.

To be honest, I think it is more important to merge this PR quickly because of the dependent PRs. I feel this is blocking our progress right now. We can change away from IPython at any later point in time.

+1 to this, lets get this PR in and sort out doctest / ipython later

@dcherian

dcherian commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

I concur. Working doc examples are very valuable!

@dstansby

dstansby commented Jan 2, 2025 •

Copy link
Copy Markdown
Contributor

I'm sorry, but I really want to insist we don't merge this with IPython in it, because it's a change that has demonstratable downsides. On a point of principle, just opening a large PR shouldn't come with an assumption that it's going to get merged without engaging with review. On a practical point, I spent about an hour sorting this out at #2623 today. So 🙏 can folks review and merge #2623 into this PR first.

@jhamman

jhamman commented Jan 2, 2025

Copy link
Copy Markdown
Member Author

Let's deescalate this a bit.

If we can't get the doctests pr passing with ci today, we can merge this as is (with ipython) and go back to doctests as soon as it's ready.

I'm hearing general agreement that this PR is directionally aligned with where we want to go. Let's continue to iterate but in a non blocking way.

@dstansby - I know this isn't your ideal but I think it's what is best for the project overall.

@dstansby

dstansby commented Jan 2, 2025 •

Copy link
Copy Markdown
Contributor

Okay, quick update, I'm trialling enabling doctests at #2626, which I think I have working. If it works, that PR should fail, and then I'll close and merge into #2623. Sorry it's a bit complex, but avoids me messing around with GH actions config in #2623 to get the tests running.

* Use doctests for arrays.rst

* Use doctests for attributes.rst

* Use doctests for config.rst

* Use doctests for consolidated metadata

* Use doctests for groups.rst

* Use doctests for preformance.rst

* Use doctests for storage.rst

* Remove ipython config for docs

* Fix performance doctest output

* Enable doctests
@dstansby
dstansby dismissed their stale review January 2, 2025 23:19

Changes addressed

@dstansby

dstansby commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

Quick update, I've:

  • added a new doctests CI run (was much simpler than running doctests in all the existing test runs)
  • added a new hatch environment that can be used to run the doctests (hatch run doctest:run) or automatically fix the output if it changes (hatch run doctest:fix)

Some of the doctests need fixing because of #2463 being merged, but hopefully the new CI job and hatch env make that a lot easier for someone else to do.

@dstansby dstansby left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doctests need fixing - otherwise I think this is 👍

@normanrz

normanrz commented Jan 3, 2025

Copy link
Copy Markdown
Member

Doctests need fixing - otherwise I think this is 👍

Working on it.

@normanrz
normanrz requested a review from dstansby January 3, 2025 14:15

@dstansby dstansby left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚢 - a couple of optional comments. I haven't reviewed any of the actual content this time.

Comment thread tests/test_tree.py

@pytest.mark.parametrize("root_name", [None, "root"])
def test_tree(root_name: Any) -> None:
os.environ["OVERRIDE_COLOR_SYSTEM"] = "truecolor"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of interest, why's this needed?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_tree.py expects the tree to be rendered with the control sequences for text formatting (e.g. bold font). However, by default rich doesn't include that when running within pytest, because it detects that it is not running in a real terminal as the output is captured by pytest. This overrides the behavior in rich. This issue only surfaced now, because I added rich to the test dependencies and therefore test_tree.py wasn't skipped anymore.
https://github.com/zarr-developers/zarr-python/blob/main/tests/test_tree.py#L31-L54

@dstansby
dstansby merged commit d6384f5 into main Jan 3, 2025
@dstansby
dstansby deleted the docs/3.0-user-guide branch January 3, 2025 14:31
@normanrz normanrz mentioned this pull request Jan 3, 2025
d-v-b added a commit that referenced this pull request Nov 12, 2026
* chore(deps-dev): bump the python-dependencies group across 1 directory with 13 updates

Bumps the python-dependencies group with 13 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [obstore](https://github.com/geospatial-jeff/pyasyncio-benchmark) | `0.12.0` | `0.12.1` |
| [coverage](https://github.com/coveragepy/coveragepy) | `7.16.2` | `7.17.2` |
| [pytest](https://github.com/pytest-dev/pytest) | `9.1.1` | `9.2.0` |
| [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) | `1.4.0` | `1.4.1.post1` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.170.5` | `6.170.14` |
| [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) | `5.3.0` | `5.3.1` |
| [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed) | `5.0.3` | `5.0.4` |
| [uv](https://github.com/astral-sh/uv) | `0.13.5` | `0.13.7` |
| [moto[s3,server]](https://github.com/getmoto/moto) | `5.2.4` | `5.2.5` |
| [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.7` | `9.8.0` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.5` | `1.12.6` |
| [ruff](https://github.com/astral-sh/ruff) | `0.17.1` | `0.17.3` |
| [mypy](https://github.com/python/mypy) | `2.4.0` | `2.4.1` |



Updates `obstore` from 0.12.0 to 0.12.1
- [Commits](https://github.com/geospatial-jeff/pyasyncio-benchmark/commits)

Updates `coverage` from 7.16.2 to 7.17.2
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.16.2...7.17.2)

Updates `pytest` from 9.1.1 to 9.2.0
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@9.1.1...9.2.0)

Updates `pytest-asyncio` from 1.4.0 to 1.4.1.post1
- [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases)
- [Commits](pytest-dev/pytest-asyncio@v1.4.0...v1.4.1.post1)

Updates `hypothesis` from 6.170.5 to 6.170.14
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.170.5...v6.170.14)

Updates `pytest-benchmark` from 5.3.0 to 5.3.1
- [Release notes](https://github.com/ionelmc/pytest-benchmark/releases)
- [Changelog](https://github.com/ionelmc/pytest-benchmark/blob/master/CHANGELOG.rst)
- [Commits](ionelmc/pytest-benchmark@v5.3.0...v5.3.1)

Updates `pytest-codspeed` from 5.0.3 to 5.0.4
- [Release notes](https://github.com/CodSpeedHQ/pytest-codspeed/releases)
- [Changelog](https://github.com/CodSpeedHQ/pytest-codspeed/blob/master/CHANGELOG.md)
- [Commits](CodSpeedHQ/pytest-codspeed@v5.0.3...v5.0.4)

Updates `uv` from 0.13.5 to 0.13.7
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](astral-sh/uv@0.13.5...0.13.7)

Updates `moto[s3,server]` from 5.2.4 to 5.2.5
- [Release notes](https://github.com/getmoto/moto/releases)
- [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md)
- [Commits](getmoto/moto@5.2.4...5.2.5)

Updates `mkdocs-material[imaging]` from 9.7.7 to 9.8.0
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.7.7...9.8.0)

Updates `markdown-exec[ansi]` from 1.12.5 to 1.12.6
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.5...1.12.6)

Updates `ruff` from 0.17.1 to 0.17.3
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.17.1...0.17.3)

Updates `mypy` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v2.4.0...v2.4.1)

---
updated-dependencies:
- dependency-name: obstore
  dependency-version: 0.12.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: coverage
  dependency-version: 7.17.2
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: pytest
  dependency-version: 9.2.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: pytest-asyncio
  dependency-version: 1.4.1.post1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.170.14
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytest-benchmark
  dependency-version: 5.3.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: pytest-codspeed
  dependency-version: 5.0.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.13.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: moto[s3,server]
  dependency-version: 5.2.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocs-material[imaging]
  dependency-version: 9.8.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.12.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.17.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mypy
  dependency-version: 2.4.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>

* test: keep the suite running with pytest 9.2 and hypothesis 6.170.7+

pytest 9.2 applies the configured filterwarnings while importing plugins
(pytest#12697), so the "error" filter turns the ast.Num DeprecationWarning
that astor (a dependency of pytest-accept) triggers at import on Python
< 3.14 into a startup failure (exit code 3). Ignore that one astor-scoped
deprecation, which pytest <= 9.1 never routed through the filters.

pytest 9.2 also requires packaging>=24, which conflicts with the min_deps
pin packaging==22.* (the declared runtime floor packaging>=22.0). Keep that
floor tested by overriding pytest to <9.2 in the min_deps env only, via
UV_OVERRIDE and ci/min_deps-overrides.txt; every other env uses pytest 9.2.0.

hypothesis 6.170.7 annotates run_state_machine_as_test as taking
type[RuleBasedStateMachine] (its docstring still accepts any zero-arg
factory), so mypy now reports arg-type instead of no-untyped-call at the
four call sites that pass a factory function. Update the ignore codes;
no test logic changes.

* test: drop the unused pytest-accept plugin

pytest-accept was added in #2589 for a `hatch run doctest:fix` script that
ran `pytest --accept` over the RST doctests. That script and those doctests
are gone, and nothing in the justfile or CI passes `--accept`. The plugin's
only remaining effect was importing astor (last released in 2019) at every
pytest startup, whose `ast.Num` lookup pytest 9.2 turns into a startup error
under our "error" warning filter. Removing the plugin removes astor, so the
astor-scoped filter added for pytest 9.2 is no longer needed.

Assisted-by: ClaudeCode:claude-opus-5-5

* build(deps): require packaging>=24.0

pytest 9.2 requires packaging>=24, so the min_deps env could no longer pin
packaging==22.*. Instead of holding min_deps on pytest<9.2 through a
UV_OVERRIDE file, raise the runtime floor to packaging>=24.0 (released
March 2024) and test that floor in min_deps. zarr uses only
packaging.version.Version and parse, which behave the same on 22 and 24.
min_deps now runs the same pytest as every other env, and the override
file and env var are gone.

Assisted-by: ClaudeCode:claude-opus-5-5

* docs: add a changelog entry for the packaging>=24.0 floor

Assisted-by: ClaudeCode:claude-opus-5-5

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: invisiblefix-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements to the documentation

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[v3] Documentation: update tutorial docs

5 participants