Skip to content

Commit 0b2eda4

Browse files
d-v-bclaude
andauthored
docs: test more code snippets in docs (#4018)
* docs: spec for closing the silently-unexecuted docs-block gap (#4016) Design for fixing issue #4016 (invalid create_array(mode="w") in docs) and preventing recurrence: per-case remediation of the 12 non-executed python blocks (S3 via moto, config blocks via exec="true", GPU via the gpu marker, explicit opt-out for non-Python blocks) plus a guard test asserting every docs python block is either executed or explicitly opted out with a reason. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: unify docs-block marker model (s3 + gpu) in #4016 spec Replace the bespoke hidden-moto-setup-block approach for the S3 example with a marker-bound model: a block declares markers="s3"/"gpu" on the fence, and the harness binds each marker to the infra/env it needs (s3 -> moto fixture in the default doctest env; gpu -> gputest env via pytest -m gpu). Symmetric declaration; the asymmetry is only in what each marker resolves to. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: link spec to upstream issue #4017 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: implementation plan for docs-block validation (#4016, #4017) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: spike s3 default-endpoint mechanism for docs (no storage_options) Result: A-env-var A bare zarr.create_array("s3://bucket/key", ...) with NO storage_options reaches a moto server when AWS_ENDPOINT_URL is set process-wide. s3fs/ aiobotocore honor the env var, so the visible docs block can stay clean. Caveat: moto[s3,server] currently lives only in the 'remote-tests' dependency group, not 'test'; the doctest hatch env (dependency-groups= ['test']) does NOT have moto installed. The downstream real-fixture task must add moto[s3,server] (and requests) to the doctest env extras. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: register s3 pytest marker * test: parse markers= on docs blocks and add moto s3 fixture binding Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: fix invalid s3 create_array example and run it against moto (#4016) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: execute config-setting examples in performance.md and arrays.md Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: make cli zarr.open example runnable against a local store * docs: execute gpu example under the gpu marker Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: fix exec=on typo and explicitly opt out non-runnable blocks Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: make dask performance example runnable (or opt out if dask absent) * docs: record plan corrections from execution (spike result, gpu marker mechanism) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: guard that every docs python block is executed or opted out (#4017) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: separate `test` flag from `exec` so infra-bound examples don't break the build markdown-exec's `exec="true"` means "run at build to render output"; build runners have no GPU/cupy and no moto server, so tagging the GPU/S3 examples exec="true" made `mkdocs build --strict` abort. Introduce a separate `test="true"` flag that our tests/test_docs.py harness keys on (markdown-exec ignores it): a block is validated if exec="true" OR test="true". The GPU and S3 examples become test="true" (+markers) and are no longer run at build. Also: a test="true"-only python fence placed before an exec="true" block of the same page disrupts markdown-exec's build execution of the later block (the quickstart ZipStore example failed with FileNotFoundError). Move the S3 example to the end of quick-start.md so no shared-session exec block follows it; document the constraint in the guard docstring and the design spec. Verified: full docs test suite green (57 passed, 2 skipped), `mkdocs build --strict` exits 0, prek (ruff/mypy/...) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: add news fragment for docs-block validation (#4016, #4017) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: harden docs_s3_backend teardown and make cli example idempotent Address roborev branch-review findings (job 186): - Medium: moto-api reset POST ran before server.stop()/env restore in the finally block; if it raised, the fixed-port server thread leaked and AWS_ENDPOINT_URL was left dangling. Nest the reset in its own try/finally so server.stop() and env restoration always run. - Low: f"{S3_ENDPOINT}/moto-api/reset" double-slashed (constant ends in /); drop the extra slash. - Low: AWS_SECRET_ACCESS_KEY/AWS_ACCESS_KEY_ID were setdefault'd but never restored; save/restore all three mutated env vars uniformly. - Low: cli.md create_array lacked overwrite=True, non-idempotent across local runs; add it. Verified: full docs suite green (57 passed, 2 skipped), s3+cli pass on repeated runs, mkdocs build --strict exits 0, prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: actually run the gpu docs example on GPU; align collector/guard scope Address roborev branch-review findings (job 188): - Medium: the gpu docs example ran in NO environment. The gputest env lacks pytest-examples, so test_docs.py's module-level importorskip("pytest_examples") skipped the whole module under `pytest -m gpu` -- the gpu case was never collected even on GPU hardware, yet the guard reported it "validated" via test="true". Add pytest-examples to the gputest env; confirmed gpu-demo now collects under `hatch -e gputest run pytest -m gpu --co`. - Low: _session_params (collection) didn't exclude docs/superpowers/ while the guard did -- an asymmetry that could run a cache-doc block as a real test without the guard flagging it. Extract a shared _is_published_docs() helper used by both, so collection and guard agree on scope. Verified: doctest suite green (57 passed, 2 skipped), gpu-demo collectable in gputest, mkdocs build --strict exits 0, prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: enforce test-only block placement; drop redundant marker round-trip Address roborev branch-review findings (job 190): - Low: the "test=true-only block must not precede an exec=true block on the same page" constraint was documented but unenforced (only caught by mkdocs --strict). Add test_test_only_blocks_come_last, which fails fast/locally when a test-only block has a smaller start_line than a later same-file exec=true block. Negative- checked: it catches an exec block added after the gpu test-only block. - Low: _markers_for built pytest.MarkDecorator objects that _session_params immediately reduced to .name and rebuilt; replace with _marker_names returning raw strings, building decorators once at param time. Verified: doctest suite green (58 passed, 2 skipped), placement test passes and fails on a planted violation, prek clean, mkdocs build --strict exits 0. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * ci: make docs build strict; correct placement-hazard scope Address roborev branch-review finding (job 192): the placement guard's docstring claimed any non-exec="true" python fence before an exec="true" block breaks the build, but the guard only checked test="true" blocks, leaving exec="false"-before- exec="true" arrangements (data_types.md, performance.md) unguarded. Investigation (experiments + markdown-exec's SuperFences integration) established the real mechanism: a non-executed python fence (test="true" OR exec="false") before an exec="true" block disrupts build-time execution of a later *state-dependent* block (needs a cross-block dependency to surface, which is why the standalone exec="true" blocks under those exec="false" opt-outs build fine). CI ran non-strict `mkdocs build`, so such a failure would have merged as a silent warning. - ci/docs.yml: `mkdocs build` -> `mkdocs build --strict` so any build-time exec failure (including the exec="false" case) fails CI authoritatively. Verified: a clean build currently emits zero warnings, so --strict passes today. - Narrow the placement guard's docstring and the spec to the real (state-dependent) mechanism, and frame test_test_only_blocks_come_last as a conservative fast-feedback convention with --strict as the authoritative check -- no longer over-claiming. Verified: docs suite green (58 passed, 2 skipped), `mkdocs build --strict` exits 0, prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: remove design spec/plan caches from version control The spec and implementation plan under docs/superpowers/ were working artifacts, not published documentation (they were never in the mkdocs nav). The spec is preserved in a public gist; the plan is a local execution record. Remove both from the repo and drop the now-stale spec path from the test_docs.py module docstring. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: drop superpowers-docs exclusion now that those files are gone The _is_published_docs helper existed only to skip docs/superpowers/ design-doc caches; those files are no longer in the repo, so the helper and its three call sites were dead code referencing a nonexistent directory. Remove them — every find_examples(DOCS_ROOT) result is now published docs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test: share one moto S3 backend across fsspec and docs tests Both test_store/test_fsspec.py and test_docs.py stood up their own moto ThreadedMotoServer (ports 5555 and 5556). Extract a single session-scoped `moto_server` fixture + MOTO_ENDPOINT_URL constant into tests/conftest.py and have both consumers reuse it: - test_fsspec.py: s3_base now returns the shared moto_server; its per-test `s3` fixture (bucket "test", explicit endpoint, event-loop cleanup) is unchanged. - test_docs.py: docs_s3_backend depends on moto_server and adds only its docs-specific layer (process-wide AWS_ENDPOINT_URL + "example-bucket"); it no longer owns the server lifecycle. One server now serves the whole session; each consumer creates and moto-api-resets its own bucket. Verified: test_fsspec.py (96 passed), the docs suite (58 passed), both together in one session (154 passed), and the full standard suite in the optional env (5956 passed); prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * docs: document the exec vs test code-block distinction for contributors The contributing guide explained exec="true" but said nothing about test="true", the exec="false"+reason opt-out, the guard that requires one of them, or the placement constraint -- so a contributor could write a bare block and hit the guard with no explanation. Add a "Validating code blocks: exec vs test" section covering: - exec="true" (build-render) vs test="true" (validate-only) and when to use each - the exec="false" reason="..." opt-out and the test_no_unvalidated_blocks guard - markers="gpu"/"s3" for infra-bound blocks - the placement rule (test-only blocks come last) + --strict CI Attribute examples are shown as inline code rather than nested ```python fences, so pytest-examples' find_examples never mistakes the teaching examples for real blocks (verified: only the two genuine blocks in contributing.md are collected). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 8932bbe commit 0b2eda4

15 files changed

Lines changed: 328 additions & 88 deletions

File tree

‎.github/workflows/docs.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,10 @@ jobs:
2424
persist-credentials: false
2525
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
2626
- run: uv sync --group docs
27-
- run: uv run mkdocs build
27+
# --strict turns warnings into errors, so a docs code block that fails to execute
28+
# at build time (e.g. a non-exec python fence disrupting a later exec="true" block)
29+
# fails CI instead of merging as a silent warning.
30+
- run: uv run mkdocs build --strict
2831
env:
2932
DISABLE_MKDOCS_2_WARNING: "true"
3033
NO_MKDOCS_2_WARNING: "true"

‎changes/4016.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested.

‎docs/contributing.md‎

Lines changed: 61 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ If you find a bug, please raise a [GitHub issue](https://github.com/zarr-develop
1212

1313
1. A minimal, self-contained snippet of Python code reproducing the problem. You can format the code nicely using markdown, e.g.:
1414

15-
```python
15+
```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable"
1616
import zarr
1717
g = zarr.group()
1818
# etc.
@@ -225,10 +225,10 @@ hatch --env docs run serve
225225

226226
#### Adding executable code blocks in the documentation
227227

228-
Zarr uses [Markdown Exec](https://pawamoy.github.io/markdown-exec/usage/) to execute code blocks in Markdown files. Add `exec="on"` to a code block header for it to be executed when the docs are built. For example:
228+
Zarr uses [Markdown Exec](https://pawamoy.github.io/markdown-exec/usage/) to execute code blocks in Markdown files. Add `exec="true"` to a code block header for it to be executed when the docs are built. For example:
229229

230230
````md
231-
```python exec="on"
231+
```python exec="true"
232232
print("Hello world")
233233
```
234234
````
@@ -253,6 +253,64 @@ renders as:
253253
print("Hello world")
254254
```
255255

256+
#### Validating code blocks: `exec` vs `test`
257+
258+
Every Python code block in the documentation is checked by a test
259+
(`tests/test_docs.py`) so that examples cannot quietly rot — the bug that motivated
260+
this was an example calling `zarr.create_array(..., mode="w")`, an argument that does
261+
not exist, which went unnoticed because nothing ran it. A block declares *how* it is
262+
validated using one of two independent attributes:
263+
264+
- **`exec="true"`** — Markdown Exec runs the block **at docs-build time to render its
265+
output** into the page. This is the attribute described above; it is also what the
266+
test suite executes. Use it for ordinary examples whose output should appear in the
267+
docs.
268+
- **`test="true"`** — the block is **run by the test suite only**, *not* at build time.
269+
Use this for an example that should be validated but cannot run in the docs-build
270+
environment — for example one that needs a GPU or a cloud backend. Markdown Exec
271+
leaves a `test="true"` block as a static, syntax-highlighted snippet (it never
272+
executes it), while the test suite still runs it (see the marker note below).
273+
274+
A block may carry both (`exec="true" test="true"`), though in practice `exec="true"`
275+
already implies it is tested, so you rarely need `test="true"` alongside it.
276+
277+
The two attributes are kept separate on purpose: `exec=` controls *build-time rendering*
278+
and `test=` controls *test-time validation*. Tagging a GPU/cloud example `exec="true"`
279+
would make `mkdocs build` try to run it on a machine without that infrastructure and fail
280+
the build; `test="true"` lets it be validated without being built.
281+
282+
##### Opting a block out of validation
283+
284+
A handful of blocks genuinely cannot run and are not executable Python — a REPL
285+
transcript, a deliberately-incorrect "before" snippet, a `--8<--` file include. Mark
286+
these explicitly by opening the fence with
287+
`exec="false" reason="REPL output transcript, not executable source"` (supply a reason
288+
that fits the block).
289+
290+
`exec="false"` with a non-empty `reason` is an explicit, greppable opt-out. A test
291+
(`test_no_unvalidated_blocks`) requires **every** Python block to be either `exec="true"`,
292+
`test="true"`, or `exec="false"` with a reason — so a block can never silently skip
293+
validation. A bare ` ```python ` fence, or a typo like `exec="on"`, fails that test.
294+
295+
##### Marker-bound blocks (GPU, S3)
296+
297+
A `test="true"` block that needs special infrastructure declares a pytest marker with
298+
`markers="..."`, which binds it to that infrastructure in the test suite:
299+
300+
- `markers="gpu"` — run only under `pytest -m gpu` (the GPU CI environment); skipped
301+
elsewhere via `importorskip("cupy")`.
302+
- `markers="s3"` — run against a mock S3 (moto) backend supplied by a test fixture, so
303+
the example can use a bare `s3://…` URL with no test-only connection details on show.
304+
305+
##### Placement of `test="true"` blocks
306+
307+
Because Markdown Exec does not execute a `test="true"` (or `exec="false"`) block, placing
308+
one *before* an `exec="true"` block on the same page can disrupt the build-time execution
309+
of that later block. Put `test="true"` blocks **after** all `exec="true"` blocks on the
310+
page (or on a page where they are the only Python block). The `test_test_only_blocks_come_last`
311+
test enforces this, and the CI docs build runs with `--strict` so any such breakage fails
312+
the build rather than passing as a warning.
313+
256314
#### Building documentation without executing code blocks
257315

258316
Sometimes, you may want the documentation to build quicker. You can disable code block execution by commenting out the [markdown-exec plugin](https://github.com/zarr-developers/zarr-python/blob/884a8c91afcc3efe28b3da952be3b85125c453cb/mkdocs.yml#L132) in the mkdocs configuration file. This will make code blocks and cross references render incorrectly (i.e., expect build warnings), but also reduces build time by ~3x. Be sure to undo the commenting out before opening your pull request.

‎docs/quick-start.md‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -127,18 +127,6 @@ be done in a separate step.
127127
Zarr supports persistent storage to disk or cloud-compatible backends. While examples above
128128
utilized a [`zarr.storage.LocalStore`][], a number of other storage options are available.
129129

130-
Zarr integrates seamlessly with cloud object storage such as Amazon S3 and Google Cloud Storage
131-
using external libraries like [s3fs](https://s3fs.readthedocs.io) or
132-
[gcsfs](https://gcsfs.readthedocs.io):
133-
134-
```python
135-
136-
import s3fs
137-
138-
z = zarr.create_array("s3://example-bucket/foo", mode="w", shape=(100, 100), chunks=(10, 10), dtype="f4")
139-
z[:, :] = np.random.random((100, 100))
140-
```
141-
142130
A single-file store can also be created using the [`zarr.storage.ZipStore`][]:
143131

144132
```python exec="true" session="quickstart" source="above"
@@ -173,4 +161,18 @@ z = zarr.open_array(store, mode='r')
173161
print(z[:])
174162
```
175163

164+
Zarr also integrates seamlessly with cloud object storage such as Amazon S3 and Google
165+
Cloud Storage using external libraries like [s3fs](https://s3fs.readthedocs.io) or
166+
[gcsfs](https://gcsfs.readthedocs.io):
167+
168+
```python test="true" session="s3demo" markers="s3" source="above"
169+
import zarr
170+
import numpy as np
171+
172+
z = zarr.create_array(
173+
"s3://example-bucket/foo", shape=(100, 100), chunks=(10, 10), dtype="f4"
174+
)
175+
z[:, :] = np.random.random((100, 100))
176+
```
177+
176178
Read more about Zarr's storage options in the [User Guide](user-guide/index.md).

‎docs/user-guide/arrays.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -619,7 +619,7 @@ Without the `shards` argument, there would be 10,000 chunks stored as individual
619619
Because the feature is still stabilizing, it is disabled by default and
620620
must be explicitly enabled:
621621

622-
```python
622+
```python exec="true" session="arrays-rectilinear"
623623
import zarr
624624
zarr.config.set({"array.rectilinear_chunks": True})
625625
```

‎docs/user-guide/cli.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,13 @@ This will write new `zarr.json` files to `input.zarr`, leaving the existing v2 m
4545

4646
To open the array/group using the new metadata use:
4747

48-
```python
48+
```python exec="true" session="cli-open" source="above"
4949
import zarr
50-
zarr_with_v3_metadata = zarr.open('path/to/input.zarr', zarr_format=3)
50+
51+
# create a small array to open (stands in for the migrated store)
52+
zarr.create_array("data/cli-demo.zarr", shape=(4, 4), chunks=(2, 2), dtype="i4", overwrite=True)
53+
54+
zarr_with_v3_metadata = zarr.open("data/cli-demo.zarr", zarr_format=3)
5155
```
5256

5357
Once you are happy with the conversion, you can run the following to remove the old v2 metadata:

‎docs/user-guide/data_types.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,7 @@ print(type(a.dtype))
360360

361361
But if we inspect the metadata for the array, we can see the Zarr data type object:
362362

363-
```python
363+
```python exec="false" reason="REPL output transcript, not executable source"
364364
type(a.metadata.data_type)
365365
<class 'zarr.core.dtype.npy.int.Int64'>
366366
```

‎docs/user-guide/examples/custom_dtype.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22

33
## Source Code
44

5-
```python
5+
```python exec="false" reason="pymdownx snippet include directive, not python source"
66
--8<-- "examples/custom_dtype/custom_dtype.py"
77
```

‎docs/user-guide/gpu.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Zarr can use GPUs to accelerate your workload by running `zarr.Config.enable_gpu
1616
[`zarr.config`][] configures Zarr to use GPU memory for the data
1717
buffers used internally by Zarr via `enable_gpu()`.
1818

19-
```python
19+
```python test="true" session="gpu-demo" markers="gpu" source="above"
2020
import zarr
2121
import cupy as cp
2222
zarr.config.enable_gpu()

‎docs/user-guide/performance.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ determines the maximum number of concurrent I/O operations.
204204
The default value is 10, which is a conservative value. You may get improved performance by tuning
205205
the concurrency limit. You can adjust this value based on your specific needs:
206206

207-
```python
207+
```python exec="true" session="perf-concurrency"
208208
import zarr
209209

210210
# Set concurrency for the current session
@@ -234,7 +234,7 @@ By default it is `None`, which lets Python choose the pool size (typically
234234

235235
You can set it explicitly when you want more predictable resource usage:
236236

237-
```python
237+
```python exec="true" session="perf-workers"
238238
import zarr
239239

240240
zarr.config.set({'threading.max_workers': 8})
@@ -260,7 +260,7 @@ For example, if you're running Dask with 10 threads and Zarr's default concurren
260260

261261
**Recommendation**: When using Dask with many threads, configure Zarr's concurrency settings:
262262

263-
```python
263+
```python exec="false" reason="requires dask, which is not in the docs test environment"
264264
import zarr
265265
import dask.array as da
266266

0 commit comments

Comments
 (0)