Skip to content

Commit 4a1a441

Browse files
committed
docs(blog): render post list and fill in release-post details
Configure the Material blog plugin so docs/blog/index.md renders a reverse-chronological post list instead of a blank page. The plugin ships with mkdocs-material, which is already pinned, so no new dependency is needed. Archive and category indexes are disabled -- one post doesn't justify the extra navigation. Fill in the [NEEDS INPUT] placeholders in the 3.3.0 post from the release notes and user guide: the zarr-developers#3524 regression report, the zarr-developers#3885 / zarr-developers#3925 / zarr-developers#3987 / zarr-developers#3004 pull requests, runnable config examples for both FusedCodecPipeline and the coalescing knobs, and links to the release notes and config guide. Also correct two accuracy issues: FusedCodecPipeline is opt-in and experimental (not on by default, unlike coalescing), and the sharding example now consistently says four inner chunks rather than switching to three mid-paragraph. Assisted-by: ClaudeCode:claude-opus-5
1 parent d1533e1 commit 4a1a441

4 files changed

Lines changed: 121 additions & 11 deletions

File tree

‎docs/blog/.authors.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
authors:
2+
d-v-b:
3+
name: Davis Bennett
4+
description: Zarr-Python core developer
5+
avatar: https://github.com/d-v-b.png
6+
url: https://github.com/d-v-b

‎docs/blog/index.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Blog
2+
3+
News, release highlights, and design notes from the Zarr-Python developers.

‎docs/blog/posts/3.3.0-release.md‎

Lines changed: 104 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,42 @@
1+
---
2+
date: 2026-07-15
3+
authors:
4+
- d-v-b
5+
categories:
6+
- Release
7+
---
8+
19
# Zarr-Python 3.3.0
210

311
We're happy to announce the release of version 3.3.0 of Zarr-Python. It's been a while since our last release,
4-
and we're bringing some exciting additions in 3.3.0. For the full release notes, see [NEEDS INPUT], otherwise stick around for an overview of two performance-centric highlights of this release.
12+
and we're bringing some exciting additions in 3.3.0. For the full release notes, see the [3.3.0 release notes](../../release-notes.md), otherwise stick around for an overview of two performance-centric highlights of this release.
13+
14+
<!-- more -->
515

616
## Faster low-latency storage
717

8-
[NEEDS INPUT]
9-
<list issues and PRs related to this feature>
18+
Relevant issues and pull requests:
19+
20+
- [#3524](https://github.com/zarr-developers/zarr-python/issues/3524) -- the performance report that started this work
21+
- [#3885](https://github.com/zarr-developers/zarr-python/pull/3885) -- synchronous codec APIs and the `FusedCodecPipeline`
1022

1123
### The cost of async overhead
1224

1325
Zarr-Python 3.x uses async routines for fetching data and decoding chunks. In terms of code, this means our store (data fetching) and codec (chunk decoding) APIs are both async. This makes
1426
I/O against high-latency storage backends like cloud object storage efficient. But for *low-latency* storage, like in-process memory or the file system, async routines add measurable overhead and offer no benefit. Async only adds value when there's work to be done while waiting for I/O to complete, but when I/O latency is low, it completes too quickly to run anything while waiting, and we are left paying the performance bill for obligatory async task scheduling that offered no value.
1527

16-
This performance problem became acute when Zarr-Python users reported that in-memory array indexing workloads ran *slower* in Zarr-Python 3.x relative to Zarr-Python 2.18 (issue [NEEDS INPUT]). Fortunately this performance regression had a straightforward fix (I don't say "easy" because it was a lot of work).
28+
This performance problem became acute when Zarr-Python users reported that in-memory array indexing workloads ran *slower* in Zarr-Python 3.x relative to Zarr-Python 2.18 ([#3524](https://github.com/zarr-developers/zarr-python/issues/3524)). Fortunately this performance regression had a straightforward fix (I don't say "easy" because it was a lot of work).
1729

1830
### Synchronous execution restores performance
1931

2032
If async overhead makes low-latency storage slow, does *removing* that overhead restore performance? Yes, it does!
2133

22-
In a series of PRs ([NEEDS INPUT]) we defined synchronous versions of our storage and codec APIs, and then combined them in a new codec orchestration class called `FusedCodecPipeline`. The `FusedCodecPipeline` is an alternative to the default (the `BatchedCodecPipeline`) that gives large speedups for low-latency storage.
34+
In [#3885](https://github.com/zarr-developers/zarr-python/pull/3885) we defined synchronous versions of our storage and codec APIs -- the `SyncByteGetter` and `SyncByteSetter` protocols, plus a `get_ranges_sync` method on the `Store` ABC -- and then combined them in a new codec orchestration class called `FusedCodecPipeline`. The `FusedCodecPipeline` is an opt-in alternative to the default (the `BatchedCodecPipeline`) that gives large speedups for low-latency storage. It is currently marked [experimental](../../user-guide/experimental.md), so we may change it as we learn more; the default pipeline is untouched, and existing code keeps working unless you opt in.
35+
36+
The win here is *not* a faster compressor or a different on-disk format -- the bytes written are byte-for-byte identical. It is the removal of async scheduling overhead (including some [nasty `asyncio.to_thread` overhead](https://github.com/python/cpython/issues/136084)), plus a few vectorized fast paths for dense, uncompressed shards.
2337

2438
On this author's 10-core Apple M4 laptop, the `FusedCodecPipeline` delivers the following results against memory-backed arrays:
39+
2540
- uncompressed writes are *~4 times faster*
2641
- uncompressed reads are *~5 times faster*
2742
- compressed writes are *~2 times faster*
@@ -34,23 +49,102 @@ thread-based parallelism is worse than useless and slows things down. But for co
3449

3550
### How to use it
3651

37-
[NEEDS INPUT] code example showing how to set the global config or use the `config` attribute of an array
52+
Select the pipeline through the [runtime configuration](../../user-guide/config.md) by setting `codec_pipeline.path`. Set it globally to affect every array created or opened afterwards:
53+
54+
```python exec="true" session="blog-330" source="above"
55+
import zarr
56+
57+
zarr.config.set(
58+
{"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"}
59+
)
60+
```
61+
62+
Or scope it to a block of code by using `zarr.config.set` as a context manager, which is the safer choice if you only want the new pipeline for part of your program:
63+
64+
```python exec="true" session="blog-330" source="above" result="ansi"
65+
import numpy as np
66+
import zarr
67+
from zarr.storage import MemoryStore
68+
69+
with zarr.config.set(
70+
{"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"}
71+
):
72+
arr = zarr.create_array(
73+
store=MemoryStore(),
74+
shape=(1000, 1000),
75+
chunks=(100, 100),
76+
shards=(1000, 1000),
77+
dtype="float32",
78+
)
79+
arr[:] = np.random.random((1000, 1000)).astype("float32")
80+
result = arr[:]
81+
82+
print(result.shape)
83+
```
84+
85+
Thread-based parallelism is configured separately, via `codec_pipeline.max_workers`. It defaults to `None`, meaning a pool sized to `os.cpu_count()`. As noted above, memory-backed and uncompressed workloads often do better with a single worker:
86+
87+
```python exec="true" session="blog-330" source="above"
88+
import zarr
89+
90+
# A fixed-size thread pool for codec compute.
91+
zarr.config.set({"codec_pipeline.max_workers": 8})
92+
93+
# Or "auto", sized to the number of CPUs.
94+
zarr.config.set({"codec_pipeline.max_workers": None})
95+
```
96+
97+
To return to the default pipeline, set `codec_pipeline.path` back to the batched implementation:
98+
99+
```python exec="true" session="blog-330" source="above"
100+
import zarr
101+
102+
zarr.config.set(
103+
{"codec_pipeline.path": "zarr.core.codec_pipeline.BatchedCodecPipeline"}
104+
)
105+
```
38106

39107
## Faster sharded reads
40108

41-
[NEEDS INPUT] <list issues and PRs related to this change>
109+
Relevant issues and pull requests:
110+
111+
- [#3004](https://github.com/zarr-developers/zarr-python/pull/3004) -- optimize partial shard reads
112+
- [#3925](https://github.com/zarr-developers/zarr-python/pull/3925) -- `Store.get_ranges` for concurrent, coalesced multi-range reads
113+
- [#3987](https://github.com/zarr-developers/zarr-python/pull/3987) -- control coalescing through `ArrayConfig` and the runtime config
42114

43115
### How sharding works
44116

45-
Chunks encoded with the `sharding_indexed` contain subchunks, or "inner chunks". For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. A reader can satisfy a request for 3 inner chunks by issuing 3 separate byte-range requests. That's what previous versions of Zarr-Python did.
117+
Chunks encoded with the `sharding_indexed` codec contain subchunks, or "inner chunks". For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. A reader can satisfy a request for all four inner chunks by issuing four separate byte-range requests. That's what previous versions of Zarr-Python did.
46118

47119
### Interval equivalence
48120

49-
Byte ranges, being intervals, obey some combination rules: the values in two half-open intervals `[a, b), [b, c)` can be captured by the single interval `[a, c)`. That means a reader could get all three inner chunks with *one* byte-range request by requesting a range of bytes starting with the first byte of the first subchunk and ending with the last byte of the last subchunk. When individual requests are expensive, this kind of optimization is worth a lot; we could use one request instead of three, in the previous example.
121+
Byte ranges, being intervals, obey some combination rules: the values in two half-open intervals `[a, b), [b, c)` can be captured by the single interval `[a, c)`. That means a reader could get all four inner chunks with *one* byte-range request by requesting a range of bytes starting with the first byte of the first subchunk and ending with the last byte of the last subchunk. When individual requests are expensive, this kind of optimization is worth a lot; we could use one request instead of four, in the previous example.
50122

51123
### Byte range coalescing
52124

53-
We call this procedure -- rewriting `N` byte range requests into `M` byte range requests, where `M` <= `N` -- "byte range coalescing", and it's a new performance optimization shipping in Zarr-Python 3.3.0. It's on by default with base settings we think are good, so most users won't need to tune anything. If you do need to adjust some settings, see the full documentation of the feature: [NEEDS INPUT] link to official docs
125+
We call this procedure -- rewriting `N` byte range requests into `M` byte range requests, where `M` <= `N` -- "byte range coalescing", and it's a new performance optimization shipping in Zarr-Python 3.3.0. Unlike the `FusedCodecPipeline`, this one is on by default with base settings we think are good, so most users won't need to tune anything.
126+
127+
Two knobs control it, both documented in the [runtime configuration guide](../../user-guide/config.md). Nearby byte ranges in the same shard are merged into a single request when the gap between them is no larger than `array.sharding_coalesce_max_gap_bytes` (default 1 MiB) and the merged read stays within `array.sharding_coalesce_max_bytes` (default 16 MiB). The gap threshold is what trades wasted bytes against saved requests: raising it reads more data you didn't ask for, in exchange for fewer round trips.
128+
129+
You can set them globally, or per array by passing `config={...}` to [`zarr.create_array`][]:
130+
131+
```python exec="true" session="blog-330" source="above" result="ansi"
132+
import zarr
133+
from zarr.storage import MemoryStore
134+
135+
arr = zarr.create_array(
136+
store=MemoryStore(),
137+
shape=(1000, 1000),
138+
chunks=(100, 100),
139+
shards=(1000, 1000),
140+
dtype="float32",
141+
config={
142+
"sharding_coalesce_max_gap_bytes": 4 << 20, # 4 MiB
143+
"sharding_coalesce_max_bytes": 64 << 20, # 64 MiB
144+
},
145+
)
146+
print(arr.shape)
147+
```
54148

55149
## Going faster
56150

‎mkdocs.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,6 @@ nav:
9696
- contributing.md
9797
- Blog:
9898
- blog/index.md
99-
- blog/posts/3.3.0-release.md
10099
hooks:
101100
- mkdocs_hooks.py
102101

@@ -155,6 +154,14 @@ extra_css:
155154

156155
plugins:
157156
- autorefs
157+
- blog:
158+
blog_dir: blog
159+
post_dir: "{blog}/posts"
160+
post_url_format: "{slug}"
161+
# The blog is a simple reverse-chronological list of posts; the archive
162+
# and category indexes add navigation we don't have the volume to justify.
163+
archive: false
164+
categories: false
158165
- search
159166
- markdown-exec
160167
- mkdocstrings:

0 commit comments

Comments
 (0)