Repository navigation
Commit bd24f4f
authored
fix: keep FusedCodecPipeline compute off the event-loop thread (#4194)
* fix: byte-order handling for structured dtypes in the bytes codec (#220)
* fix: byte-order handling for structured dtypes in the bytes codec
The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).
Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.
Closes #4141
Assisted-by: ClaudeCode:claude-fable-5
* test: fold structured byte-order cases into existing bytes codec tests
Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.
Assisted-by: ClaudeCode:claude-fable-5
* refactor: rename stored_dtype to view_dtype in BytesCodec decode
The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.
Assisted-by: ClaudeCode:claude-fable-5
* docs: note that the decode-side byte-order conversion copies the chunk
Assisted-by: ClaudeCode:claude-fable-5
* fix(codec_pipeline): keep FusedCodecPipeline compute off the event-loop thread
FusedCodecPipeline.read/write ran their synchronous fast path inline on
the coroutine servicing the request — i.e. on the global zarr_io event
loop thread. Single-chunk batches decoded inline on the loop and
multi-chunk batches blocked the loop in pool.map, so every sync-API call
from every user thread serialized behind each other's codec compute.
The blocked window scales with codec cost, which is why users reported
the fused pipeline as "slower for zstd-compressed data" under
multi-threaded (dask-style, one chunk per call) access: at 8 reader
threads on 4 MiB zstd chunks it was 3.4x slower than
BatchedCodecPipeline, and throughput did not scale with threads at all
(336 -> 439 ms from 1 to 8 threads, versus 669 -> 121 ms for batched).
Offload the synchronous batch to a worker thread with asyncio.to_thread:
one hop per batch, not per chunk, preserving the fused pipeline's win
over per-chunk async scheduling while keeping the loop free. After the
fix the same workload scales 625 -> 109 ms from 1 to 8 threads, beating
batched at every thread count; single-threaded performance is unchanged
(the hop costs ~75 us per batch).
The regression test asserts deterministically (no timing) that codec
compute never runs on a thread with a running event loop, covering
single- and multi-chunk reads and writes through the sync API. A new
benchmark covers the many-threads/one-chunk-per-call access pattern.
Assisted-by: ClaudeCode:claude-fable-5
* docs: rename change note to upstream PR number (4194)
towncrier's issue_format links to zarr-developers/zarr-python issues, so
247 (the fork PR number) would render a link to an unrelated old issue.
Assisted-by: ClaudeCode:claude-fable-5
* test: guard event-loop test against vacuity if the sync fast path stops triggering
The test asserts a negative (compute never ran on the loop thread). If a
refactor made the sync fast path stop triggering, the traced ChunkTransform
methods would never be called (the async fallback uses AsyncChunkTransform)
and the test would pass while guarding nothing. Assert the traced hooks
actually ran.
Assisted-by: ClaudeCode:claude-fable-51 parent 0c07bf5 commit bd24f4f
4 files changed
Lines changed: 133 additions & 2 deletions
File tree
- changes
- src/zarr/core
- tests
- benchmarks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1182 | 1182 | | |
1183 | 1183 | | |
1184 | 1184 | | |
1185 | | - | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
1186 | 1194 | | |
1187 | 1195 | | |
1188 | 1196 | | |
| |||
1235 | 1243 | | |
1236 | 1244 | | |
1237 | 1245 | | |
1238 | | - | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
1239 | 1251 | | |
1240 | 1252 | | |
1241 | 1253 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
37 | 98 | | |
38 | 99 | | |
39 | 100 | | |
| |||
0 commit comments