Repository navigation
Commit a2bc98c
committed
perf: cut the fill_value check on the chunk write path
`NDBuffer.all_equal` runs on every chunk write, because that is how
`write_empty_chunks=False` (the default) decides a chunk is uniformly
`fill_value` and can be skipped. Instrumented on an uncompressed float32
write it was 62.6% of total write time. Two independent causes.
The bit-pattern comparison for a zero `fill_value`, which exists so that
`-0.0` stays distinct from `0.0` (zarr-developers#3144), viewed both sides as `V{itemsize}`.
NumPy has vectorised integer comparison loops and no void one, so that falls
back to a generic elementwise path: 27x slower than the equivalent unsigned
integer view on float32 and 33x on float64, measured on numpy 2.5.3. Viewing
as an unsigned integer of the same width is equally bit-exact. Widths with no
same-size integer, longdouble and complex128, keep the void form.
The scan also had no early exit, so a buffer that is not uniformly
`fill_value` still had to be read in full before `np.array_equal` could
report the first mismatch, which on the write path is the common case. It now
walks slices of the leading axis, doubling in length, and returns on the first
slice that differs. The leading axis is used rather than a flattened view
because a chunk is normally a strided view into a larger array and cannot be
flattened without a copy; gating the early exit on contiguity makes it
silently never engage. A buffer that really is uniform is still covered in the
same number of element comparisons.
Both `fill_value` regimes benefit, for different reasons: zarr's own default
is `0.0` and takes the first change, while xarray writes `fill_value=NaN` and
takes only the second.
Behaviour is unchanged, including for `-0.0`, NaN, and non-contiguous buffers.
Assisted-by: ClaudeCode:claude-opus-51 parent 7291469 commit a2bc98c
3 files changed
Lines changed: 167 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
31 | 40 | | |
32 | 41 | | |
33 | 42 | | |
| |||
531 | 540 | | |
532 | 541 | | |
533 | 542 | | |
534 | | - | |
535 | | - | |
536 | | - | |
537 | | - | |
538 | | - | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
539 | 546 | | |
540 | 547 | | |
541 | 548 | | |
542 | 549 | | |
543 | | - | |
| 550 | + | |
544 | 551 | | |
545 | | - | |
546 | | - | |
547 | | - | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
548 | 561 | | |
549 | | - | |
550 | | - | |
551 | | - | |
552 | | - | |
553 | | - | |
554 | | - | |
555 | | - | |
556 | | - | |
557 | | - | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
558 | 569 | | |
559 | 570 | | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
560 | 603 | | |
561 | 604 | | |
562 | 605 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
0 commit comments