Skip to content

Commit 8dd33e3

Browse files
committed
fix(codecs)!: BytesCodec defaults to little-endian on every host
`BytesCodec()` defaulted to `sys.byteorder`, so the same code wrote big-endian chunks on s390x and little-endian chunks elsewhere. The bytes codec originally defaulted to "little"; the host default came in as a side effect of replacing attrs with dataclasses (zarr-developers#1660), and was later pinned by a test that only restated it. `"little"` also matches `default_serializer_v3`. Chunks written on big-endian hosts with a bare `BytesCodec()` are now little-endian. Chunks written before this change remain readable, because the codec records its byte order in the metadata. Refs zarr-developers#3438 Assisted-by: ClaudeCode:claude-opus-5-5
1 parent 5a52062 commit 8dd33e3

3 files changed

Lines changed: 11 additions & 9 deletions

File tree

‎changes/4435.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
Fix byte-order bugs found by running the test suite on a big-endian (s390x) host.
22

33
- Zarr V3 data type metadata has no byte order, so V3 data types without an explicit `endianness` (such as `Float64()`, or a data type parsed from `"float64"`) now use the host byte order instead of always little-endian. On big-endian hosts, arrays opened from V3 metadata now return native arrays, and `dtype="float64"` and `dtype=np.float64` produce the same data type. Nothing changes on little-endian hosts. Stored chunks are still little-endian by default.
4+
- `BytesCodec()` now defaults to `endian="little"` instead of the host byte order, so chunks written with a default `bytes` codec are byte-identical on every host. On big-endian hosts, code that constructs `BytesCodec()` without an `endian` now writes little-endian chunks; these were already readable everywhere.
45
- `ShardingCodec` now defaults its inner and index `bytes` codecs to little-endian, so sharded arrays written with default settings are byte-identical on every host.
56
- The `scale_offset` codec no longer fails on arrays whose byte order is not the host's, for example a `">f8"` or `">u8"` array on a little-endian host.
67
- Base64-encoded fill values of structured data types in Zarr V3 metadata are now read and written as little-endian bytes, whatever the byte order of the fields in memory.

‎src/zarr/codecs/bytes.py‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
from __future__ import annotations
22

3-
import sys
43
import warnings
54
from dataclasses import dataclass, replace
65
from typing import TYPE_CHECKING, ClassVar, Final, Literal
@@ -41,13 +40,17 @@ def _parse_endian(data: object) -> EndianLiteral:
4140

4241
@dataclass(frozen=True)
4342
class BytesCodec(ArrayBytesCodec):
44-
"""bytes codec"""
43+
"""bytes codec
44+
45+
`endian` defaults to `"little"` on every host, so stored bytes do not depend on the
46+
byte order of the machine that wrote them.
47+
"""
4548

4649
is_fixed_size = True
4750

4851
endian: EndianLiteral | None
4952

50-
def __init__(self, *, endian: Endian | EndianLiteral | None = sys.byteorder) -> None:
53+
def __init__(self, *, endian: Endian | EndianLiteral | None = "little") -> None:
5154
if endian is None:
5255
endian_parsed: EndianLiteral | None = None
5356
else:

‎tests/test_codecs/test_bytes.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33
from __future__ import annotations
44

55
import enum
6-
import sys
76
import warnings
87
from typing import TYPE_CHECKING, Any, Literal, cast
98

@@ -292,14 +291,13 @@ def test_endian_attribute_error_for_unknown_member() -> None:
292291
getattr(Endian, "not_a_member") # noqa: B009
293292

294293

295-
def test_bytes_codec_default_endian_matches_system() -> None:
294+
def test_bytes_codec_default_endian_is_little() -> None:
296295
"""
297-
Constructing `BytesCodec()` with no arguments yields a codec whose
298-
`endian` matches `sys.byteorder`. This replaces the previous
299-
`default_system_endian = Endian(sys.byteorder)` module-level binding.
296+
Constructing `BytesCodec()` with no arguments yields a little-endian codec on
297+
every host, so stored bytes do not depend on the machine that wrote them.
300298
"""
301299
codec = BytesCodec()
302-
assert codec.endian == sys.byteorder
300+
assert codec.endian == "little"
303301

304302

305303
def _make_array_spec(dtype: Any) -> ArraySpec:

0 commit comments

Comments
 (0)