ZStd test fail on 32bit atchitectures

On i386, 5 zstd related tests fail due to

E   SystemError: Negative size passed to PyBytes_FromStringAndSize`

The relevant part of the test log is below

$ python3 -m pytest -k "not test_non_numpy_inputs and not test_non_numpy_inputs and not test_native_functions" -v --pyargs numcodecs --ignore=./numcodecs/pcodec.py --ignore=./numcodecs/zarr3.py --ignore=./numcodecs/tests/test_zarr3.py
============================= test session starts ==============================
platform linux -- Python 3.13.6, pytest-8.4.1, pluggy-1.6.0 -- /usr/bin/python3.13
cachedir: .pytest_cache
rootdir: /builds/science-team/numcodecs/debian/output/source_dir/.pybuild/cpython3_3.13_numcodecs/build
configfile: pyproject.toml
plugins: cov-5.0.0, typeguard-4.4.2
collecting ... collected 740 items / 27 deselected / 4 skipped / 713 selected

numcodecs/astype.py::numcodecs.astype.AsType PASSED                      [  0%]

[CUT]

numcodecs/tests/test_zstd.py::test_streaming_decompression FAILED        [100%]

=================================== FAILURES ===================================
_____________________ test_pyzstd_streaming[Hello World!] ______________________

input = b'Hello World!'

    @pytest.mark.parametrize("input", test_data)
    def test_pyzstd_streaming(input):
        """
        Test if Zstd can decode a single frame and concatenated frames in streaming
        mode where the decompressed size is not recorded in the frame header.
        """
        pyzstd_c = pyzstd.ZstdCompressor()
        pyzstd_d = pyzstd.ZstdDecompressor()
        pyzstd_e = pyzstd.EndlessZstdDecompressor()
        z = Zstd()
    
        d_bytes = input
        pyzstd_c.compress(d_bytes)
        c_bytes = pyzstd_c.flush()
>       assert z.decode(c_bytes) == d_bytes
               ^^^^^^^^^^^^^^^^^

numcodecs/tests/test_pyzstd.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
numcodecs/zstd.pyx:388: in numcodecs.zstd.Zstd.decode
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   SystemError: Negative size passed to PyBytes_FromStringAndSize

numcodecs/zstd.pyx:229: SystemError
_ test_pyzstd_streaming[\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00\x0f\x00\x00\x00\x10\x00\x00\x00\x11\x00\x00\x00\x12\x00\x00\x00\x13\x00\x00\x00\x14\x00\x00\x00\x15\x00\x00\x00\x16\x00\x00\x00\x17\x00\x00\x00\x18\x00\x00\x00\x19\x00\x00\x00\x1a\x00\x00\x00\x1b\x00\x00\x00\x1c\x00\x00\x00\x1d\x00\x00\x00\x1e\x00\x00\x00\x1f\x00\x00\x00 \x00\x00\x00!\x00\x00\x00"\x00\x00\x00#\x00\x00\x00$\x00\x00\x00%\x00\x00\x00&\x00\x00\x00'\x00\x00\x00(\x00\x00\x00)\x00\x00\x00*\x00\x00\x00+\x00\x00\x00,\x00\x00\x00-\x00\x00\x00.\x00\x00\x00/\x00\x00\x000\x00\x00\x001\x00\x00\x002\x00\x00\x003\x00\x00\x004\x00\x00\x005\x00\x00\x006\x00\x00\x007\x00\x00\x008\x00\x00\x009\x00\x00\x00:\x00\x00\x00;\x00\x00\x00<\x00\x00\x00=\x00\x00\x00>\x00\x00\x00?\x00\x00\x00@\x00\x00\x00A\x00\x00\x00B\x00\x00\x00C\x00\x00\x00D\x00\x00\x00E\x00\x00\x00F\x00\x00\x00G\x00\x00\x00H\x00\x00\x00I\x00\x00\x00J\x00\x00\x00K\x00\x00\x00L\x00\x00\x00M\x00\x00\x00N\x00\x00\x00O\x00\x00\x00P\x00\x00\x00Q\x00\x00\x00R\x00\x00\x00S\x00\x00\x00T\x00\x00\x00U\x00\x00\x00V\x00\x00\x00W\x00\x00\x00X\x00\x00\x00Y\x00\x00\x00Z\x00\x00\x00[\x00\x00\x00\\x00\x00\x00]\x00\x00\x00^\x00\x00\x00_\x00\x00\x00`\x00\x00\x00a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00e\x00\x00\x00f\x00\x00\x00g\x00\x00\x00h\x00\x00\x00i\x00\x00\x00j\x00\x00\x00k\x00\x00\x00l\x00\x00\x00m\x00\x00\x00n\x00\x00\x00o\x00\x00\x00p\x00\x00\x00] _

input = b'\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07...0h\x00\x00\x00i\x00\x00\x00j\x00\x00\x00k\x00\x00\x00l\x00\x00\x00m\x00\x00\x00n\x00\x00\x00o\x00\x00\x00p\x00\x00\x00'

    @pytest.mark.parametrize("input", test_data)
    def test_pyzstd_streaming(input):
        """
        Test if Zstd can decode a single frame and concatenated frames in streaming
        mode where the decompressed size is not recorded in the frame header.
        """
        pyzstd_c = pyzstd.ZstdCompressor()
        pyzstd_d = pyzstd.ZstdDecompressor()
        pyzstd_e = pyzstd.EndlessZstdDecompressor()
        z = Zstd()
    
        d_bytes = input
        pyzstd_c.compress(d_bytes)
        c_bytes = pyzstd_c.flush()
>       assert z.decode(c_bytes) == d_bytes
               ^^^^^^^^^^^^^^^^^

numcodecs/tests/test_pyzstd.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
numcodecs/zstd.pyx:388: in numcodecs.zstd.Zstd.decode
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   SystemError: Negative size passed to PyBytes_FromStringAndSize

numcodecs/zstd.pyx:229: SystemError
_ test_pyzstd_streaming[\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00] _

input = b'\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00'

    @pytest.mark.parametrize("input", test_data)
    def test_pyzstd_streaming(input):
        """
        Test if Zstd can decode a single frame and concatenated frames in streaming
        mode where the decompressed size is not recorded in the frame header.
        """
        pyzstd_c = pyzstd.ZstdCompressor()
        pyzstd_d = pyzstd.ZstdDecompressor()
        pyzstd_e = pyzstd.EndlessZstdDecompressor()
        z = Zstd()
    
        d_bytes = input
        pyzstd_c.compress(d_bytes)
        c_bytes = pyzstd_c.flush()
>       assert z.decode(c_bytes) == d_bytes
               ^^^^^^^^^^^^^^^^^

numcodecs/tests/test_pyzstd.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
numcodecs/zstd.pyx:388: in numcodecs.zstd.Zstd.decode
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   SystemError: Negative size passed to PyBytes_FromStringAndSize

numcodecs/zstd.pyx:229: SystemError
_ test_pyzstd_streaming[+\x00*\x00\x1c\x00'\x00\x1c\x00\x0e\x00\r\x00#\x00*\x00!\x00(\x00\x05\x00\x16\x00$\x00\x11\x00\x13\x00\n\x00(\x00\t\x00\x10\x00\x0c\x00-\x00\x04\x00\x08\x00\x07\x00\x1f\x00\x16\x00\x15\x00\x08\x00\x07\x00\t\x00\x13\x00(\x000\x00\x11\x00\x0e\x00\x1b\x00\x1a\x00\x1b\x00-\x00\x15\x00\x1c\x00&\x00\x0c\x00,\x00\n\x00\x11\x00"\x00\x1b\x00-\x00'\x00\x1b\x00\n\x00] _

input = b'+\x00*\x00\x1c\x00\'\x00\x1c\x00\x0e\x00\r\x00#\x00*\x00!\x00(\x00\x05\x00\x16\x00$\x00\x11\x00\x13\x00\n\x00(\x00\t...x00\x1b\x00\x1a\x00\x1b\x00-\x00\x15\x00\x1c\x00&\x00\x0c\x00,\x00\n\x00\x11\x00"\x00\x1b\x00-\x00\'\x00\x1b\x00\n\x00'

    @pytest.mark.parametrize("input", test_data)
    def test_pyzstd_streaming(input):
        """
        Test if Zstd can decode a single frame and concatenated frames in streaming
        mode where the decompressed size is not recorded in the frame header.
        """
        pyzstd_c = pyzstd.ZstdCompressor()
        pyzstd_d = pyzstd.ZstdDecompressor()
        pyzstd_e = pyzstd.EndlessZstdDecompressor()
        z = Zstd()
    
        d_bytes = input
        pyzstd_c.compress(d_bytes)
        c_bytes = pyzstd_c.flush()
>       assert z.decode(c_bytes) == d_bytes
               ^^^^^^^^^^^^^^^^^

numcodecs/tests/test_pyzstd.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
numcodecs/zstd.pyx:388: in numcodecs.zstd.Zstd.decode
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   SystemError: Negative size passed to PyBytes_FromStringAndSize

numcodecs/zstd.pyx:229: SystemError
_________________________ test_streaming_decompression _________________________

    def test_streaming_decompression():
        # Test input frames with unknown frame content size
        codec = Zstd()
    
        # If the zstd command line interface is available, check the bytes
        cli = zstd_cli_available()
        if cli:
            view_zstd_streaming_bytes()
    
        # Encode bytes directly that were the result of streaming compression
        bytes_val = b'(\xb5/\xfd\x00Xa\x00\x00Hello World!'
>       dec = codec.decode(bytes_val)
              ^^^^^^^^^^^^^^^^^^^^^^^

numcodecs/tests/test_zstd.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
numcodecs/zstd.pyx:388: in numcodecs.zstd.Zstd.decode
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   SystemError: Negative size passed to PyBytes_FromStringAndSize

numcodecs/zstd.pyx:229: SystemError
----------------------------- Captured stdout call -----------------------------
    bytes_val = b'(\xb5/\xfd\x00Xa\x00\x00Hello world!'
    bytes3 = b'(\xb5/\xfd\x00X$\x02\x00\xa4\x03ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz\x01\x00:\xfc\xdfs\x05\x05L\x00\x00\x08s\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08k\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08c\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08[\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08S\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08K\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08C\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08u\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08m\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08e\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08]\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08U\x01\x00\xfc\xff9\x10\x02L\x00\x00\x08M\x01\x00\xfc\xff9\x10\x02M\x00\x00\x08E\x01\x00\xfc\x7f\x1d\x08\x01'

---------- coverage: platform linux, python 3.13.6-final-0 -----------
Coverage XML written to file coverage.xml

=========================== short test summary info ============================
SKIPPED [2] numcodecs/tests/test_pcodec.py:7: pcodec not available
SKIPPED [2] numcodecs/tests/test_zfpy.py:11: ZFPY not available
SKIPPED [15] numcodecs/tests/test_blosc.py:62: codec has been removed
SKIPPED [1] numcodecs/tests/test_ndarray_like.py:8: could not import 'cupy': No module named 'cupy'
SKIPPED [1] numcodecs/tests/test_ndarray_like.py:20: could not import 'cupy': No module named 'cupy'
SKIPPED [1] numcodecs/tests/test_ndarray_like.py:32: could not import 'cupy': No module named 'cupy'
SKIPPED [1] numcodecs/tests/test_ndarray_like.py:44: could not import 'cupy': No module named 'cupy'
XFAIL numcodecs/tests/test_pyzstd.py::test_pyzstd_simple_multiple_frames_decode[Hello World!]
XFAIL numcodecs/tests/test_pyzstd.py::test_pyzstd_simple_multiple_frames_decode[\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00\x0f\x00\x00\x00\x10\x00\x00\x00\x11\x00\x00\x00\x12\x00\x00\x00\x13\x00\x00\x00\x14\x00\x00\x00\x15\x00\x00\x00\x16\x00\x00\x00\x17\x00\x00\x00\x18\x00\x00\x00\x19\x00\x00\x00\x1a\x00\x00\x00\x1b\x00\x00\x00\x1c\x00\x00\x00\x1d\x00\x00\x00\x1e\x00\x00\x00\x1f\x00\x00\x00 \x00\x00\x00!\x00\x00\x00"\x00\x00\x00#\x00\x00\x00$\x00\x00\x00%\x00\x00\x00&\x00\x00\x00'\x00\x00\x00(\x00\x00\x00)\x00\x00\x00*\x00\x00\x00+\x00\x00\x00,\x00\x00\x00-\x00\x00\x00.\x00\x00\x00/\x00\x00\x000\x00\x00\x001\x00\x00\x002\x00\x00\x003\x00\x00\x004\x00\x00\x005\x00\x00\x006\x00\x00\x007\x00\x00\x008\x00\x00\x009\x00\x00\x00:\x00\x00\x00;\x00\x00\x00<\x00\x00\x00=\x00\x00\x00>\x00\x00\x00?\x00\x00\x00@\x00\x00\x00A\x00\x00\x00B\x00\x00\x00C\x00\x00\x00D\x00\x00\x00E\x00\x00\x00F\x00\x00\x00G\x00\x00\x00H\x00\x00\x00I\x00\x00\x00J\x00\x00\x00K\x00\x00\x00L\x00\x00\x00M\x00\x00\x00N\x00\x00\x00O\x00\x00\x00P\x00\x00\x00Q\x00\x00\x00R\x00\x00\x00S\x00\x00\x00T\x00\x00\x00U\x00\x00\x00V\x00\x00\x00W\x00\x00\x00X\x00\x00\x00Y\x00\x00\x00Z\x00\x00\x00[\x00\x00\x00\\x00\x00\x00]\x00\x00\x00^\x00\x00\x00_\x00\x00\x00`\x00\x00\x00a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00e\x00\x00\x00f\x00\x00\x00g\x00\x00\x00h\x00\x00\x00i\x00\x00\x00j\x00\x00\x00k\x00\x00\x00l\x00\x00\x00m\x00\x00\x00n\x00\x00\x00o\x00\x00\x00p\x00\x00\x00]
XFAIL numcodecs/tests/test_pyzstd.py::test_pyzstd_simple_multiple_frames_decode[\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00]
XFAIL numcodecs/tests/test_pyzstd.py::test_pyzstd_simple_multiple_frames_decode[+\x00*\x00\x1c\x00'\x00\x1c\x00\x0e\x00\r\x00#\x00*\x00!\x00(\x00\x05\x00\x16\x00$\x00\x11\x00\x13\x00\n\x00(\x00\t\x00\x10\x00\x0c\x00-\x00\x04\x00\x08\x00\x07\x00\x1f\x00\x16\x00\x15\x00\x08\x00\x07\x00\t\x00\x13\x00(\x000\x00\x11\x00\x0e\x00\x1b\x00\x1a\x00\x1b\x00-\x00\x15\x00\x1c\x00&\x00\x0c\x00,\x00\n\x00\x11\x00"\x00\x1b\x00-\x00'\x00\x1b\x00\n\x00]
FAILED numcodecs/tests/test_pyzstd.py::test_pyzstd_streaming[Hello World!] - SystemError: Negative size passed to PyBytes_FromStringAndSize
FAILED numcodecs/tests/test_pyzstd.py::test_pyzstd_streaming[\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00\x0f\x00\x00\x00\x10\x00\x00\x00\x11\x00\x00\x00\x12\x00\x00\x00\x13\x00\x00\x00\x14\x00\x00\x00\x15\x00\x00\x00\x16\x00\x00\x00\x17\x00\x00\x00\x18\x00\x00\x00\x19\x00\x00\x00\x1a\x00\x00\x00\x1b\x00\x00\x00\x1c\x00\x00\x00\x1d\x00\x00\x00\x1e\x00\x00\x00\x1f\x00\x00\x00 \x00\x00\x00!\x00\x00\x00"\x00\x00\x00#\x00\x00\x00$\x00\x00\x00%\x00\x00\x00&\x00\x00\x00'\x00\x00\x00(\x00\x00\x00)\x00\x00\x00*\x00\x00\x00+\x00\x00\x00,\x00\x00\x00-\x00\x00\x00.\x00\x00\x00/\x00\x00\x000\x00\x00\x001\x00\x00\x002\x00\x00\x003\x00\x00\x004\x00\x00\x005\x00\x00\x006\x00\x00\x007\x00\x00\x008\x00\x00\x009\x00\x00\x00:\x00\x00\x00;\x00\x00\x00<\x00\x00\x00=\x00\x00\x00>\x00\x00\x00?\x00\x00\x00@\x00\x00\x00A\x00\x00\x00B\x00\x00\x00C\x00\x00\x00D\x00\x00\x00E\x00\x00\x00F\x00\x00\x00G\x00\x00\x00H\x00\x00\x00I\x00\x00\x00J\x00\x00\x00K\x00\x00\x00L\x00\x00\x00M\x00\x00\x00N\x00\x00\x00O\x00\x00\x00P\x00\x00\x00Q\x00\x00\x00R\x00\x00\x00S\x00\x00\x00T\x00\x00\x00U\x00\x00\x00V\x00\x00\x00W\x00\x00\x00X\x00\x00\x00Y\x00\x00\x00Z\x00\x00\x00[\x00\x00\x00\\x00\x00\x00]\x00\x00\x00^\x00\x00\x00_\x00\x00\x00`\x00\x00\x00a\x00\x00\x00b\x00\x00\x00c\x00\x00\x00d\x00\x00\x00e\x00\x00\x00f\x00\x00\x00g\x00\x00\x00h\x00\x00\x00i\x00\x00\x00j\x00\x00\x00k\x00\x00\x00l\x00\x00\x00m\x00\x00\x00n\x00\x00\x00o\x00\x00\x00p\x00\x00\x00] - SystemError: Negative size passed to PyBytes_FromStringAndSize
FAILED numcodecs/tests/test_pyzstd.py::test_pyzstd_streaming[\n\x00\x00\x00\x0b\x00\x00\x00\x0c\x00\x00\x00\r\x00\x00\x00\x0e\x00\x00\x00] - SystemError: Negative size passed to PyBytes_FromStringAndSize
FAILED numcodecs/tests/test_pyzstd.py::test_pyzstd_streaming[+\x00*\x00\x1c\x00'\x00\x1c\x00\x0e\x00\r\x00#\x00*\x00!\x00(\x00\x05\x00\x16\x00$\x00\x11\x00\x13\x00\n\x00(\x00\t\x00\x10\x00\x0c\x00-\x00\x04\x00\x08\x00\x07\x00\x1f\x00\x16\x00\x15\x00\x08\x00\x07\x00\t\x00\x13\x00(\x000\x00\x11\x00\x0e\x00\x1b\x00\x1a\x00\x1b\x00-\x00\x15\x00\x1c\x00&\x00\x0c\x00,\x00\n\x00\x11\x00"\x00\x1b\x00-\x00'\x00\x1b\x00\n\x00] - SystemError: Negative size passed to PyBytes_FromStringAndSize
FAILED numcodecs/tests/test_zstd.py::test_streaming_decompression - SystemError: Negative size passed to PyBytes_FromStringAndSize
===== 5 failed, 685 passed, 23 skipped, 27 deselected, 4 xfailed in 18.72s =====

@mkitti

cc @mkitti

Is this 32-bit Linux?

Yes, it is on GNU/Linux i386 (Debian sid).
Sorry for not having specified it in the initial description.

@d-v-b

Well there is an implicit cast there from size_t in the Zstd buffer to Py_ssize_t. I'm not sure why Python uses a signed integer there.

Anyways, that would suggest a large value for size_t. That should not occur as well.

I'm going to guess one of the 64-bit constants is not relevant or some other overflow issue.

@d-v-b , do we have 32-bit CI? If we still support 32-bit, we should have it. 32-bit Python should be sufficient.

here's our current parameterization in ci:

numcodecs/.github/workflows/ci.yaml

Line 17 in f42a233

platform: [ubuntu-latest, ubuntu-22.04-arm, windows-latest, macos-13, macos-14]

not sure if there's a 32 bit option, or if we need to make a special job for this via alpine linux and qemu

We do not need a 32-bit machine. We just need 32-bit Python, I think...

https://github.com/marketplace/actions/setup-miniconda#example-11-alternative-architectures

I guess there is a x86 version via setup miniconda action

You can also do x86 via setup-python action:
https://github.com/actions/setup-python

@avalentino

@avalentino , how did you install numcodecs?

I'm trying to update the debian package for numcodecs to the current latest version (v0.16.2).
Basically the process implied building the package from sources using only debian packages to satisfy dependencies.

The errors seem to be coming from https://github.com/zarr-developers/numcodecs/blob/main/numcodecs%2Fzstd.pyx#L229

I think the issue is that dest_size is a 32-bit size_t and the value from Zstd_getFrameContentSize is 64-bit. Truncation prevents the branch to my streaming function.

I'm pretty sure I see the bug. I encountered it when working on numcodecs.js, which is also a 32-bit system.

The issue is summarized on this slide of mine.

By the way, did you think this worked with prior versions of numcodecs?

Certainly the tests may have passed, but I think actually trying to decode these buffers would have failed.

I think this issue arose now because we added tests for this recently. However, I think the actual issue has existed for some time.

I'm realizing now that you are the Debian package maintainer, so I understand why you are building for i386.

Could you see if #782 at 786d31f works for you?

I was able to reproduce via an Alpine Linux chroot:
https://github.com/zarr-developers/numcodecs/actions/runs/17059583576/job/48363805347?pr=784

Could you see if #782 at 786d31f works for you?

thanks for the fix
I'm not sure that I will be able to test it by this morning but, for sure, I will provide some feedback ASAP.

Meanwhile we have found a issue in one of the xarray unit tests that could that could be related to this one.

I report below the relevant part but the full test log is at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1110804

You may want to give a look.

Please note that, this time, it is not on i386, it is on amd64

> =================================== FAILURES ===================================
> ___________ TestZarrDirectoryStore.test_chunk_encoding_with_dask[3] ____________
> 
> self = <xarray.tests.test_backends.TestZarrDirectoryStore object at 0x7fa25119edf0>
> 
>     @requires_dask
>     @pytest.mark.skipif(
>         ON_WINDOWS,
>         reason="Very flaky on Windows CI. Can re-enable assuming it starts consistently passing.",
>     )
>     def test_chunk_encoding_with_dask(self) -> None:
>         # These datasets DO have dask chunks. Need to check for various
>         # interactions between dask and zarr chunks
>         ds = xr.DataArray((np.arange(12)), dims="x", name="var1").to_dataset()
>     
>         # - no encoding specified -
>         # zarr automatically gets chunk information from dask chunks
>         ds_chunk4 = ds.chunk({"x": 4})
>         with self.roundtrip(ds_chunk4) as actual:
>             assert (4,) == actual["var1"].encoding["chunks"]
>     
>         # should fail if dask_chunks are irregular...
>         ds_chunk_irreg = ds.chunk({"x": (5, 4, 3)})
>         with pytest.raises(ValueError, match=r"uniform chunk sizes."):
>             with self.roundtrip(ds_chunk_irreg) as actual:
>                 pass
>     
>         # should fail if encoding["chunks"] clashes with dask_chunks
>         badenc = ds.chunk({"x": 4})
>         badenc.var1.encoding["chunks"] = (6,)
>         with pytest.raises(ValueError, match=r"named 'var1' would overlap"):
>             with self.roundtrip(badenc) as actual:
>                 pass
>     
>         # unless...
> >       with self.roundtrip(badenc, save_kwargs={"safe_chunks": False}) as actual:
> 
> xarray/tests/test_backends.py:2544: 
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> /usr/lib/python3.13/contextlib.py:141: in __enter__
>     return next(self.gen)
> xarray/tests/test_backends.py:2314: in roundtrip
>     self.save(data, store_target, **save_kwargs)
> xarray/tests/test_backends.py:2296: in save
>     return dataset.to_zarr(store=store_target, **kwargs, **self.version_kwargs)
> xarray/core/dataset.py:2270: in to_zarr
>     return to_zarr(  # type: ignore[call-overload,misc]
> xarray/backends/api.py:2240: in to_zarr
>     writes = writer.sync(
> xarray/backends/common.py:358: in sync
>     delayed_store = chunkmanager.store(
> xarray/namedarray/daskmanager.py:247: in store
>     return store(
> /usr/lib/python3/dist-packages/dask/array/core.py:1245: in store
>     compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
> /usr/lib/python3/dist-packages/dask/base.py:397: in compute_as_if_collection
>     return schedule(dsk2, keys, **kwargs)
> /usr/lib/python3/dist-packages/dask/threaded.py:91: in get
>     results = get_async(
> /usr/lib/python3/dist-packages/dask/local.py:516: in get_async
>     raise_exception(exc, tb)
> /usr/lib/python3/dist-packages/dask/local.py:324: in reraise
>     raise exc
> /usr/lib/python3/dist-packages/dask/local.py:229: in execute_task
>     result = task(data)
> /usr/lib/python3/dist-packages/dask/_task_spec.py:745: in __call__
>     return self.func(*new_argspec)
> /usr/lib/python3/dist-packages/dask/_task_spec.py:171: in _execute_subgraph
>     res = execute_graph(final, keys=[outkey])
> /usr/lib/python3/dist-packages/dask/_task_spec.py:984: in execute_graph
>     cache[key] = node(cache)
> /usr/lib/python3/dist-packages/dask/_task_spec.py:745: in __call__
>     return self.func(*new_argspec)
> /usr/lib/python3/dist-packages/dask/array/core.py:4539: in store_chunk
>     return load_store_chunk(x, out, index, lock, return_stored, False)
> /usr/lib/python3/dist-packages/dask/array/core.py:4521: in load_store_chunk
>     out[index] = x
> /usr/lib/python3/dist-packages/zarr/core/array.py:2524: in __setitem__
>     self.set_orthogonal_selection(pure_selection, value, fields=fields)
> /usr/lib/python3/dist-packages/zarr/_compat.py:43: in inner_f
>     return f(*args, **kwargs)
> /usr/lib/python3/dist-packages/zarr/core/array.py:2980: in set_orthogonal_selection
>     return sync(
> /usr/lib/python3/dist-packages/zarr/core/sync.py:163: in sync
>     raise return_result
> /usr/lib/python3/dist-packages/zarr/core/sync.py:119: in _runner
>     return await coro
> /usr/lib/python3/dist-packages/zarr/core/array.py:1415: in _set_selection
>     await self.codec_pipeline.write(
> /usr/lib/python3/dist-packages/zarr/core/codec_pipeline.py:479: in write
>     await concurrent_map(
> /usr/lib/python3/dist-packages/zarr/core/common.py:68: in concurrent_map
>     return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
> /usr/lib/python3/dist-packages/zarr/core/common.py:66: in run
>     return await func(*item)
> /usr/lib/python3/dist-packages/zarr/core/codec_pipeline.py:373: in write_batch
>     chunk_array_decoded = await self.decode_batch(
> /usr/lib/python3/dist-packages/zarr/core/codec_pipeline.py:185: in decode_batch
>     chunk_bytes_batch = await bb_codec.decode(
> /usr/lib/python3/dist-packages/zarr/abc/codec.py:129: in decode
>     return await _batching_helper(self._decode_single, chunks_and_specs)
> /usr/lib/python3/dist-packages/zarr/abc/codec.py:407: in _batching_helper
>     return await concurrent_map(
> /usr/lib/python3/dist-packages/zarr/core/common.py:68: in concurrent_map
>     return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
> /usr/lib/python3/dist-packages/zarr/core/common.py:66: in run
>     return await func(*item)
> /usr/lib/python3/dist-packages/zarr/abc/codec.py:420: in wrap
>     return await func(chunk, chunk_spec)
> /usr/lib/python3/dist-packages/zarr/codecs/zstd.py:78: in _decode_single
>     return await asyncio.to_thread(
> /usr/lib/python3.13/asyncio/threads.py:25: in to_thread
>     return await loop.run_in_executor(None, func_call)
> /usr/lib/python3.13/concurrent/futures/thread.py:59: in run
>     result = self.fn(*self.args, **self.kwargs)
> /usr/lib/python3/dist-packages/zarr/core/buffer/cpu.py:213: in as_numpy_array_wrapper
>     return prototype.buffer.from_bytes(func(buf.as_numpy_array()))
> numcodecs/zstd.pyx:261: in numcodecs.zstd.Zstd.decode
>     ???
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
> 
> >   ???
> E   RuntimeError: Zstd decompression error: invalid input data
> 
> numcodecs/zstd.pyx:191: RuntimeError

Could you see if #782 at 786d31f works for you?

I confirm that it works on Debian sid i386
Thanks

Fix Zstd decompression negative size issue on 32-bit platforms (Fixes #…

Navigation Menu

Repository navigation

ZStd test fail on 32bit atchitectures #781

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ZStd test fail on 32bit atchitectures #781

Description

Activity

d-v-b commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

avalentino commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

d-v-b commented on Aug 18, 2025

d-v-b commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

avalentino commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

mkitti commented on Aug 18, 2025

mkitti commented on Aug 19, 2025

mkitti commented on Aug 19, 2025

mkitti commented on Aug 19, 2025

avalentino commented on Aug 19, 2025

avalentino commented on Aug 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions