PyArrow OSError: Unexpected end of stream

Since the release of pyarrow 22.0.0 the CI started failing, with errors similar to the one below. Please note that the error doesn't seem to be spatialdata-related. Full CI run here.

tests/test_xenium.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/spatialdata_io/_utils.py:59: in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:192: in xenium
    return_values = _get_tables_and_circles(path, cells_as_circles, specs, gex_only)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:570: in _get_tables_and_circles
    metadata = pd.read_parquet(path / XeniumKeys.CELL_METADATA_FILE)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:669: in read_parquet
    return impl.read(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:265: in read
    pa_table = self.api.parquet.read_table(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1899: in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1538: in read
    table = self._dataset.to_table(
pyarrow/_dataset.pyx:589: in pyarrow._dataset.Dataset.to_table
    ???
pyarrow/_dataset.pyx:3969: in pyarrow._dataset.Scanner.to_table
    ???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OSError: Unexpected end of stream

pyarrow/error.pxi:92: OSError

The pyarrow release also broke some tests in spatialdata, but these have been already fixed here: scverse/spatialdata#1003, and a release will be made soon.

Two of the new Xenium datasets appear to be corrupted. Here is a script to reproduce the problem with the latest pyarrow.

##
import pyarrow.parquet as pq

# does not work
pq.read_table('data/Xenium_V1_Protein_Human_Kidney_tiny_outs/cells.parquet')
pq.read_table('data/Xenium_V1_MultiCellSeg_Human_Ovary_tiny_outs/cells.parquet')

# works
pq.read_table('data/Xenium_V1_Human_Ovary_tiny_outs/cells.parquet')
pq.read_table('data/Xenium_V1_human_Lung_2fov_outs/cells.parquet')
pq.read_table('data/Xenium_V1_human_Breast_2fov_outs/cells.parquet')

This change in pyarrow will fix this: apache/arrow@main...pitrou:arrow:gh47981-pq-impala-file
I recommend staying at pyarrow v21 if possible until that fix gets released.

Thanks for the info! I see the corresponding PR (apache/arrow#47992) has been merged already, but a release is not out yet.

Since the above impacts the Xenium tests but should not impact users (since users would not use the truncated parquet files), I will skip the tests until the new pyarrow release is available.

I will actually just add pyarrow==21 in the test extra dependency group.

Navigation Menu

Repository navigation

PyArrow OSError: Unexpected end of stream #334

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PyArrow OSError: Unexpected end of stream #334

Description

Activity

LucaMarconato commented on Oct 27, 2025

pryvkin10x commented on Nov 11, 2025

LucaMarconato commented on Dec 17, 2025

LucaMarconato commented on Dec 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions