You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][Python][Parquet] pyarrow.lib.ArrowInvalid: Invalid number of indices: 0 when reading a parquet file #47981
importurllib.requestimportpathlibimportpyarrow.parquetaspqp=pathlib.Path("nation.impala.parquet")
ifnotp.exists():
urllib.request.urlretrieve(
"https://github.com/Parquet/parquet-compatibility/raw/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet",
p
)
pq.read_table(p)
which raises with
Traceback (most recent call last):
File "/Users/toaugspurger/gh/dask/dask/bug.py", line 12, in <module>
pq.read_table(p)
File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1899, in read_tablereturn dataset.read(columns=columns, use_threads=use_threads,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1538, in read
table =self._dataset.to_table(
^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 589, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 3969, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_statuspyarrow.lib.ArrowInvalid: Invalid number of indices: 0
I've been able to bisect and find the first commit that produces the failure for the above parquet file.
The commit: 33d1f32 fails reading the file.
The previous commit: 353a0a2 does not fail reading the file.
This was fixing some undefined behavior found on fuzzing, we should investigate further the file to understand what is the problem but this will give us a hint. It might probably tell us that there's a problem with the file generation.
These are extremely old Parquet files that were generated while the Parquet format was still being designed.
One of the columns, of length 25, has RLE_BIT_PACKED-encoded dictionary integers over a bit width of 5. The RLE_BIT_PACKED data is 17 bytes and contains a single bit-packed run with count=4, which is really 32 values. But 17 bytes (of which 1 byte header) cannot encode 32 5-bit values (only 25). So, while the RLE_BIT_PACKED data can technically encode the 25 integers, it lacks the padding implicit in the spec.
Now, we could relax this a bit in our decoder, but the question is whether more recent files share this property.
changed the title [-]`pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/-][+][C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/+]on Oct 29, 2025
We're seeing a different error message, in addition to this one, on similar files that were also generated with that version of polars. Can confirm that they also have this truncated RLE issue.
Is this error message caused by the same underlying issue? OSError: Unexpected end of stream scverse/spatialdata-io#334
Activity
raulcd commentedon Oct 28, 2025
I've been able to bisect and find the first commit that produces the failure for the above parquet file.
The commit: 33d1f32 fails reading the file.
The previous commit: 353a0a2 does not fail reading the file.
This was fixing some undefined behavior found on fuzzing, we should investigate further the file to understand what is the problem but this will give us a hint. It might probably tell us that there's a problem with the file generation.
CC @pitrou
pitrou commentedon Oct 29, 2025
Ok, I looked at this.
These are extremely old Parquet files that were generated while the Parquet format was still being designed.
One of the columns, of length 25, has RLE_BIT_PACKED-encoded dictionary integers over a bit width of 5. The RLE_BIT_PACKED data is 17 bytes and contains a single bit-packed run with count=4, which is really 32 values. But 17 bytes (of which 1 byte header) cannot encode 32 5-bit values (only 25). So, while the RLE_BIT_PACKED data can technically encode the 25 integers, it lacks the padding implicit in the spec.
Now, we could relax this a bit in our decoder, but the question is whether more recent files share this property.
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…
[-]`pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/-][+][C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/+]TomAugspurger commentedon Oct 31, 2025
FWIW, this just came up in one of Dask's unit tests. I suspect it's not critical, especially if this file is arguably invalid.
thisisnic commentedon Nov 6, 2025
The linked R issue here looks like it's happening with (AFAIK) more recent files: #48066
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…
pitrou commentedon Nov 10, 2025
Thanks for the pointer @thisisnic . It looks like those files were produced with Polars.
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…
pryvkin10x commentedon Nov 10, 2025
We're seeing a different error message, in addition to this one, on similar files that were also generated with that version of polars. Can confirm that they also have this truncated RLE issue.
Is this error message caused by the same underlying issue?
OSError: Unexpected end of streamscverse/spatialdata-io#334
pryvkin10x commentedon Nov 11, 2025
Can confirm that pitrou's commit above fixes the
Unexpected end of streamerror as well. Thanks.pitrou commentedon Nov 12, 2025
Thanks for the heads up @pryvkin10x . This is good to know.
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…
GH-47981: [C++][Parquet] Add compatibility with non-compliant RLE str…
pitrou commentedon Nov 13, 2025
Issue resolved by pull request 47992
#47992
apacheGH-47981: [C++][Parquet] Add compatibility with non-compliant R…