Skip to content

[C++][Python][Parquet] pyarrow.lib.ArrowInvalid: Invalid number of indices: 0 when reading a parquet file #47981

Closed

Description

@TomAugspurger

Describe the bug, including details regarding any error messages, version, and platform.

Something about this parquet file form https://github.com/Parquet/parquet-compatibility/ causes an exception while reading with pyarrow 22.0.0:

import urllib.request
import pathlib
import pyarrow.parquet as pq

p = pathlib.Path("nation.impala.parquet")
if not p.exists():
    urllib.request.urlretrieve(
        "https://github.com/Parquet/parquet-compatibility/raw/master/parquet-testdata/impala/1.1.1-NONE/nation.impala.parquet",
        p
    )

pq.read_table(p)

which raises with

Traceback (most recent call last):
  File "/Users/toaugspurger/gh/dask/dask/bug.py", line 12, in <module>
    pq.read_table(p)
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1899, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/toaugspurger/gh/dask/.venv/lib/python3.12/site-packages/pyarrow/parquet/core.py", line 1538, in read
    table = self._dataset.to_table(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 589, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3969, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Invalid number of indices: 0

pyarrow 21.0.0 was able to read that file.

Component(s)

Parquet, Python

Activity

  1. raulcd commented on Oct 28, 2025

    @raulcd
    Member

    I've been able to bisect and find the first commit that produces the failure for the above parquet file.
    The commit: 33d1f32 fails reading the file.
    The previous commit: 353a0a2 does not fail reading the file.

    This was fixing some undefined behavior found on fuzzing, we should investigate further the file to understand what is the problem but this will give us a hint. It might probably tell us that there's a problem with the file generation.

    CC @pitrou

  2. self-assigned this
    on Oct 28, 2025
  3. pitrou commented on Oct 29, 2025

    @pitrou
    Member

    Ok, I looked at this.

    These are extremely old Parquet files that were generated while the Parquet format was still being designed.

    One of the columns, of length 25, has RLE_BIT_PACKED-encoded dictionary integers over a bit width of 5. The RLE_BIT_PACKED data is 17 bytes and contains a single bit-packed run with count=4, which is really 32 values. But 17 bytes (of which 1 byte header) cannot encode 32 5-bit values (only 25). So, while the RLE_BIT_PACKED data can technically encode the 25 integers, it lacks the padding implicit in the spec.

    Now, we could relax this a bit in our decoder, but the question is whether more recent files share this property.

  4. added a commit that references this issue on Oct 29, 2025
  5. changed the title [-]`pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/-] [+][C++][Python][Parquet] `pyarrow.lib.ArrowInvalid: Invalid number of indices: 0` when reading a parquet file[/+] on Oct 29, 2025
  6. TomAugspurger commented on Oct 31, 2025

    @TomAugspurger
    ContributorAuthor

    FWIW, this just came up in one of Dask's unit tests. I suspect it's not critical, especially if this file is arguably invalid.

  7. thisisnic commented on Nov 6, 2025

    @thisisnic
    Member

    The linked R issue here looks like it's happening with (AFAIK) more recent files: #48066

  8. added a commit that references this issue on Nov 10, 2025
  9. pitrou commented on Nov 10, 2025

    @pitrou
    Member

    Thanks for the pointer @thisisnic . It looks like those files were produced with Polars.

  10. added a commit that references this issue on Nov 10, 2025
  11. pryvkin10x commented on Nov 10, 2025

    @pryvkin10x

    We're seeing a different error message, in addition to this one, on similar files that were also generated with that version of polars. Can confirm that they also have this truncated RLE issue.

    Is this error message caused by the same underlying issue?
    OSError: Unexpected end of stream
    scverse/spatialdata-io#334

  12. pryvkin10x commented on Nov 11, 2025

    @pryvkin10x

    Can confirm that pitrou's commit above fixes the Unexpected end of stream error as well. Thanks.

  13. pitrou commented on Nov 12, 2025

    @pitrou
    Member

    Thanks for the heads up @pryvkin10x . This is good to know.

  14. added 3 commits that reference this issue on Nov 12, 2025
  15. added this to the 23.0.0 milestone on Nov 13, 2025
  16. pitrou commented on Nov 13, 2025

    @pitrou
    Member

    Issue resolved by pull request 47992
    #47992

  17. added a commit that references this issue on May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions