Skip to content

read_table_parquet raises KeyError when a column is absent from the stored astropy table_meta_yaml #20639

Open

Description

@scttfrdmn

Description

Table.read(..., format="parquet") raises KeyError: '_healpix_29' on Gaia DR3's HATS-partitioned parquet files.

Root cause

In astropy/io/misc/parquet.py, read_table_parquet (lines 313–317 at v8.0.1, lines 332–336 on main):

header_cols = {x["name"]: x for x in meta_hdr["datatype"]}
for col in table.columns.values():
    for attr in ("description", "format", "unit", "meta"):
        if attr in header_cols[col.name]:

This looks up every column of the file in the table_meta_yaml header. Here the header was written by astropy, and a tool then added a column without rewriting the yaml: hats-import adds _healpix_29. Any column missing from the yaml makes the read fail outright, even though the data is fine. pyarrow reads the same file without complaint.

Suggested fix

Treat a column that has no header entry as having no stored attributes:

col_hdr = header_cols.get(col.name, {})
if attr in col_hdr:
    setattr(col, attr, col_hdr[attr])

Reproduction

  1. Read any Gaia DR3 HATS partition file with astropy.table.Table.read(path, format="parquet") on astropy 8.0.1.
  2. It raises KeyError('_healpix_29').
  3. pyarrow.parquet.read_table(path) on the same file succeeds.

A minimal synthetic case that should hit the same lookup (not run; derived from the code above):

  1. Write a Table to parquet with astropy.
  2. Add a column with pyarrow, keeping the original schema metadata.
  3. Read the file back with astropy.

Versions: astropy 8.0.1 (conda-forge), pyarrow from the same env, Linux aarch64 and x86_64.

Activity

  1. akshat-lakhera commented on Oct 19, 2026

    @akshat-lakhera
  2. bsipocz commented on Oct 19, 2026

    @bsipocz
  3. bsipocz commented on Oct 19, 2026

    @bsipocz
    Member

    @scttfrdmn - could you please add a link to one of these files, so I know we're looking at the same ones?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions