FITS duplicate keywords: fast parser last-wins vs Header first-wins (shape/BSCALE disagree) #20798

Description

astropy.io.fits has two FITS header parsers that disagree when a standard keyword is repeated:

• The fast Cython parser (parse_header in astropy/io/fits/_utils.pyx), used by _BasicHeader / file-backed fits.open(), keeps the last occurrence (plain dict assignment).
• The public Header class keeps the first occurrence, as documented in its docstring and in docs/io/fits/usage/headers.rst.

fits.open() builds the HDU from the fast parser, then later rebuilds hdu.header with Header.fromstring(). As a result, the values shown on hdu.header (e.g. NAXIS1, BITPIX, BSCALE) can differ from the values actually used to interpret the array. On the same bytes, fits.open() and fits.HDUList.fromstring() can also return different shapes, because the buffer path does not go through _BasicHeader.

This is a correctness / consistency bug (duplicate keywords are discouraged by the FITS standard but still accepted silently today).

Expected behavior

Both parsers should use the same rule. The public Header contract is first-wins, so the fast parser should keep the first occurrence of a repeated keyword. After that, hdu.header["NAXIS1"] should match hdu.shape, and fits.open() / fromstring() should agree on the same bytes.

How to Reproduce

import io
import numpy as np
from astropy.io import fits
from astropy.io.fits import Card
from astropy.io.fits.header import _BasicHeader

def card(keyword, value):
    if keyword == "SIMPLE":
        value = True
    else:
        value = int(value)
    return Card(keyword, value).image.encode("ascii")

def header(cards):
    raw = b"".join(card(k, v) for k, v in cards) + b"END".ljust(80)
    return raw + b" " * (-len(raw) % 2880)

def padded(blob):
    return blob + b"\0" * (-len(blob) % 2880)

h = header([("SIMPLE", "T"), ("BITPIX", "16"), ("NAXIS", "1"),
            ("NAXIS1", "3"), ("NAXIS1", "6")])
print("Header:", fits.Header.fromstring(h.decode("ascii"))["NAXIS1"])
_, basic = _BasicHeader.fromfile(io.BytesIO(h))
print("_BasicHeader:", basic["NAXIS1"])
blob = padded(h + np.arange(1, 7, dtype=">i2").tobytes())
with fits.open(io.BytesIO(blob)) as hdul:
    print("fits.open header/shape/data:", hdul[0].header["NAXIS1"], hdul[0].shape, hdul[0].data)
with fits.HDUList.fromstring(blob) as hdul:
    print("fromstring header/shape/data:", hdul[0].header["NAXIS1"], hdul[0].shape, hdul[0].data)

Observed on astropy 8.0.2:

Header: 3
_BasicHeader: 6
fits.open header/shape/data: 3 (6,) [1 2 3 4 5 6]
fromstring header/shape/data: 3 (3,) [1 2 3]

Fix

Make parse_header keep the first occurrence (if key not in cards) so it matches Header. I have a minimal patch plus two regression tests ready and will open a PR linking this issue.

AI assistance note

This issue was identified with AI-assisted static code review and checked against the source before reporting; happy to answer questions or help with a fix. A human reviewed the finding, reproduced it locally on astropy 8.0.2, wrote/ran the proposed fix and regression tests, and is accountable for this report (contact: [email protected] / @ucsurf), per the Astropy generative-AI policy.

System Details

astropy 8.0.2 (also checked against current main)
Python 3.13.5
Linux