fits.BinTableHDU error with uint8 data

Using astropy 1.0.6 and numpy 1.10.1, this fails:

import numpy as np
from astropy.io import fits

#- Fails
dtype = [('x', '>f4'), ('y', 'uint8')]
data = np.zeros(10, dtype=dtype)
hdu = fits.BinTableHDU(data)

#- Fails
dtype = [('x', '>f4'), ('y', 'uint8')]
data = np.zeros(10, dtype=dtype)
hdu = fits.BinTableHDU(data, uint=True)

Both fail with:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sbailey/anaconda/envs/fitsbug/lib/python2.7/site-packages/astropy/io/fits/hdu/table.py", line 327, in __init__
    new_data[k] = data[k] - bzeros[dtype.itemsize]
KeyError: 1

The problem is from the uint8 type in the numpy structured array; using int8 works:

#- Works
dtype = [('x', '>f4'), ('y', 'int8')]
data = np.zeros(10, dtype=dtype)
hdu = fits.BinTableHDU(data)

Surprisingly this also works:

dtype = [('x', '>f4'), ('y', 'uint8', (3,4))]
data = np.zeros(10, dtype=dtype)
hdu = fits.BinTableHDU(data, uint=True)

I tripped across this from reading a fits file that had uint8 data and then not being able to write the data back out into a new file so simply avoiding uint8 is not an option.

FWIW in this

hdu = fits.BinTableHDU(data, uint=True)

the uint=True doesn't really do anything. It's an internal detail used only when reading files. Normally unsigned ints are supported in new tables transparently without having to do anything special.

Anyways, I see the problem here. Turns out the 8-bit integers are natively unsigned in FITS tables, and signed 8-bit integers have to be represented using TZERO. This is the opposed of all other integer types. Oof--after all this time FITS still finds new ways to horrify me.

Turns out this applies to images as well--BITPIX = 8 implies unsigned ints, not signed ints. I'm not exactly sure what PyFITS does in this case but I'll have to check.

This should be fixed (with tests) for both images and tables, and should include a clarification in the documentation that the sense of "pseudo-unsigned ints" is reversed for 1-byte ints (and that signed bytes should be considered "pseudo-signed ints").

Okay, I confirmed that for images unsigned bytes are handled properly, but signed bytes simply aren't supported at all (though it should be possible to support them with bzero).

Surprisingly this also works:

dtype = [('x', '>f4'), ('y', 'uint8', (3,4))]
data = np.zeros(10, dtype=dtype)
hdu = fits.BinTableHDU(data, uint=True)

Suprising indeed. I know why it is though. On this line it checks if the type is unsigned int by checking the dtype.kind attribute. However, for a multi-dimensional type this is 'V', not 'u' (or whatever the base kind happens to be). So that's a minor bug in its own right.

fix for the fits.BinTableHDU error with uint8 data astropy#4267

@sbailey

@sbailey and @embray i have come with fix please review it
AMAN3003@ef9ec6d

@embray

@embray and @sbailey key erroris raised when dictinary key is unable to find its value. here 1:because 10 is in categories of unint8
also if we pass int8 its forking fine because in that case it not entering in bug part of the code
also @embray is right it has nothing to do with hdu = fits.BinTableHDU(data, uint=True) &
hdu = fits.BinTableHDU(data)

@embray

@embray can i add test for it

@AMAN3003

@AMAN3003 This fix is not correct though, because as I pointed out unsigned bytes are stored natively in FITS, while it's signed bytes that need special handling, converse to all other byte types. This needs to be fixed for images as well as tables. Now that I think about it it's actually a pretty serious bug, but somehow it hasn't come up before.

Can someone confirm whether a FITS file containing signed bytes created by Astropy is read correctly by other software, say, CFITSIO or IRAF?

Using code snippet below to test reading/writing images, I found:

np.int8
- astropy.io.fits was not able to write data with dtype=np.int8 (i.e. raised exception)
- fitsio (wrappering cfitsio) was able to write dtype=np.int8, and astropy was then able to read it but it came back as np.float32 instead of np.int8
np.uint64
- fitsio incorrectly read np.uint64 data written by astropy (no exception, but wrong data)
- astropy was able to self consistently read np.uint64 data that it had written
- fitsio wrote a corrupted np.uint64 file that neither astropy not fitsio were able to read
all other combinations of reading and writing np.(u)int* worked

I'm not sure if that is the test you were requesting though, but wow this is a mess.

from astropy.io import fits
import fitsio
import numpy as np

image = np.random.uniform(0,1000, size=(5,5))

def test_readwrite(writefunc, image):
    for dtype in (
            np.int8, np.int16, np.int32, np.int64,
            np.uint8, np.uint16, np.uint32, np.uint64,
        ):
        x = image.astype(dtype)
        try:
            writefunc('blat.fits', x, overwrite=True)
        except (IOError, KeyError):
            print("Unable to write image with dtype =", dtype)
            continue

        print('Testing dtype ', dtype)
        try:
            a = fits.getdata('blat.fits')
            print('    astropy:', a.dtype, np.all(x==a))
        except AttributeError:
            print("    ERROR: astropy failed to read dtype", dtype)

        try:
            b = fitsio.read('blat.fits')
            print('    fitsio: ', b.dtype, np.all(x==b))
        except IOError:
            print("    ERROR: fitsio failed to read dtype", dtype)


test_readwrite(fits.writeto, image)
test_readwrite(fitsio.write, image)

Results:

Unable to write image with dtype = <type 'numpy.int8'>
Testing dtype  <type 'numpy.int16'>
    astropy: >i2 True
    fitsio:  int16 True
Testing dtype  <type 'numpy.int32'>
    astropy: >i4 True
    fitsio:  int32 True
Testing dtype  <type 'numpy.int64'>
    astropy: >i8 True
    fitsio:  int64 True
Testing dtype  <type 'numpy.uint8'>
    astropy: uint8 True
    fitsio:  uint8 True
Testing dtype  <type 'numpy.uint16'>
    astropy: uint16 True
    fitsio:  uint16 True
Testing dtype  <type 'numpy.uint32'>
    astropy: uint32 True
    fitsio:  uint32 True
Testing dtype  <type 'numpy.uint64'>
    astropy: uint64 True
    fitsio:  int64 False

In [65]: test_readwrite(fitsio.write, image)
Testing dtype  <type 'numpy.int8'>
    astropy: float32 True
    fitsio:  int8 True
Testing dtype  <type 'numpy.int16'>
    astropy: >i2 True
    fitsio:  int16 True
Testing dtype  <type 'numpy.int32'>
    astropy: >i4 True
    fitsio:  int32 True
Testing dtype  <type 'numpy.int64'>
    astropy: >i4 True
    fitsio:  int32 True
Testing dtype  <type 'numpy.uint8'>
    astropy: uint8 True
    fitsio:  uint8 True
Testing dtype  <type 'numpy.uint16'>
    astropy: uint16 True
    fitsio:  uint16 True
Testing dtype  <type 'numpy.uint32'>
    astropy: uint32 True
    fitsio:  uint32 True
Testing dtype  <type 'numpy.uint64'>
    ERROR: astropy failed to read dtype <type 'numpy.uint64'>
    ERROR: fitsio failed to read dtype <type 'numpy.uint64'>

@sbailey

@sbailey Thanks for doing those tests, that's perfect!

As for uint64, I believe CFITSIO has lacked support for that for a while, so the problem is in CFITSIO I think (but someone please correct me if that's wrong--I'm not sure--I just remember that being the case some time ago).

But for signed/unsigned bytes I think astropy.io.fits is definitely broken. Aman's attempt at a fix had the right idea, but unfortunately it needs to do more to treat signed/unsigned bytes as a separate special case (and needs to work for images as well as tables).

@saimn

All examples from the original post work, but I'm confused of what the desired test results should be for #4267 (comment) . I've edited the code, so it's copy-pasteable.
@saimn or @MSeifert04 - could you have a quick look of this and possible close it?

The original issue with uint8 data in BinTableHDU seems to be solved.
The only issue that remains I think, is with int8 data. Related to the post above, fitsio can write a file with int8 data, with the file is bad (BITPIX=64, BZERO=9223372036854775808). But int8 is another issue, probably not easy to support and not too common, I think we can close this one.

Navigation Menu

Repository navigation

fits.BinTableHDU error with uint8 data #4267

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fits.BinTableHDU error with uint8 data #4267

Description

Activity

embray commented on Nov 16, 2015

embray commented on Nov 16, 2015

embray commented on Nov 16, 2015

embray commented on Nov 16, 2015

embray commented on Nov 16, 2015

AMAN3003 commented on Feb 13, 2016

AMAN3003 commented on Feb 13, 2016

AMAN3003 commented on Feb 13, 2016

embray commented on Feb 17, 2016

sbailey commented on Feb 17, 2016

embray commented on Feb 18, 2016

bsipocz commented on Mar 18, 2019

saimn commented on Mar 18, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions