io.fits and Lomb-Scargle test failures on ppc64le

@olebole

I noticed the following test failures on ppc64le:

__________________ TestChecksumFunctions.test_groups_hdu_data __________________

self = <astropy.io.fits.tests.test_checksum.TestChecksumFunctions object at 0x4111f63668>

    def test_groups_hdu_data(self):
        imdata = np.arange(100.0)
        imdata.shape = (10, 1, 1, 2, 5)
        pdata1 = np.arange(10) + 0.1
        pdata2 = 42
        x = fits.hdu.groups.GroupData(imdata, parnames=['abc', 'xyz'],
                                      pardata=[pdata1, pdata2], bitpix=-32)
        hdu = fits.GroupsHDU(x)
        hdu.writeto(self.temp('tmp.fits'), overwrite=True, checksum=True)
        with fits.open(self.temp('tmp.fits'), checksum=True) as hdul:
            assert comparerecords(hdul[0].data, hdu.data)
            assert 'CHECKSUM' in hdul[0].header
>           assert hdul[0].header['CHECKSUM'] == '3eDQAZDO4dDOAZDO'
E           AssertionError: assert 'AAGND29NA8GNA89N' == '3eDQAZDO4dDOAZDO'
E             - 3eDQAZDO4dDOAZDO
E             + AAGND29NA8GNA89N

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_checksum.py:145: AssertionError
______________________ TestDiff.test_different_table_data ______________________

self = <astropy.io.fits.tests.test_diff.TestDiff object at 0x41129ea048>

    def test_different_table_data(self):
        """
        Test diffing table data on columns of several different data formats
        and dimensions.
        """
    
        ca1 = Column('A', format='L', array=[True, False])
        ca2 = Column('B', format='X', array=[[0], [1]])
        ca3 = Column('C', format='4I', dim='(2, 2)',
                     array=[[0, 1, 2, 3], [4, 5, 6, 7]])
        ca4 = Column('D', format='J', bscale=2.0, array=[0.0, 2.0])
        ca5 = Column('E', format='A3', array=['abc', 'def'])
        ca6 = Column('F', format='E', unit='m', array=[0.0, 1.0])
        ca7 = Column('G', format='D', bzero=-0.1, array=[0.0, 1.0])
        ca8 = Column('H', format='C', array=[0.0+1.0j, 2.0+3.0j])
        ca9 = Column('I', format='M', array=[4.0+5.0j, 6.0+7.0j])
        ca10 = Column('J', format='PI(2)', array=[[0, 1], [2, 3]])
    
        cb1 = Column('A', format='L', array=[False, False])
        cb2 = Column('B', format='X', array=[[0], [0]])
        cb3 = Column('C', format='4I', dim='(2, 2)',
                     array=[[0, 1, 2, 3], [5, 6, 7, 8]])
        cb4 = Column('D', format='J', bscale=2.0, array=[2.0, 2.0])
        cb5 = Column('E', format='A3', array=['abc', 'ghi'])
        cb6 = Column('F', format='E', unit='m', array=[1.0, 2.0])
        cb7 = Column('G', format='D', bzero=-0.1, array=[2.0, 3.0])
        cb8 = Column('H', format='C', array=[1.0+1.0j, 2.0+3.0j])
        cb9 = Column('I', format='M', array=[5.0+5.0j, 6.0+7.0j])
        cb10 = Column('J', format='PI(2)', array=[[1, 2], [3, 4]])
    
        ta = BinTableHDU.from_columns([ca1, ca2, ca3, ca4, ca5, ca6, ca7,
                                       ca8, ca9, ca10])
        tb = BinTableHDU.from_columns([cb1, cb2, cb3, cb4, cb5, cb6, cb7,
                                       cb8, cb9, cb10])
    
        diff = TableDataDiff(ta.data, tb.data, numdiffs=20)
        assert not diff.identical
        # The column definitions are the same, but not the column values
        assert diff.diff_columns == ()
        assert diff.diff_values[0] == (('A', 0), (True, False))
        assert diff.diff_values[1] == (('B', 1), ([1], [0]))
        assert diff.diff_values[2][0] == ('C', 1)
        assert (diff.diff_values[2][1][0] == [[4, 5], [6, 7]]).all()
        assert (diff.diff_values[2][1][1] == [[5, 6], [7, 8]]).all()
        assert diff.diff_values[3] == (('D', 0), (0, 2.0))
        assert diff.diff_values[4] == (('E', 1), ('def', 'ghi'))
        assert diff.diff_values[5] == (('F', 0), (0.0, 1.0))
        assert diff.diff_values[6] == (('F', 1), (1.0, 2.0))
        assert diff.diff_values[7] == (('G', 0), (0.0, 2.0))
        assert diff.diff_values[8] == (('G', 1), (1.0, 3.0))
>       assert diff.diff_values[9] == (('H', 0), (0.0+1.0j, 1.0+1.0j))
E       AssertionError: assert (('I', 0), ((4+5j), (5+5j))) == (('H', 0), (1j, (1+1j)))
E         At index 0 diff: ('I', 0) != ('H', 0)
E         Use -v to get the full diff

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_diff.py:556: AssertionError
__________________ TestGroupsFunctions.test_create_groupdata ___________________

self = <astropy.io.fits.tests.test_groups.TestGroupsFunctions object at 0x4111dcb9b0>

    def test_create_groupdata(self):
        """
        Basic test for creating GroupData from scratch.
        """
    
        imdata = np.arange(100.0)
        imdata.shape = (10, 1, 1, 2, 5)
        pdata1 = np.arange(10, dtype=np.float32) + 0.1
        pdata2 = 42.0
        x = fits.hdu.groups.GroupData(imdata, parnames=['abc', 'xyz'],
                                      pardata=[pdata1, pdata2], bitpix=-32)
        assert x.parnames == ['abc', 'xyz']
        assert (x.par('abc') == pdata1).all()
        assert (x.par('xyz') == ([pdata2] * len(x))).all()
>       assert (x.data == imdata).all()
E       assert False
E        +  where False = <built-in method all of numpy.ndarray object at 0x410ff49f30>()
E        +    where <built-in method all of numpy.ndarray object at 0x410ff49f30> = array([[[[[ 0...dtype=float32) == array([[[[[ 0...98., 99.]]]]])
E             Use -v to get the full diff.all

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_groups.py:151: AssertionError
______________ TestImageFunctions.test_scale_bzero_with_int_data _______________

self = <astropy.io.fits.tests.test_image.TestImageFunctions object at 0x4111f51a90>

    def test_scale_bzero_with_int_data(self):
        """
        Regression test for https://github.com/astropy/astropy/issues/4600
        """
    
        a = np.arange(100, 200, dtype=np.int16)
    
        hdu1 = fits.PrimaryHDU(data=a.copy())
        hdu2 = fits.PrimaryHDU(data=a.copy())
        # Previously the following line would throw a TypeError,
        # now it should be identical to the integer bzero case
        hdu1.scale('int16', bzero=99.0)
        hdu2.scale('int16', bzero=99)
>       assert np.allclose(hdu1.data, hdu2.data)
E       assert False
E        +  where False = <function allclose at 0x400327b510>(array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16), array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16))
E        +    where <function allclose at 0x400327b510> = np.allclose
E        +    and   array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.image.PrimaryHDU object at 0x4111f516a0>.data
E        +    and   array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.image.PrimaryHDU object at 0x4111f51780>.data

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_image.py:1059: AssertionError
____________ TestCompressedImage.test_compression_with_gzip_column _____________

self = <astropy.io.fits.tests.test_image.TestCompressedImage object at 0x401b64cba8>

    def test_compression_with_gzip_column(self):
        """
        Regression test for https://github.com/spacetelescope/PyFITS/issues/71
        """
    
        arr = np.zeros((2, 7000), dtype='float32')
    
        # The first row (which will be the first compressed tile) has a very
        # wide range of values that will be difficult to quantize, and should
        # result in use of a GZIP_COMPRESSED_DATA column
        arr[0] = np.linspace(0, 1, 7000)
        arr[1] = np.random.normal(size=7000)
    
        hdu = fits.CompImageHDU(data=arr)
        hdu.writeto(self.temp('test.fits'))
    
        with fits.open(self.temp('test.fits')) as hdul:
            comp_hdu = hdul[1]
    
            # GZIP-compressed tile should compare exactly
>           assert np.all(comp_hdu.data[0] == arr[0])
E           assert False
E            +  where False = <function all at 0x4003260378>(array([ 0.000...dtype=float32) == array([ 0.000...dtype=float32)
E            +    where <function all at 0x4003260378> = np.all
E               Use -v to get the full diff)

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_image.py:1659: AssertionError
________ TestCompressedImage.test_scale_bzero_with_compressed_int_data _________

self = <astropy.io.fits.tests.test_image.TestCompressedImage object at 0x41129da2e8>

    def test_scale_bzero_with_compressed_int_data(self):
        """
        Regression test for https://github.com/astropy/astropy/issues/4600
        and https://github.com/astropy/astropy/issues/4588
    
        Identical to test_scale_bzero_with_int_data() but uses a compressed
        image.
        """
    
        a = np.arange(100, 200, dtype=np.int16)
    
        hdu1 = fits.CompImageHDU(data=a.copy())
        hdu2 = fits.CompImageHDU(data=a.copy())
        # Previously the following line would throw a TypeError,
        # now it should be identical to the integer bzero case
        hdu1.scale('int16', bzero=99.0)
        hdu2.scale('int16', bzero=99)
>       assert np.allclose(hdu1.data, hdu2.data)
E       assert False
E        +  where False = <function allclose at 0x400327b510>(array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16), array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16))
E        +    where <function allclose at 0x400327b510> = np.allclose
E        +    and   array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.compressed.CompImageHDU object at 0x41129da5c0>.data
E        +    and   array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.compressed.CompImageHDU object at 0x4113000400>.data

../../.tox/py37-test/lib/python3.7/site-packages/astropy/io/fits/tests/test_image.py:1704: AssertionError

@olebole - have you seen these before in Debian testing?

How did Travis CI not catch these stuff before?

We only have arm and s390x not PPC. But I can add all three to the GitHub Actions config 😺

I didn't see that, and I checked that I didn't disable these tests with a patch. A test log is f.e. here for 4.1, the build log for 4.2rc1 with tests is here. The difference is that I use cfitsio as an external, precompiled library. However, I also couldn't find a relevant patch in the cfitsio package. So, I have no good idea for the moment.

Refs:

Debian Astropy patches (here for 4.2rc1)
Debian Cfitsio patches

I also checked all failed Debian builds on ppc64el for io.fits failures, but I am quite sure that I have never seen your failure.

There are a bunch of failures in astropy/timeseries/periodograms/lombscargle/tests/test_lombscargle.py now too, bringing total failure up to 87.

Wow, this one is quite surprising. It has nothing to do with compression, cfitsio etc., just some basic arithmetic.
https://github.com/astropy/astropy/runs/1949615134?check_suite_focus=true

    def test_scale_bzero_with_int_data(self):
        """
        Regression test for https://github.com/astropy/astropy/issues/4600
        """
    
        a = np.arange(100, 200, dtype=np.int16)
    
        hdu1 = fits.PrimaryHDU(data=a.copy())
        hdu2 = fits.PrimaryHDU(data=a.copy())
        # Previously the following line would throw a TypeError,
        # now it should be identical to the integer bzero case
        hdu1.scale('int16', bzero=99.0)
        hdu2.scale('int16', bzero=99)
>       assert np.allclose(hdu1.data, hdu2.data)
E       assert False
E        +  where False = <function allclose at 0x40039b3e18>(array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16), array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16))
E        +    where <function allclose at 0x40039b3e18> = np.allclose
E        +    and   array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,   0,   0,   0,... 0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n         0,   0,   0,   0,   0,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.image.PrimaryHDU object at 0x412d4e7198>.data
E        +    and   array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,\n        14,  15,  16,  17,  18,  19,  20,  21,...81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,\n        92,  93,  94,  95,  96,  97,  98,  99, 100], dtype=int16) = <astropy.io.fits.hdu.image.PrimaryHDU object at 0x412d4e70f0>.data

I'm going to close these as I don't think we ever ran into these again after and this was never an issue during Debian packaging

But this arch is still commented in CI. How do we know?

astropy/.github/workflows/ci_cron_weekly.yml

Lines 109 to 110 in 8dfd151

    
           # Uncomment when we ready to fix the failures, see PR 11697 
        
           #- arch: ppc64le

Ah I didn't realise it was commented out! Will see if I can investigate at some point.

Navigation Menu

Repository navigation

io.fits and Lomb-Scargle test failures on ppc64le #11026

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# Uncomment when we ready to fix the failures, see PR 11697
	#- arch: ppc64le

io.fits and Lomb-Scargle test failures on ppc64le #11026

Description

Activity

pllim commented on Nov 11, 2020

astrofrog commented on Nov 11, 2020

olebole commented on Nov 11, 2020

olebole commented on Nov 12, 2020

pllim commented on Feb 22, 2021

saimn commented on Feb 22, 2021

astrofrog commented on Jan 21, 2023

pllim commented on Jan 21, 2023

astrofrog commented on Jan 21, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions