Skip to content

Fixed bug with checksum on CompImageHDU if data was read in and written out straight away - #16957

Closed
astrofrog wants to merge 4 commits into
astropy:mainfrom
astrofrog:fix-comp-hdu-bug
Closed

Fixed bug with checksum on CompImageHDU if data was read in and written out straight away#16957
astrofrog wants to merge 4 commits into
astropy:mainfrom
astrofrog:fix-comp-hdu-bug

Conversation

@astrofrog

@astrofrog astrofrog commented Sep 6, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

While discussing checksums with @saimn in #14753 I realised that that bug only occurs under a very specific condition - if we read in data and write it out straight away, and that in fact everything works as expected if one modifies the data or sets the data. The fix ends for this issue on main actually ends up being simple, and is done in this PR.

Essentially, if a compressed FITS file is read in and written out straight away, ZHECKSUM/ZDATASUM are missing compared to cases where the data is set or modified.

The following script illustrates the problem:

import numpy as np
from astropy.io import fits

from astropy.io.fits.hdu.base import _ValidHDU
_ValidHDU._get_timestamp = lambda self: "2013-12-20T13:36:10"


def show_checksums(filename):
    with open(filename, 'rb') as f:
        header = f.read()
    found = False
    for i in range(2880, len(header), 80):
        if b'SUM' in header[i:i+8]:
            print(header[i:i+80].decode('ascii'))
            found = True
    if not found:
        print('No checksum keywords found')


data = np.arange(40).reshape((5, 8))

print('-' * 72)
print('Original, export without checksum')

hdu1 = fits.CompImageHDU(data)
hdu1.writeto('original_nochecksum.fits', overwrite=True)

show_checksums('original_nochecksum.fits')

print('-' * 72)
print('Original, export with checksum')

hdu2 = fits.CompImageHDU(data)
hdu2.writeto('original_checksum.fits', checksum=True, overwrite=True)

show_checksums('original_checksum.fits')

print('-' * 72)
print('Read in original_nochecksum.fits, export with checksum (no change to data)')

with fits.open('original_nochecksum.fits') as hdulist:
    hdulist.writeto('read_and_exported_checksum.fits', checksum=True, overwrite=True)

show_checksums('read_and_exported_checksum.fits')

print('-' * 72)
print('Read in original_nochecksum.fits, export with checksum re-assigning data to itself')

with fits.open('original_nochecksum.fits') as hdulist:
    hdulist[1].data = hdulist[1].data.copy()
    hdulist.writeto('read_reassign_and_exported_checksum.fits', checksum=True, overwrite=True)

show_checksums('read_reassign_and_exported_checksum.fits')

print('-' * 72)
print('Read in original_nochecksum.fits, export with checksum and changing data')

with fits.open('original_nochecksum.fits') as hdulist:
    hdulist[1].data = data * 2
    hdulist.writeto('read_modify_and_exported_checksum.fits', checksum=True, overwrite=True)

show_checksums('read_modify_and_exported_checksum.fits')

Running this on main gives:

------------------------------------------------------------------------
Original, export without checksum
No checksum keywords found
------------------------------------------------------------------------
Original, export with checksum
ZHECKSUM= 'M1PqM1OnM1OnM1On'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '780     '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'ADGBBDF9ADFAADF9'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum (no change to data)
CHECKSUM= 'JKH9LJ98JJE8JJ98'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum re-assigning data to itself
ZHECKSUM= 'M1PqM1OnM1OnM1On'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '780     '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'ADGBBDF9ADFAADF9'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum and changing data
ZHECKSUM= 'K4PjM1MjK1MjK1Mj'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '1560    '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'S5a4T2X4S2a4S2U4'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '2240124460'         / data unit checksum updated 2013-12-20T13:36:10

In the first case, there is no checksum, which is expected. In the second case the decompressed HDU checksum and datasum are computed and stored in ZHECKSUM/ZDATASUM. However, if we read in the file with no checksums and export it immediately without touching data, then ZHECKSUM/ZDATASUM are missing, which is inconsistent. Furthermore, as shown in the fourth case, if we read the file in and reassign the data to itself, then ZHECKSUM/ZDATASUM are back, and if we change the data then of course the checksums change (fifth case).

The bug here was that the image checksums in CompImageHDU._prewriteto were computed only if self._has_data is True, but if you just read in a compressed file and don't interact with the data, it is never loadded and self._has_data is False, even though in principle there is data that could be used to compute a checksum. I have therefore re-arranged the code slightly to compute the image checksum regardless of self._has_data but instead doing it only if checksum is specified and if self.data is not None (the check of checksum is needed because accessing self.data is expensive as it might trigger the whole data to decompress - we need to do this in any case if we are computing the checksum, but might as well avoid hitting self.data if checksum=False. With this minor change, the output is then:

------------------------------------------------------------------------
Original, export without checksum
No checksum keywords found
------------------------------------------------------------------------
Original, export with checksum
ZHECKSUM= 'M1PqM1OnM1OnM1On'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '780     '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'ADGBBDF9ADFAADF9'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum (no change to data)
ZHECKSUM= 'M1PqM1OnM1OnM1On'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '780     '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'ADGBBDF9ADFAADF9'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum re-assigning data to itself
ZHECKSUM= 'M1PqM1OnM1OnM1On'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '780     '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'ADGBBDF9ADFAADF9'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '1054568040'         / data unit checksum updated 2013-12-20T13:36:10 
------------------------------------------------------------------------
Read in original_nochecksum.fits, export with checksum and changing data
ZHECKSUM= 'K4PjM1MjK1MjK1Mj'   / HDU checksum updated 2013-12-20T13:36:10       
ZDATASUM= '1560    '           / data unit checksum updated 2013-12-20T13:36:10 
CHECKSUM= 'S5a4T2X4S2a4S2U4'   / HDU checksum updated 2013-12-20T13:36:10       
DATASUM = '2240124460'         / data unit checksum updated 2013-12-20T13:36:10

The checksums are the same in the second, third, and fourth case.

@astrofrog
Fixed a bug that caused checksums written out with a CompImageHDU to …
eae43e1 
…be incorrect if the data was read in and written out without any changes
@astrofrog
Update checksums in test
8374611
@astrofrog astrofrog added the Bug label Sep 6, 2024
@astrofrog
astrofrog requested a review from saimn as a code owner September 6, 2024 13:12
@github-actions

github-actions Bot commented Sep 6, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@astrofrog astrofrog added this to the v6.1.4 milestone Sep 6, 2024
@astrofrog

astrofrog commented Sep 6, 2024

Copy link
Copy Markdown
Member Author

pre-commit.ci autofix

@astrofrog astrofrog mentioned this pull request Sep 6, 2024
Closed
@pllim pllim modified the milestones: v6.1.4, v6.1.5 Sep 27, 2024
@saimn

saimn commented Sep 30, 2024

Copy link
Copy Markdown
Contributor

@astrofrog - I think we can close this one ?

@astrofrog astrofrog closed this Sep 30, 2024
@astrofrog

astrofrog commented Sep 30, 2024

Copy link
Copy Markdown
Member Author

Indeed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saimn saimn Awaiting requested review from saimn saimn is a code owner

Assignees

No one assigned

Labels

Bug io.fits

Projects

None yet

Milestone

v6.1.5

Development

Successfully merging this pull request may close these issues.

3 participants

@astrofrog @saimn @pllim