Skip to content

Reading gzipped FITS file is slow #14855

Closed

Description

@saimn

Description

Reading a gzipped file can be very slow compared to ungzipping the file in memory before reading it.
https://github.com/saimn/misc-astro/blob/master/astropy/FITS%20compression.ipynb
This is because the code uses the file handler from gzip.open to read (and uncompress on the fly) small blocks of bytes, which is much slower than decompressing the whole file at once.
The current behavior is useful to limit the memory usage, but sometimes unpractical when reading big files, where it can be much much slower.
@astrofrog - Should we consider adding a new parameter to decompress the whole file in memory ? This is what I use in my code currently to circumvent this (passing io.BytesIO(gzip.open(filename).read()) to fits.open, see notebook).

Expected behavior

No response

How to Reproduce

  1. Get package from '...'
  2. Then run '...'
  3. An error occurs.
# Put your Python code snippet here.

Versions

import platform; print(platform.platform())
import sys; print("Python", sys.version)
import astropy; print("astropy", astropy.version)
import numpy; print("Numpy", numpy.version)
import erfa; print("pyerfa", erfa.version)
import scipy; print("Scipy", scipy.version)
import matplotlib; print("Matplotlib", matplotlib.version)

Activity

  1. astrofrog commented on May 23, 2023

    @astrofrog
    Member

    Yes I think adding a parameter for this would make sense!

  2. enthusiastic2003 commented on May 24, 2023

    @enthusiastic2003

    @astrofrog , just to be clear, where do you want to put the extra parameter?

  3. saimn commented on May 29, 2023

    @saimn
    ContributorAuthor

    The parameter should be added to fitsopen (https://github.com/astropy/astropy/blob/main/astropy/io/fits/hdu/hdulist.py#L46) and propagated to _try_read_compressed (https://github.com/astropy/astropy/blob/main/astropy/io/fits/file.py#L500) where the file can be decompressed in memory.
    It should maybe also apply to .zip and .bz2 files, not sure (some tests should be made to check).
    And we need to find a good name :)

  4. saimn commented on Jun 20, 2023

    @saimn
    ContributorAuthor

    Same issue with bz2 (see updated notebook above).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions