You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Reading a gzipped file can be very slow compared to ungzipping the file in memory before reading it. https://github.com/saimn/misc-astro/blob/master/astropy/FITS%20compression.ipynb
This is because the code uses the file handler from gzip.open to read (and uncompress on the fly) small blocks of bytes, which is much slower than decompressing the whole file at once.
The current behavior is useful to limit the memory usage, but sometimes unpractical when reading big files, where it can be much much slower. @astrofrog - Should we consider adding a new parameter to decompress the whole file in memory ? This is what I use in my code currently to circumvent this (passing io.BytesIO(gzip.open(filename).read()) to fits.open, see notebook).
Activity
astrofrog commentedon May 23, 2023
Yes I think adding a parameter for this would make sense!
enthusiastic2003 commentedon May 24, 2023
@astrofrog , just to be clear, where do you want to put the extra parameter?
saimn commentedon May 29, 2023
The parameter should be added to
fitsopen(https://github.com/astropy/astropy/blob/main/astropy/io/fits/hdu/hdulist.py#L46) and propagated to_try_read_compressed(https://github.com/astropy/astropy/blob/main/astropy/io/fits/file.py#L500) where the file can be decompressed in memory.It should maybe also apply to
.zipand.bz2files, not sure (some tests should be made to check).And we need to find a good name :)
saimn commentedon Jun 20, 2023
Same issue with
bz2(see updated notebook above).