You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I am working with fits files that contain a few thousand extensions (spectra) each. Looping over each extension causes my computer to run out of file handles and my code crashes with a "Too many open files" error. I think this is related to the memmap error in the warning box on the astropy.io.fits documentation:
"When opening a file with memmap=True, because of how mmap works this means that when the HDU data is accessed (i.e. hdul[0].data) another handle to the FITS file is opened by mmap. This means that even after calling hdul.close() the mmap still holds an open handle to the data so that it can still be accessed by unwary programs that were built with the assumption that the .data attribute has all the data in-memory."
I get the crash without setting memmap, and with setting memmap = True. It goes away with memmap=True and del hdu[ext].data or just setting memmap = False. It looks to me like I'm defaulting to memmap = True. Is that the correct behavior? If so, it should probably be noted more clearly in the documentation.
Actually it defaults to the value set in the astropy/io/fits/conf (which does not exist at least in my local installation) before defaulting to True.
But I am with you: it should ne noted in the documentation, what the default is.
Activity
MSeifert04 commentedon Sep 29, 2015
Actually it defaults to the value set in the
astropy/io/fits/conf(which does not exist at least in my local installation) before defaulting to True.But I am with you: it should ne noted in the documentation, what the default is.
embray commentedon Sep 29, 2015
Yeah, looking at http://docs.astropy.org/en/stable/io/fits/api/files.html#open it seems this isn't properly documented.
embray commentedon Sep 29, 2015
By the way, there are some fixes in the current development version that should lessen the problem with "too many open files".
saimn commentedon Aug 31, 2017
Closing in favor of #6497 whose title and description better reflects what should be done.