Skip to content

astropy.io.fits defaults to memmap=True ?  #4196

Closed

Description

@alainahenry

Hi, I am working with fits files that contain a few thousand extensions (spectra) each. Looping over each extension causes my computer to run out of file handles and my code crashes with a "Too many open files" error. I think this is related to the memmap error in the warning box on the astropy.io.fits documentation:

"When opening a file with memmap=True, because of how mmap works this means that when the HDU data is accessed (i.e. hdul[0].data) another handle to the FITS file is opened by mmap. This means that even after calling hdul.close() the mmap still holds an open handle to the data so that it can still be accessed by unwary programs that were built with the assumption that the .data attribute has all the data in-memory."

I get the crash without setting memmap, and with setting memmap = True. It goes away with memmap=True and del hdu[ext].data or just setting memmap = False. It looks to me like I'm defaulting to memmap = True. Is that the correct behavior? If so, it should probably be noted more clearly in the documentation.

Activity

  1. MSeifert04 commented on Sep 29, 2015

    @MSeifert04
    Contributor

    Actually it defaults to the value set in the astropy/io/fits/conf (which does not exist at least in my local installation) before defaulting to True.
    But I am with you: it should ne noted in the documentation, what the default is.

  2. embray commented on Sep 29, 2015

    @embray
    Member

    By the way, there are some fixes in the current development version that should lessen the problem with "too many open files".

  3. saimn commented on Aug 31, 2017

    @saimn
    Contributor

    Closing in favor of #6497 whose title and description better reflects what should be done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions