Memory management in Astropy tables

I am using Astropy tables to cross match some galaxy catalogues and running in to memory issues on a university HPC. I wonder if there is any way to gain access to the memory mapping feature available with the general astropy.io.fits module. I like astropy.table because it is easy to display in notebooks but may have to stop using it because of memory issues. I may have to rewrite code to tile the sky for memory reasons anyway but wondered if there was an easy way to prevent astropy from loading the whole table into memory.

Many thanks for reading.

Raphael.

Will #6458 solve your problem?

I'm not sure, I am going to try using astropy.io.fits instead of astropy.Table so I can see if opening fits files with memmap=True reduces the memory used. I wonder if anyone knows if astropy.Table inherits from astropy.io.fits and therefore if it defaults to memmap=True anyway.

#3958 -- Looks like you cannot pass memmap directly to Table yet.

#1380 -- memmap might not work in some modes.

@taldcroft

I wonder if anyone knows if astropy.Table inherits from astropy.io.fits

The actual code that reads FITS into Table is in https://github.com/astropy/astropy/blob/master/astropy/io/fits/connect.py . While it is true that FITS reader defaults to memmap=True, the part where FITS data is parsed into a Table would require the entire table to be read into memory (maybe @taldcroft can confirm). So it is possible to only load one extension at a time, but you cannot load only part of the same extension.

p.s. Please correct me if I am wrong!

@pllim

@pllim - yes astropy Table does not support memory mapping, and indeed reading a FITS extension as a Table requires putting the whole thing in to memory.

I can imagine a relatively simple extension to the FITS connect code to allow reading in only a subset of the rows into a table.

@pllim - as a side note, is the default for memmap True? The docs say it is None, but they don't indicate what that implies. http://astropy.readthedocs.io/en/latest/io/fits/api/files.html#astropy.io.fits.open

is the default for memmap True?

The logic is in https://github.com/astropy/astropy/blob/master/astropy/io/fits/file.py . I followed a few branches of the logic and seems to me default is True for "normal" cases? Feel free to double check.

I also opened a file locally and checked for my use case:

>>> from astropy.io import fits
>>> pf = fits.open('alpha_lyr_stis_008.fits')
>>> pf._file.memmap
True
>>> pf.close()

memmap is True by default, since PyFITS 3.1. It is None in the fits.open signature because there is also a config item and it allows to check if it is explicitly set to True (https://github.com/astropy/astropy/blob/astropy/io/fits/hdu/hdulist.py#L141)

So currently Table will put the whole table in to memory, but it should be possible to add optional arguments to specify a list of columns to include or exclude, to read only a few columns (as include_names and exclude_names for io.ascii). Thanks to memmap and using .field() to read to columns data, it should help reduce the memory usage I guess ?

I think this could be an important issue for making astropy competitive with stilts which is extremely memory efficient, particularly as future surveys promise to create larger and larger quantities of data. I plan to spend some effort on tiling the sky to deal with this issue but think it may be a long term project unless astropy does not aspire to handle 'big data'. I watch with interest. Thank you everyone for your help I guess it is sensible to close the topic now.

I'm not clear on why Table wouldn't take advantage of this. Can't it load a column from a FITS table without making a copy?

@embray

I have the same question as @embray - why can't we support memory mapping?

It should indeed be possible to have table columns be views on (parts of) the underlying fits table. It may not need more than adding and propagating a copy argument.

@saimn

@saimn At the hardware level, mmap only works for starting at an offset and then extending for size bytes. There is no way to reduce the memory size, other than specifying chunks and then repeatedly processing these chunks in a loop.

So, did #6821 fix this?

Probably, since it is now possible to use memmap with Table.read.

Navigation Menu

Repository navigation

Memory management in Astropy tables #6491

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Memory management in Astropy tables #6491

Description

Activity

pllim commented on Aug 30, 2017

raphaelshirley commented on Aug 31, 2017

pllim commented on Aug 31, 2017

pllim commented on Aug 31, 2017

taldcroft commented on Aug 31, 2017

pllim commented on Aug 31, 2017

saimn commented on Aug 31, 2017

saimn commented on Aug 31, 2017

raphaelshirley commented on Sep 1, 2017

embray commented on Oct 12, 2017

astrofrog commented on Oct 12, 2017

mhvk commented on Oct 14, 2017

manodeep commented on May 1, 2018

pllim commented on Feb 20, 2019

saimn commented on Feb 20, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions