Repository navigation
Give example for Table.read with memmap=True #8573
Description
In #6821 @astrofrog added the memmap=True option to Table.read for FITS files. The PR shows an example where this gives huge performance improvements in execution time and memory use.
Could someone please add an example to http://docs.astropy.org/en/stable/table/index.html where memmap=True for FITS read does give an advantage?
And if possible also add a short explanation why this works? I thought Table data is column-wise in-memory (as explained at http://docs.astropy.org/en/stable/table/implementation_details.html), but FITS BINTABLE stores row by row on disk. So I don't understand how memmap=True can work with Table at all.
@saimn - this issue / question from me is triggered by your #7404 (comment) where you say that Table.read with memmap=True does manage to load FITS without having to copy the whole table into memory.
Looking for a small standalone example I can play with with %time and %memit to understand what is going on ...
Activity
saimn commentedon Apr 12, 2019
You're right about columns, so when accessing a column all the data is loaded in memory (unless I'm missing another way to do it ? maybe @astrofrog knows). But thanks to memmap you can load only a subset of the rows. I made quickly a notebook to illustrate:
https://github.com/saimn/misc-astro/blob/master/astropy/Table_memmap.ipynb
cdeil commentedon Apr 12, 2019
@saimn - Thank you!
Now I understand, it makes sense.
I get the same results as you in the notebook (for me on macOS) - you could add a mention to
pip install ipython_memory_usage(https://github.com/ianozsvald/ipython_memory_usage) if someone else wants to try it out.In Gammapy we are always working with columns, so what we do now to never memmap (which came out of the discussions / advice in #7440) is the right approach (cc @adonath)
saimn commentedon Apr 12, 2019
Indeed, I added a note about this.
Yes, but actually with
Table.readit is False by default.cdeil commentedon Apr 12, 2019
Yes, but we sometimes read multiple FITS tables from a given file, and then do it roughly like this:
It should be a bit faster than opening the file twice, but probably we could change to this for simplicity and performance would be basically the same, if file open/close is inexpensive compared to table read.
saimn commentedon Apr 12, 2019
Okay, the performance depend on your files, if headers have a lot of keywords or not. In general the first solution is faster because you don't need to parse again the primary and first extension to read the second one.