Repository navigation
Large memory footprint in astropy.io.votable.parse_single_table #8946
Description
I was trying to read two columns from an ~ 1 GB votable file (demo Gaia dr2 data). The file itself contains ~ 96 columns. The code I used was:
from astropy.io.votable import parse_single_table
columns = ['phot_g_mean_mag', 'parallax']
table = parse_single_table("async_20190630210155.vot", columns=columns)
print("Done reading table")Here's the file info:
$ ls -alh async_20190630210155.vot
-rw-rw-rw- 1 msinha 1195219923 1.1G Jul 1 14:01 async_20190630210155.votLooking at the memory footprint, I saw that python was taking ~12 GB during the read and I cancelled the kernel (this is within a Jupyter notebook). Here's my screenshot showing the memory usage:
While I know that there is a significant python overhead, it still seems like a lot of memory to read only 2 columns (out of 96). By my math, the (minimum) possible size is 2/96*1 GB ~ 0.02 GB
Since I am new to both astropy and votables, perhaps I am doing something incorrectly. Happy to provide further info or help debug, as necessary. In case there is something inherently wrong with the file itself, here's a dropbox link to the file.
Cheers,
Manodeep
Activity
manodeep commentedon Jul 3, 2019
I ran the memory profiler within the jupyter notebook (new jupyter session, new browser session) to see what might be going on.
The system activity monitor had the memory usage roughly constant at 11.6 GB for the ~20 mins that I let the code run (the read did not complete, that's why the last two lines were not executed). The function memory profile accounts for less than 1 GB.
I am not sure how to include the memory profile for all the called functions, but it seems that the excessive memory is coming from some function within the
votableroutines.manodeep commentedon Jul 3, 2019
For completeness, here's the output from
line_profiler:Nothing surprising. All of the runtime is in the
parse_single_tablefunction.pllim commentedon Jul 9, 2019
Thanks for reporting this. Sounds like a bug, perhaps due to memory leak. 🤔
To profile the underlying calls, for the memory part, might need to call them directly, which is not very fun. For the performance part, might have to fall back to cProfile.
It all starts here:
astropy/astropy/io/votable/table.py
Lines 170 to 187 in 7e751f1
manodeep commentedon Jul 9, 2019
Thanks @pllim. Since
topcatread in the file just fine, I can confirm that this is indeed a bug withinastropyand not something peculiar about the file.pllim commentedon Aug 13, 2019
Finally got around to downloading this file. I just fired up
volintand immediately I see one of the CPU at 100% capacity (probably expected, though the CPU switches from one to the other once in a while) and memory use quickly ramped up to about over 20 GB and one point (whaaaat?) and then ramp down a bit and is staying much or less around 16-17 GB. 😱 Given how long you reported that it takes to parse the file (20 mins), I am gonna have to kill this job. Something is obviously not right...When I kill the job, here is the traceback it spitted out before it died:
This appears to be a BINARY2 format.
Note to self: What is this undocumented
_debug_python_based_parseroption intable.py? 🤔pllim commentedon Mar 13, 2020
manodeep commentedon Mar 13, 2020
Probably wouldn't be the case in the immigrant library ;-)
pllim commentedon Mar 13, 2020
manodeep commentedon Mar 13, 2020
@pllim Here you go async_subset.vot, 22MB
pllim commentedon Mar 13, 2020
Thanks! However, something isn't quite right. The file isn't binary anymore. And while the parsing is instantaneous this time without using much memory, the table looks invalid.
Also, it looks like
columnsis ignored?pllim commentedon Mar 13, 2020
Going back to the 1GB file, I ran this, walked away, came back an hour later, and amazingly it finished. Just posting this here for future reference. So for sure it did not get stuck in a loop somewhere, just eats up lots of memory and takes up to an hour to parse. 😱
pllim commentedon Mar 13, 2020
9 remaining items
jdavies-st commentedon Sep 30, 2026
After having spent time spelunking in the
io.votableC internals due to a different issue, I revisited this issue, using the original file (2.5 GB uncompressed, Gaia DR2) and current main branch.File is 3,000,000 rows x 96 columns, BINARY2. Big! But not crazy big. Should fit into laptop memory, but doesn't.
Today, on astropy main branch, speed is OK today; memory is not.
Parsing takes ~40 us per row and scales linearly with number of rows, so a little over 2 minutes for this file. Peak memory is about 16 GB, more than 6 times the file size, and passing
columns=does not reduce. The hour-long runs reported above were likely memory swapping/paging on laptops? My laptop here has a bit more memory than standard, but it also started swapping the full size file, so I made smaller sized versions of the same file for testing - 1%, 5%, 10%, etc. Same columns, fewer rows in the binary data section.The cause of the huge memory usage is mostly in
iterparse.c, as @pllim found.endElementbuilds the end tag event's text from a shared buffer but never empties it. Every enclosing end tag that follows then gets its own copy of the last element's text. For a BINARY2 table, , and each carry another full copy of the 2.4 GB stream.A toy example is below, showing this using the event iterator and logging the events between tags. If we look at the end tags only:
The pure-Python parser does roughly the same, so moving from the default fast parser to the python one doesn't help.
So fixing both of these would help reduce memory.
There's probably some smaller memory redundancies to add to this:
_get_binary_data_streamdecodes the stream withbase64.b64decode(text.encode("ascii")). This step makes a copy of all 2.4 GB in order to turn the text into bytes. There's a one-liner to do this without making a copy.nrowsattribute onTABLE). Typical of VOT query results. So the result array starts small, and each time it fills up it is replaced by one 1.5 times larger, with every row read so far copied across. At the end it is copied once more to trim it to the exact number of rows. During each copy, the old and new arrays are both in memory, so its probably adding an extra 1.5x size in memory here as well.I don't think the earlier fix (#10076) changed the memory use. The tokenizer doubles its text buffer each time the text outgrows it (1 GB, 2 GB, 4 GB, and so on), and #10076 corrected how that next size is calculated above 4 GB. This file's text is 2.4 GB, so the old calculation was already OK for it.
So there's some low-hanging fruit here. I have not looked at the behavior in reading just 2 columns, other than to say it doesn't seem to change memory usage, as already reported in this issue.
Stop XML end events from repeating the text of nested elements
Stop XML end events from repeating the text of nested elements
Stop XML end events from repeating the text of nested elements
Stop XML end events from repeating the text of nested elements
Reduce memory needed to read VOTable BINARY and BINARY2 streams (#20509)
Grow VOTable result arrays in place instead of copying them
Grow VOTable result arrays in place instead of copying them
Grow VOTable result arrays in place instead of copying them
Grow VOTable result arrays in place instead of copying them (#20743)