Add a votable parse memory use benchmark - #158

One thing I notice here is that the peakmem_ tests hide smaller changes when comparing commits.

These benchmarks report the peak memory of the whole process that runs the benchmark. That includes the Python interpreter, astropy and the other packages it imports, the modules in this benchmark suite along with anything kept in memory in setup(), and asv itself. So lots of overhead. And that's the right number for "how much memory does a process need to do this?".

But astropy's github actions benchmark CI asks a different question: did this pull request change memory use? It answers by dividing the PR's result by main's and flagging changes larger than 1.3x. The overhead above is in both numbers, so it pulls the ratio towards 1.

For the VOTable benchmarks in this PR on Linux, the overhead is about 330 MB. A change that halved the memory use of a parse from 200 MB to 100 MB would show as 530 MB against 430 MB, a ratio of 0.81, and CI wouldn't flag it as an improvement, even though it halved memory use. One can go with larger files, but then that slows down the benchmark.

I do notice there are some track_ benchmarks that just report a number. Maybe one can use that for the case where one really wants to know if a PR improved memory usage or not without having to use very large files relative to the overhead of of the python stack? Maybe reseting the peak memory measurement, or doing a before/after?

I noticed this because the ratio above, 0.47, is less for these small files than it was for the very large ~1.2GB file in the PR. For that our improvement was ~38%, and it was larger still for the 2.5GB file from the original issue.

So it really depends on the use case - is this primarily for CI and differences? Or for total memory usage for this benchmark test case?

OK, so I've updated the benchmark to subtract off the python interpreter (and astropy, numpy, asv stack) using the track_ pattern in asv.

track_peak_memory_parse returns the process's peak memory after the parse minus its memory use just before it. asv runs each benchmark in its own process, and before the parse that process has only imported modules. So when the parse starts, the peak so far is the same as the current use. The table is built and the test VOTable files are written in setup_cache(), which runs in a separate process, so none of that work is counted.

As you can see in the table, each before/after value is ~300MB less.

Results on MacOS for astropy/astropy#20509:

$ python -m asv continuous --config asv_scan.conf.json --split --show-stderr --factor 1.3 --bench TrackMem upstream/main 8d7992300b
...
· Running 2 total benchmarks (2 commits * 1 environments * 1 benchmarks)
...
| Change   | Before [073180b4]    | After [8d799230] <xml-decode-base64-while-parsing>   |   Ratio | Benchmark (Parameter)                                            |
|----------|----------------------|------------------------------------------------------|---------|------------------------------------------------------------------|
| -        | 995M                 | 237M                                                 |    0.24 | votable.TrackMemVOTableArrays.track_peak_memory_parse('binary')  |
| -        | 991M                 | 240M                                                 |    0.24 | votable.TrackMemVOTableArrays.track_peak_memory_parse('binary2') |

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

Which now shows what I expect. And it shows the same regardless of size of the arrays, i.e. I varied LARGE_SIZE which gives the number of rows in the table, and above 100k rows, the ratios are the same. Below that, the fixed size of the parser is added in, so a 25k row table shows a 0.31 and 0.36 ratio for binary and binary2, respectively. Previously, using peakmem_ which includes the size of the interpreter stack in the numerator and denominator, it washes out the changes much more, even when the input files are very large, and at the 60 second limit of the asv benchmarks themselves. For example 1 million row tables just fail to run because of the 60 sec limit.

One thing I was wondering, do we know the peak isn't already above current usage when we take before?

If one uses setup_cache() for any setup, then it should not be a problem, as asv runs each benchmark in its own process. I added a note about that to the docstring of the helper function. Peaking before the benchmark ran would only be a problem (I think) if importing the module allocated lots of memory and then released it. Which it does not do, as far as I can tell.

If setup() was used instead of setup_cache(), and used a bunch of memory and then released it on exit, then I think your concern would be justified. So I don't do that, and the docstring specifies to use setup_cache().

It is quite difficult to figure out this memory stuff. There's no perfect measurement, as far as I can tell. Everything is a bit of a compromise.

Anything else that needs to happen in this PR?

Summary

• adds 2 new benchmarks measuring memory use and parse time for a votable read from disk, parametrized across the 3 supported serialization formats - TABLEDATA, BINARY and BINARY2.
• benchmarks add an extra ~10 to 20 seconds of benchmark runtime.
• these are the only benchmarks that measure parsing files from disk for votable currently. The existing ones only use bytesIO in-memory objects.

Just waiting for an approval from @stvoutsin :)