votable read slow

Compared with other tools (eg. stilts/topcat), the (ASCII) votable read is slow in astropy - by about a factor of 10 according to my measurements. Version 3 is doing a bit better (~20%) than version 2.

Test data: A 40MB votable with 73000 rows (containing GAIA DR2 data).
Table.read(file,format="votable") or astropy.io.votable.parse() python 2/astropy 2.0.9: 45 seconds
Table.read(file,format="votable") or astropy.io.votable.parse() python 3/astropy 3.0.5: 38 seconds
stilts (tcopy to /dev/null, including java startup + write to /dev/null): 4.8 seconds

Apart from reading votables from disk, this also impacts SAMP data transfer, which uses votables. As an example, transferring 206000 rows from topcat to an astropy.samp client takes nearly 500 seconds. The same transfer to Aladin took 130 seconds.

Thanks for reporting. What Astropy version are you using? Please give code snippets and data file. And would be nice to have some code profiling done to see where the bottleneck is.

The astropy versions are listed in the example given above and were 2.0.9 and 3.0.5.

The code snippet I used is very simple:

start=time.time()
tab=Table.read(filename,format="votable")
print(time.time()-start)

or replace the Table.read command with astropy.io.votable.parse(filename).
The comparison with stilts was done with the following shell command:

time java -jar stilts.jar tcopy in=gaia.vot ofmt=fits out=/dev/null

As data file any (larger) votable would do. The one I have been using came from the following cone query:
http://gsss.stsci.edu/webservices/vo/ConeSearch.aspx?CAT=GAIAdr2&RA=0.196125&DEC=65.1165833333&SR=0.5
which gives a 114MB (the 40MB that I quoted above were wrong) votable.

I should mention that a few warnings are displayed when reading the data file:

WARNING: W06: gaia.vot:43:0: W06: Invalid UCD 'arith.ratio': Secondary word 'arith.ratio' is not valid as a primary word [astropy.io.votable.tree]
WARNING: W50: gaia.vot:151:0: W50: Invalid unit string 'counts/s' [astropy.io.votable.tree]
WARNING: W48: gaia.vot:299:0: W48: Unknown attribute 'nrows' on TABLEDATA [astropy.io.votable.tree]
WARNING: W49: gaia.vot:300:1422: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]

Regarding profiling, this is better done by someone who knows the votable implementation in astropy. A standard python profile doesn't shed too much light on it ... at least not for me.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1   27.437   27.437   49.192   49.192 votable/tree.py:2430(_parse_tabledata)
  5301432    8.063    0.000    8.303    0.000 votable/converters.py:715(_parse_permissive)
  1472620    3.009    0.000    5.219    0.000 votable/converters.py:790(parse)
       15    2.471    0.165    2.490    0.166 numpy/ma/core.py:8044(__call__)
  7068870    1.819    0.000    1.822    0.000 votable/exceptions.py:129(warn_unknown_attrs)
  7375570    1.091    0.000    1.091    0.000 {method 'get' of 'dict' objects}
7273900/7273754    1.020    0.000    1.020    0.000 {built-in method builtins.len}
  7068870    0.859    0.000    0.859    0.000 {method 'keys' of 'dict' objects}
      303    0.560    0.002    0.562    0.002 numpy/ma/core.py:3306(__setitem__)
  1645500    0.329    0.000    0.333    0.000 {built-in method builtins.isinstance}

EDIT: Profiling result formatting

Thanks for the info. From your profiling result, looks like the bottleneck is at:

astropy/astropy/io/votable/tree.py

Lines 2430 to 2434 in 045f049

    
           def _parse_tabledata(self, iterator, colnumbers, fields, config): 
        
               # Since we don't know the number of rows up front, we'll 
        
               # reallocate the record array to make room as we go.  This 
        
               # prevents the need to scan through the XML twice.  The 
        
               # allocation is by factors of 1.5.

Lots of Python loops (usually to speed this part up is to move the loops to C level). Also if your table somehow hits the except blocks a lot, that would have performance penalty as well. More investigation would be needed to see what is the most painless way to obtain the most performance boost.

@rgreimel

@rgreimel Thank you for reporting this and running the profiling. It's very useful feedback to see where the spots are that we need to work on in astropy.

I had the glorious idea of adding @numba.jit on _parse_table - but that did not do much (I had hoped it would - but not expected it). just FYI.

@wkerzendorf

Thanks for trying anyway, @wkerzendorf !

I recently ran into the same problem. The file I'm working with is a 3.6 GB VOTable downloaded from the Gaia archive with 73 million rows (which you can download from the ESA VOSpace here). Looking inside the file, it contains a binary64 STREAM.

I tried loading the file using astropy.table.Table.read(), but gave up after 2 hours. I verified that TOPCAT was able to load the file on my laptop, in about 15 minutes. I exported the file to FITS, and then tried loading it on another computer. Loading the FITS file only took a few seconds, which means that for large tables like this, astropy's binary VOTable implementation is at least 1000x slower than its FITS implementation.

Interestingly, the Gaia archive hangs when I ask it to export this file as a FITS table, which suggests that the Gaia archive may be using astropy's VOTable implementation on the backend.

Does someone who understands the binary VOTable implementation know what the bottleneck could be here? I would assume that reading in binary data should be fast: read in the metadata, read in the binary stream, check that it's the correct length, and then use it to assign an array.

@gregreen

@gregreen , did you try to profile the code? Was it similar to above or something else?

Also, is your machine memory swapping?

Might have been improved now but someone can check...

Add Cython implementation for VOTable binary converters #18454

@gregreen

I'm reconfirming here that 8 years later, astropy TABLEDATA format reads are still slower than STILTS, though speed improvements in both Python and numpy have improved things somewhat.

Using the same Gaia DR2 cone search from above #8191 (comment) it now returns 44,260 rows and 100 columns (75 MB) rather than 73,000 rows (114 MB). Timed the same way as above

t0 = time.perf_counter()
tab = Table.read("gaia.vot", format="votable")
print(time.perf_counter() - t0)

time java -jar stilts.jar tcopy in=gaia.vot ofmt=fits out=/dev/null

Best of 5 on an Apple M3 Pro (macOS), numpy 2.5.4:

	Python 3.12	Python 3.13	Python 3.14	OpenJDK 25
astropy 8.0.2	3.48 s	3.66 s	3.41 s
astropy main (`952407c`)	3.40 s	3.61 s	3.38 s
STILTS 3.5-6				1.00 s

STILTS includes about 0.2 s of Java virtual machine startup. So astropy is about 3.4x slower on this file, against about 8x (38 s vs 4.8 s) in the report.

Our reader itself has not sped up much since 2018: on the same laptop, run as x86 under Rosetta so they can be compared.

astropy	Python	Time
3.0.5 (2018, when the issue was filed)	3.7, x86 under Rosetta	7.08 s
4.0.6	3.8, x86 under Rosetta	6.83 s
7.1.1	3.13, x86 under Rosetta	5.78 s

Most of the time is still the per-cell Python loop in _parse_tabledata, as in the profile above. Speed improvements come from newer versions of Python and numpy.

The BINARY case from @gregreen (hi Greg!) in #8191 (comment) is separate: #18454 (in 8.0) made the BINARY converters faster, and #20828 (open) reads BINARY and BINARY2 a column at a time, and in some cases, makes them much faster.

So there is still work to do on the TABLEDATA format.

A few things to add here:

STILTS runs on more than one core, as the Java Virtual Machine compiler and memory management run on a separate core from stilts.jar. And that's 1.6 s of CPU time vs 1.0 s of wall time. astropy (within Python) runs on a single core. Measured as CPU time, astropy is ~2.2x slower.
For this Gaia table, astropy uses about half the peak memory as STILTS (0.14 GB vs 0.32 GB)
STILTS records no per-cell warnings or positions and quietly turns bad values into nulls. astropy builds masked numpy columns and can report the line and column of every problem. There is some overhead to that, and all of that is in _parse_tabledata.

Speed up reading VOTable TABLEDATA cells

Navigation Menu

Repository navigation

votable read slow #8191

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def _parse_tabledata(self, iterator, colnumbers, fields, config):
	# Since we don't know the number of rows up front, we'll
	# reallocate the record array to make room as we go. This
	# prevents the need to scan through the XML twice. The
	# allocation is by factors of 1.5.

votable read slow #8191

Description

Activity

pllim commented on Nov 27, 2018

rgreimel commented on Nov 28, 2018

pllim commented on Nov 28, 2018

hamogu commented on Nov 29, 2018

wkerzendorf commented on May 6, 2019

pllim commented on May 7, 2019

gregreen commented on Sep 27, 2022

pllim commented on Sep 27, 2022

pllim commented on Jun 10, 2026

jdavies-st commented on Nov 12, 2026

jdavies-st commented on Nov 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions