Repository navigation
Parsing angle/coordinate strings is slow — mostly Python (PLY) overhead #20131
Description
What is the problem this feature will solve?
Creating an Angle (or SkyCoord) from a large array of strings is dominated by Python overhead, not the actual work.
import numpy as np
from astropy.coordinates import Angle, SkyCoord
ra = np.array(["12h41m11.1s"] * 100_000)
%timeit Angle(ra) # ~1.3 s
ra = np.array(["12h41m11.1s"] * 20_000)
dec = np.array(["+41d16m09s"] * 20_000)
%timeit SkyCoord(ra, dec) # ~0.7 s (-> ~36 s for 1M sources)Every string is parsed one at a time: Angle.__new__ loops [cls(x, unit) for x in angle] (astropy/coordinates/angles/core.py, around L184), and each call goes through the pure-Python PLY lexer/parser (_AngleParser / parse_angle in astropy/coordinates/angles/formats.py). Profiling 30k parses, ~94% of the time is in the PLY yacc/lex machinery; the actual number-crunching is negligible. So reading a coordinate column from a text catalogue (VizieR/CSV/user input) is slow — ~13 µs per angle, ~36 µs per hms/dms coordinate.
Describe the desired outcome
A fast C parser for the common formats, with the existing PLY parser kept as a fallback — the same fast-path/fallback design as the time-string parser in astropy/time/src/parse_times.c (which already does the reverse, string -> jd, direction).
One difference worth flagging up front: angle strings aren't fixed-width like ISO times, so this can't reuse parse_times.c's position-based approach. It would be a small char scanner (optional sign, up to three number+delimiter groups, optional N/S/E/W and unit) wrapped as a gufunc that loops in C and emits a value + unit code per element. Anything it doesn't recognise — non-ASCII (°′″), simple units, exotic forms — sets a status flag and falls back to PLY for just those elements, so the output stays identical to today.
The Time parser sped up its direction by ~20x, so I'd expect a similar order here — very roughly a second or two instead of ~36 s for a million coordinates.
Additional context
Relevant code: _AngleParser / parse_angle (astropy/coordinates/angles/formats.py), the per-element loop in Angle.__new__ (astropy/coordinates/angles/core.py ~L184), and the template to mirror — astropy/time/src/parse_times.c with its _parse_times glue and setup_package.py hookup. This is the input-side companion to the output-side formatting work in #19992 / #20130.
I'm happy to work on this. Since it's a bigger, hand-written C scanner than the fixed-format time parser — and a second parser to keep in sync with PLY — I wanted to check you think the speedup is worth that before I start. @taldcroft @mhvk, you both wrote parse_times.c, so I'd really value your read.
Activity
veyron-kairo commentedon Aug 10, 2026
@mhvk since we are deep in the formatting side over in #20130, would you have a view on this one too? It is the same problem on the input side, where parsing still goes through PLY per element at roughly 13 µs a string.
No rush at all. Mainly I would rather not start writing C if you think it is not worth another parser to maintain.
taldcroft commentedon Aug 10, 2026
@veyron-kairo - this seems like a good idea to me. Coordinate formats are not really evolving and once it works then I would not foresee much further maintenance. That said, C with numpy / python is not my strength. I got a lot of help on the time parser from @mhvk and the same would apply here.
mhvk commentedon Aug 10, 2026
I'm slightly hesitant, since I think reading is typically less of a bottleneck that writing, and I don't really like having multiple pieces of code doing the same thing. That said, obviously Tom found that for
Timeit was useful to have a fast reader, and, as he notes here, it is not like the code will need much maintenance.One more thought: do you think that, if we have a more flexible reader, it could be used for
Timetoo (presumably having a fixed-width option is fairly easy)?On a different note, I recall that numpy's
loadtxthas had a big overhaul, making it way faster; possibly one can use that, or if not, at least get good ideas from it (theerfadiscovery over on the output side has reminded me that it is good to avoid doing things "in house" without looking really carefully if there isn't useful stuff already in our dependencies!).taldcroft commentedon Aug 10, 2026
Thinking more, is there a real world case where this improvement would make a substantial difference in net performance? Most commonly a catalog with sexigesmal coordinates would be read from an internet server, so likely that retrieval would end up being a bottleneck.
So maybe it is worth trying to quantify what kind of net improvement would be possible assuming a factor of 10 improvement in the parsing?
veyron-kairo commentedon Aug 10, 2026
@taldcroft fair question, I measured the full pipeline.
For a 200k-row VizieR-style catalogue (7.9 MB),
Table.readtakes 0.05 s whileSkyCoordparsing takes 6.2 s, so parsing is ~99% of the local cost. At 25 MB/s, downloading the file adds only ~0.31 s.With the vectorised parser, read + parse is 28.7x faster from disk, 12.3x at 25 MB/s, and 4.3x at 5 MB/s. With 5 s server latency that drops to 2.1x, and with 30 s latency to 1.2x. So it matters most for local survey/Gaia-scale files or fast servers.
@mhvk I also revisited
loadtxt, and I don't think C is necessary.np.loadtxtcan tokenize an iterable of strings, so normalising separators withnp.stringsgives a vectorised fast path with per-element PLY fallback. That's ~30x faster on clean input and 18.6x end-to-end with 1% fallback. It degrades gracefully with more fallback and never becomes slower. Peak memory is also 774 MB vs 1737 MB for 1M strings.I checked correctness against our test-suite strings, VizieR data, 50k
SkyCoordpairs, and exactFractionarithmetic. Results are bit-identical, with 0 µas maximum separation. I also caught and fixed a grammar issue where mixed spaces/colons could incorrectly parse instead of falling back.One important limitation: PLY is 94% of
parse_angle, but parsing is only about half ofAngle(), the per-element object construction is the other major cost. So the parser alone tops out around ~2x unless it returns arrays and avoids that loop.On whether this could help
Time, it depends on the format. The fixed-width ones have nothing to gain,iso,isotandydayalready parse at 0.06 µs/element in C. Butfitshas no fast path at all, precisely because it is not fixed width, and it lands at 2.0 to 3.0 µs/element depending on whether the scale suffix and long-year form are present, so 35x to 50x slower thanisotfor the same instant. That is where a more flexible reader would actually pay off.The prototype is ~100 lines and currently covers
Angle/SkyCoordarray input. Scalars, 0-D arrays, masked/object dtypes and exponent notation still need testing, and NumPy <2.1 needs aNUMPY_LT_2_1shim.If you're both happy with this direction, I'll turn it into a proper PR against
formats.pyand drop the C-parser approach for angles.mhvk commentedon Aug 10, 2026
@veyron-kairo - thanks for the detailed analysis. I think we should avoid C unless it gives a strong performance advantage, if only because it is harder to review (and make changes in the unlikely case that is necessary). So, 👍 to proceeding with a python version.
veyron-kairo commentedon Aug 11, 2026
Thanks, that settles it. The Python version is working locally: around 110 lines in
formats.py, normalising the separators withnp.stringsand lettingnp.loadtxtdo the conversion, with anything it does not recognise passed to the existing parser.It comes out 23x faster on a 200k element column and 20x for
SkyCoord, and bit identical to parsing one element at a time. I will open the PR once #20130 is rebased, since both touchformats.py.