Repository navigation
BUG: ascii.html cannot handle table ids properly #14346
Description
Description
E.g. have this paper from MNRAS, from which I would like table2: https://academic.oup.com/mnras/article/491/4/4925/5634279
It has id='tbl2', so one would think the line should work:
from astropy.io import ascii
ascii.read('https://academic.oup.com/mnras/article/491/4/4925/5634279', htmldict={'table_id': 'tbl2'}, format='html')
But it doesn't, eventually, somehow bs4 identifies 20 tables from the paper with multiple indexes for the same one, so eventually calling Table 2 with index 5 does the trick:
from astropy.io import ascii
ascii.read('https://academic.oup.com/mnras/article/491/4/4925/5634279', htmldict={'table_id': 5}, format='html')
This is far from ideal, and one needs to be someone comfortable with bs4/debugging to find a workaround for a simple problem.
Suggestion: include html parsing all from AAS, MNRAS, and A&A papers in the test suite (given that those are the use cases for having a html table reader in astropy), so these problems would be noticed sooner.
[Edit:] A separate issue is when the full version of a table is linked in the "electronic version" of the paper, rather than included in the html. Ideally it should be possible to obtain these tables using astropy. E.g. Table 2 from this paper:
https://iopscience.iop.org/article/10.3847/1538-4357/ac743f
Activity
pllim commentedon Jan 31, 2023
Interesting, do science papers still not provide tables as machine readable files?
bsipocz commentedon Jan 31, 2023
🤷♀️ - maybe upon reasonable request? :)
But if a paper is not ingested by vizier or ned I think it's still a very present pain point (without needing to click through stuff, e.g. the pop-up, in this case, is easily readable: https://academic.oup.com/view-large/190441108, but it's sourced from a front-end detail)
pllim commentedon Jan 31, 2023
I wonder if pandas can ingest it or has similar problem. 🤔
bsipocz commentedon Jan 31, 2023
Pandas had different problems with it; a bug from 2019, and then a 403. So I gave up digging further this rabbit hole.
hamogu commentedon Feb 24, 2023
I have not looked at this particular table, but I often found our latex parser better behaved: Either it works, and if it does not, it's hopeless (table rows broken over multiple lines,
\multicolumnentries etc. However, it requires downloading the source from arxiv.Anyway, I agree that this is a bug that should be fixed and tested better.
bsipocz commentedon Mar 1, 2023
I've added a separate, but somewhat related issue that sometimes AAS journals have only a truncated version of the tables included in the html, and they link to the full version separately. Now, ideally, it should be possible to access those full versions as the snippets are not really useful for any further work.
hamogu commentedon Feb 8, 2025
The MNRAS html file actually contains most tables more than once, as @bsipocz said above ("eventually, somehow bs4 identifies 20 tables from the paper with multiple indexes for the same one"), with slightly different CSS/js markup and the CSS/js [*] will then decide which of the version will actually be visible on the screen. So, that's not an error of beautifulsoup/astropy, that's an accurate representation of what's actually in the html file. See, e.g. for "Table 1" line 1944, 1946, 1950, and 1952, which all have the same "1411 364" (the other cells are not visible in the screenshot because the lengths of the line is longer than my screen):
If you load the html file without the css in the path, you actually see each table rendered four (!) times (now all looking the same because I'm not loading the CSS/js files that do the magic):
Thus, beautifulsoup finds a lot more tables than the reader of the rendered webpage will see. The articles has 5 tables, and beautiful soup finds 5*4 + 1 = 21 tables, where the "+1" is a table of citation numbers to the article, that is not visible on the website by default, but can be folded out with some button.
That is why, as @bsipocz reports "calling Table 2 with index 5 does the trick": Table 1-4 are all copies of table 1, and table 5-8 in the html are what is rendered as table 2 in the article. And indeed
ascii.read('5634279.html', htmldict={'table_id': 8}, format='html')gives us exactly the same datatable asascii.read('5634279.html', htmldict={'table_id': 5}, format='html').I need to dig more to see why the following doesn't work. I suspect it's because the "tbl2" is used more than once (in two of the four copies of that table) and not in a field called
table_id.I'll keep investigating, but I'm writing down some ideas for what I could try when I get back to this so I don't forget:
[*] I'm not a web developer and for the purpose of debugging this, I don't need to understand how exactly they do that; it's enough to understand that they do that.
hamogu commentedon Jul 23, 2025
@taldcroft @dhomeier I would like your opinion on this bug/feature request as maintainers of
io.ascii.Essentially the problem is (@bsipocz please correct me if I don't summarize the intention of this bug report correctly) the following:
Our html table reader fails to read tables from astronomy journal articles unless the user goes pretty deep into the bowls of beautiful soup/html formatting. Of course, the html reader was originally added to do exactly that: Read tables from papers. Ideally, we'd have a one-stop-shop where the user puts in the URL of the article and the number of the table they want to read and gets back and astropy table.
I've done a few hours of investigation and looked a tables from ApJ, A&A, and MNRAS (just to pick the three most prominent journals) and found that it's going to very cumbersome to make that work. The problem is that the formatting of each journal is different and changes over time. So, in essence, we need a different parser for each journal and epoch of formatting they use. For example, the MNRAS article mentioned above (from 2020), lists every table 4 times but with different CSS and JS to get different formatting. Neither of those tables has a useful tag, but at least they are embedded in a
<div>with a useful tag, so we could search for divs, find the right one, pick one of the four and read that.In older MNRAS articles the html tables are not part of the article at all, instead the articles contains an image of the table and when you click on it you get to a separate page that actually renders that table as html. So, you would have to parse the html source, get the right link, download the page with that table, and parse it. In some cases, it seems that the second page also only has the table as a jpeg and not at html at all; I can't figure out why (maybe it has something to do with the exact time it's published, maybe the typesetting decides that based on the number of non-ascii symbols used in the table).
For ApJ, the earlier articles are displayed in iframes. Once you get the link for the iframe that actually contains the article itself (not the navigation) tables are in the end. They don't have and tags to search for, but you have to parse the html links (
<a>) order just the text headers (<h2>) that contain text liketb2orTable 2and then read the next table below it. Newer ApJ is similar to MNRAS in that the tables are displayed in separate pages.In ApJ, I've encountered captchas or "verify that you are human" even when I click on that by hand in a web browser, which indicates that it's going to near impossible to automate pulling down a link that astropy follows reliably.
A&A is similar to ApJ and MNRAS, except that all tags and formatting are different again (and I trigger "verify that you are human" less often).
So, while I personally really, really would like this feature to work, I don't think there is a way to make it work reliably enough in practice for it to be part of the astropy core.
I thus suggest to declare this out of scope, add a few sentences in the docs of the html reader to explain that it doesn't work this way and why and close this issue as "won't fix". What do you think?
The html reader itself it still useful, but requires more by-hand work from the user than it should. For example, if you (by hand) go to the pages the displays table 5 in an A&A article and "save page source" then the html reader can help you read that one table on that page.
bsipocz commentedon Jul 23, 2025
Yes, your summary is correct. If it's a close without fix, then I would recommend adding some, or even a lot of, warnings and notes to the docs about these caveats. As while some of the problems are trivial at user level (it cannot find the given table) sometimes it's way more subtle and will lead to wrong assumptions (e.g. only the first page of targets are returned, which is valid, but the user may assume they got all 600+ targets as opposed to just 40).
So my potentially controversial recommendation is to always issue a warning when the html reader is used pointing to the "caveats" listing in the docs and recommend to use vizier, or if it has to be html, then just be informed of the potential problems.
And even more controversial: consider deprecating the functionality from astropy.
hamogu commentedon Aug 4, 2025
I think this reader is still useful, just not for journals, so I wouldn't deprecate it entirely at this point. I've used it in the past to parse data from astronomers that have website either personal or for small-scale projects (too small to be in astroquery or to be ingested into e.g. MAST or IRSA), for example sometimes people host datasets from their papers on their personal webpage or maintain a list of "targets we looked at in this project" or "properties of debris disks collected from all the papers I published so far in a single table" . While that was a lot more common 10 years ago (I think efforts to convince people to put the data into more permanent places like journal articles or proper archives are paying off), it does still exist and those tables are typically just pure html with no complicated bells and whistles and are read relatively easily.
Maybe we don't need the html reader a few years from now as those type of homepages dies out, but for now I would keep it.
Explain why HTML readers might not be good choice for journal tables
Explain why HTML readers might not be good choice for journal tables
bsipocz commentedon Aug 7, 2025
Can we change the issue close status to "won't fix"?
hamogu commentedon Aug 7, 2025
I don't know how to change the close status, but I added a "won't fix" label.
pllim commentedon Aug 7, 2025
FYI -- I re-close it as "not planned" also. The option is under drop down arrow next to "reopen issue" for future reference.
However, the "not planned" thing is so new that I don't find it useful for historical tracking and I have no desire to go back and retriage 10k issues.