Repository navigation
Explain why HTML readers might not be good choice for journal tables - #18501
Conversation
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
|
This is a good idea, I just think the language can be tightened. I don't know if we have data to support the frequency of failing (what does "often" really mean?), so just leave that out. Here is suggested text: Be aware that reading tables from published HTML journal articles may not work for a variety of reasons, including inconsistent mark-ups, CAPTCHAs, changing formats, embedded javascript, or the table actually being an image. If possible you should consider retrieving the table in a standard data format such as CSV or FITS, perhaps from an archive such as CDS/Vizier. |
|
I'm fine with that wording, too, but for the record: "often" means that I tried read MNRAS tables from two epochs of formatting, A&A from 2 epochs, and AAA journals with three different epochs (2 ApJ and I don't remember the other, I think I used the supplement series) (where "epoch" means that the journal pages looks very different to the eye, i.e. the template they use has changed significantly over time) and none on them worked out of the box. In some cases (I think two or three) I could work around that (e.g. find the page that represents the iframe with with the data, when "display page source" and identify the tag or table number needed), in the remaining cases I could not (e.g. the table is only available as a jpeg on the journal homepage). I didn't want to say "it never works" because I'm sure there are counter examples somewhere, but 0/7 is certainly "often". |
|
@hamogu - ah, that sounds worse than I thought. Apart from CAPTCHA, I think I was 1/1. Then maybe: Be aware that in many cases reading tables from published HTML journal articles will not work ... |
In astropy#14346 it was noted that the HTML table reader does not easily return a table from an MNRAS article. Deeper study shows that there are problems with any major journal, due to inconsistent mark-ups, captchas, changing formats, embedded JS, or that tables are only included as jpegs in some cases. Thus, it seems impossible to make a working table parser for even the top few journals (see details of discussion in astropy#14346). This PR tries ot summarize the findings in the API and narrative docs in the locations that I hope users are most likely to see it. The discussion of the original issue suggested more extreme measures (such as rasing a warning every time the HTML reader is used or deprecating it altogether) but I deem that too invasive, since the reader itself works and is useful for a number of legitimate cases, including the case where a user manually downloads the HTML for an article and reads it after verifying that the table in question is actually contained in the file. closes astropy#14346 [skip CI]
3f4e7b2 to
9b8c9b2
Compare
|
Not sure what's up with CircleCI. Was your branch not rebased on latest upstream/main ? Anyways, doesn't matter for this PR but we will have to investigate if it comes back for your other PRs. FYI. |
… choice for journal tables
…501-on-v7.1.x Backport PR #18501 on branch v7.1.x (Explain why HTML readers might not be good choice for journal tables)
dhomeier
left a comment
dhomeier
left a comment
There was a problem hiding this comment.
I see @pllim beat me to merging this, so should be good as is. Pointing out some lower-threshold alternatives to CDS might be helpful, but it's probably impossible to really estimate the chances something will work for a given user or not. I was not really able to test much of this as I only have limited access to non-OA journals.
| for a variety of reasons, including inconsistent mark-ups, CAPTCHAs, changing formats, | ||
| embedded javascript, or the table actually being an image. If possible you should consider | ||
| retrieving the table in a standard data format such as CSV or FITS, perhaps from an archive | ||
| such as CDS/Vizier. |
There was a problem hiding this comment.
perhaps add
or download via a "save as data/text file", if provided by the journal, or as next-best option try to open the table in its own tab.
Those may still not work in all cases (e.g. using the tab-url was reported to work in #14346 (comment), but I could not reproduce it due to permission errors), but could still work as a fallback, if people fail to hunt the data down on Vizier.
There was a problem hiding this comment.
Most of the "interesting" (meaning long data tables, not the 10 row tables where people report fit results and 3 out of those 10 rows have \multicolumn{} in them) A&A tables are on CDS, and almost none of the AJ/ApJ tables are. So this is imperfect advice. The question is if this is the right place to essentially write a tutorial for different ways to extract data from an article (I have, for example, also pulled the LateX from astro-ph in the past and then read the LaTeX tables). Maybe that would in fact make an interesting tutorial for astropy learn.
There was a problem hiding this comment.
Agreed that a comprehensive list of tips and tricks would be out of scope here. I was just thinking of some low-hanging fruits, like when people are already on the article page and are just one or two clicks away from a more digestible format. Ideally standardised electronic formats should of course be directly linked, but I'd consider those cases at least a bit of good will on the publishers' side, while being forced to go to a latex source file certainly qualifies as actively unhelpful! ;-)
Plus when ApJ directly links an MRT table as in the second example, but none is available at CDS, that's certainly an important option, though perhaps also already very obvious to the user.
Description
In #14346 it was noted that the HTML table reader does not easily return a table from an MNRAS article. Deeper study shows that there are problems with any major journal, due to inconsistent mark-ups, captchas, changing formats, embedded JS, or that tables are only included as jpegs in some cases. Thus, it seems impossible to make a working table parser for even the top few journals (see details of discussion in #14346).
This PR tries ot summarize the findings in the API and narrative docs in the locations that I hope users are most likely to see it. The discussion of the original issue suggested more extreme measures (such as rasing a warning every time the HTML reader is used or deprecating it altogether) but I deem that too invasive, since the reader itself works and is useful for a number of legitimate cases, including the case where a user manually downloads the HTML for an article and reads it after verifying that the table in question is actually contained in the file.
closes #14346