io.ascii to parse link and read in full version of tables

@hamogu

AAS journals often include full data tables as "electronic only" versions. In these cases the HTML or pdf version of the paper only contains a small subset of the full table, therefor parsing the html/pdf with io.ascii leads to an incomplete dataset.

The HTML version of the paper however includes a link to an s3 resource, albeit with an expiration date, so manually harvesting that link and adding in an analysis code is not suitable and reproducible enough as the link will eventually expire.

Feature request:

parse the HTML, find this link for the full version of the table, read that one in into an astropy Table.

Example for one such Table in a paper is Table 2 in https://iopscience.iop.org/article/10.3847/1538-4357/ac743f#apjac743ft2

cc @hamogu and cc @augustfly as this is as much as an AAS journals issue as an astropy one

EDIT from @taldcroft 2023-Dec-14

After considerable discussion and some quick prototyping, it appears that the original feature request is not technically feasible due limitations in the upstream data provider(s). In particular, programatic HTTP requests for data tables are seen to be blocked by a Captcha prompt. This is circa Dec 2023 and may change in the future. People encountering this issue are encouraged to read the full comments for the details and a wide-ranging discussion about the future of data in astronomy.

Does this issue supersede the following?

BUG: ascii.html cannot handle table ids properly #14346

No. I mentioned this issue in the comments for that one, but that one is clearly for a bug of parsing a table incorrectly, even though it's fully included in the HTML file, this one is about dealing with a UX "feature" of the HTML version of the AAS journals, where one needs to do one more click to get to the actual table data.

I am super happy to track the issue here, thanks! Let me know if there's something needed from me.

Does this issue supersede the following?

The common theme between the two is that there are no tools available that properly parse data tables out of astronomical journal files, and ideally astropy.io.ascii could/should do it out of the box (luckily the field doesn't really have that many journals, so their pecualities can be taken into account even in a generic, though astronomy flavoured, library).

Does AAS have a dev to help implement this here? If so, a PR is very welcome!

Sorry, this thread is currently super under-specified, and my reply is going to come across brusque as a result. I do not know what problem is trying to be solved though I'd love to help enhance reproducibility. Is the core issue defining "reproducible enough" as stated above? Do you have some specs that you can point me to on what that means?

Here are some questions based on this thread that I hope will help me understand what is desired.

Is the problem that there is a short online table and a longer complete one? That's not a problem from our perspective; that has been the case for 23 years. It will not change. While our HTML has changed considerably over time (and all flavors are out there floating around), we are not going to provide HTML versions of useful data when we create MRT versions to replace them.
Is it that it takes more than 1 click to get to a table? Again, I don't consider this to be a problem to be solved. I feel that you have some "Journal landing page specifications" that aren't clearly stated, including "how many clicks it should take to get to a data file." A simple summary would help me and probably other journals out a bunch.
Is it that the table reference is inconsistently tagged in the HTML in some way so that makes it hard to find the S3 link? In what way? Can it be detailed? I'd love to send an issue up to the publisher to fix their HTML to make these links clearer. Screen-scraping HTML is dangerous business as there's no requirement to freeze the HTML format of any webpage anywhere (You'll find many generations on our site). The underlying problem is not dissimilar from folks who believe that they can use Zenodo URL file links for reproducibility -- nothing about plain URLs is preserved as any requirement of any website. The other issue (14346) about table references linked to this issue isn't from an AAS Journal article.
Is it that there is an S3 link that you cannot parse? The target of every S3 link in an machine-readable table that parses with format="ascii.cds" in io.ascii 95-99% of the time, throwing only warnings bc astropy.io can't understand log units in our format. The other 1-5% of the time the data is in FITS. Do you want us to help write an S3 link parser to help mime-type identify the content? That might be useful!
Is it that HTML renderings of tabular material are inconsistent across all journal websites? That's not a problem to be solved because we have a long-form data type that we create (and lives at the S3 link). It's also not a problem to be solved bc there are no requirements for HTML websites to be stable over time (sorry I keep saying the same thing over and over again!).
Also note that all the tables that don't have S3 links have tab-delimited versions that you can uncover in your HTML screen-scraping that we've worked fairly hard on to make cleaner than the HTML (though this is done algorithmically/automatically). For example, here is Table 1 rendered as tab-delimited. The tab-delimited versions aren't in key'd S3 buckets. The fundamental problem is that authors have near infinite ability to restructure their tables so predicting the header rows and columns is almost impossible, which is why we make the MRT versions that live on the S3 links.
Is it that the S3 link expires? Yeah, I am right there with you and really hate this. I've reported it to the publisher as a big a problem for reproducibility, but it is not going to be changing any time soon bc it is some security device put in place by our publisher. Since you are OK with screen-scraping HTML, could you add a call back to get the newest AWSAccessKeyID from the HTML whenever the dataset needs to be re-retrieved? Seems like this must be a common function for key-signed S3 buckets (but I believe you both know 1000% more than I about AWS).

let me add that we do have a procedure to spin up & fund small software projects to build features like this. I just need a lot more information that I feel is currently missing! thanks

Is the problem that there is a short online table and a longer complete one? That's not a problem from our perspective; that has been the case for 23 years. It will not change. While our HTML has changed considerably over time (and all flavors are out there floating around), we are not going to provide HTML versions of useful data when we create MRT versions to replace them.
...

no, it's not a problem that there are multiple tables, the problem is that there isn't a machine-ready long table version that doesn't require interactive involvement with the website. astropy.io.ascii is capable of reading in tables from HTML pages, and it indeed finds the short version. The feature request is for making it able to find the s3 link and read it in without user interaction. (See the referenced notebook above, we did copy pasted the s3 link to read it in, but the link that used to work a week ago has already expired and now throws a 403 making the data acquisition part of the notebook unreproducible, unless one goes back to the HTML page, clicks on the link again and copy-pastes).
So, yes, I think scraping the s3 uri is the desired solution here, but I would like astropy to do it out of the box, as it affects one of the 3 major astro journals, rather than I as a user need to code it up for my notebook, so does the next user.

(ultimately, of course, getting these datasets into e.g. vizier would be a nicer solution, or any other reasonably standardized data-sharing platform, that provides easy-to-use, generalized API access. But until then, as a workaround, if astropy provides HTML table reading capabilities, it would be nice if the reader would be fine-tuned to be able to handle astro journals)

Hope this helped to clear the vagueness around the feature request.

@bsipocz

I agree with @bsipocz : AAS journals should fix they way tables are stored. Personally, I would prefer to have all suitable AAS journal tables go to VizieE automatically, in the same way that A&A table do. That's not just easier to get through an API (though I admit that reading the "machine-readable tables" is not easy with a machine), it also allows searching through Vizier' I often find A&A articles, because I search some object in VizieR and then see that it's listed not only in GAIA, 2MASS, etc, but also in some table of a much smaller paper that might only deal with 10 sources.

That said, the second best thing is AAS journals to have persistent links for data tables.

And only last, as a band-aid, should astropy implement a work-around. The goal of io.ascii readers it to be able to say "read that article and get me table 2". As Brigitta says, that works in the HTML (thanks for formatting correctly as HTML), but then you only get the top 5 lines. A solution could either be "follow the link to the MRT and parse that" or "parse the full table in HTML".

Of course, one could also manually download and save the table you get from s3 (That's what I typically do. I then add it to the git repro for the project I'm working on and commit with "downloaded manually from ApJ article XYZ on the date of this commit". Not ideal, but better than nothing.) So, I think it would be a useful, but not high-priority feature for astropy to be able to point it at and ApJ HTML and say "get table 2, follow link to s3, then parse that table".

@hamogu

I agree completely with @hamogu and @bsipocz above on all points and the priorities. I confess to not being involved in active research and I find it quite surprising that AAS journals are not making every data table available for download with a permalink or (better) via documented API. VizieR seems like an excellent idea for the reason @hamogu mentioned.

Requiring scientists (or astropy code) to scrape HTML to get at data is a poor solution. That said, some kind of utility to scrape the AAS data download links from HTML could be a near-term workaround. This wouldn't necessarily be part of io.ascii but could be a specialized function in astropy.utils.data to get links that could then be used in io.ascii.

It is also worth noting in the example article (https://iopscience.iop.org/article/10.3847/1538-4357/ac743f#apjac743ft2) that the downloaded "ASCII" first table is not in a machine readable format at all. AAS certainly has the power to enforce that all submitted tables are actually machine readable. There is really no general way for astropy to read this table even if the download link is scraped from the HTML.

Table 1 
CLQ Candidate Selection


TDSS Target Type	N _QSOs	N _Cand	% Cand	N _CLQ	% CLQ	
A	17,511	0	0.0	0	0	
B	44,832	1	0.0	0	0	
FES_DE	653	9	1.4	6	0.9	
     ... <snip>
RQS3v	1579	4	0.3	0	0	
Total	64,039	61	0.1	19	0.03	
Note. Target selection methods for our TDSS CLQs. <snip>

I agree with the sentiment that AAS should indeed enforce good practices. OTOH, I'm only interested in Table2, that's the one with the s3 holding, and that one is luckily machine readable. It's not super clear though why it only has 167 objects as opposed to the table caption of 169 objects.

from astropy.io import ascii
ascii.read('https://cfn-live-content-bucket-iop-org.s3.amazonaws.com/journals/0004-637X/933/2/180/revision2/apjac743ft2_mrt.txt?AWSAccessKeyId=AKIAYDKQL6LTV7YY2HIK&Expires=1681940192&Signature=WqLIyKg10DdAH5CsYISejHjw40Q%3D')

@pllim

Thanks to @pllim to pointing me to this issue. I would just like to add that it would be good to also think about how to handle notes when importing tables from the standard ApJ Table format. The current way of reading in tables seems to ignore this block of text, which is reasonable given how flexible the contents can be, but I think it would be useful to at least read it as a string that can then be parsed by the user.

@hamogu

ping @hamogu - this should really be something we get to the bottom of, now that there are paid people. Or just ping AAS to have a fix from them. Either way our science users run into the problem each time a new project is started up making the process more inconvenient than it should be.

There is a tiny bit more than the ones that load directly. Some authors do the effort to submit their data directly to CDS to see if the data could be accepted into VizieR, and some additional tables are selected by the team to be standardized. I was just suggesting that any parser for the html could make use of that and test if this is already existing in MRT somewhere.

@augustfly

I would have said that you want to take an article DOI, walk the HTML to find all data links, provide a summary of those links to the user to "pick" the one to download, use, and parse or summarize the contents of each data link. If so then I think the captcha problem comes in the multiple repeat "hits" to the server -- the initial HTML parse went OK, but the captcha was hit when an individual data file was downloaded. This language -- "multiple or sequential repeat hits to the server triggering the captcha" is IOP's language. Any clarification you can provide may help us argue for removing that problem. I'm hopeful at least.

@augustfly - that's basically it. I hit a captcha on my very first try to download a data table in that notebook. I just tried to re-run it and got a captcha just trying to download the original article HTML. So I would say that any naive programmatic access to IOP content at this time is unreliable. By naive I mean that experts might understand how to fool the IOP servers but I do not -- my simple requests.get(url) often results in a captcha.

I was just suggesting that any parser for the html could make use of that and test if this is already existing in MRT somewhere.

That's a great idea; I'm just not sure how to do that in most use cases. It depends on what people put into the parser, e.g. if they download that html file and work on that locally, it's going to be hard to scan the HTML for identifying marks, go out to VizieR, look for something similar, ans ten work on that. And by "hard" I mean, it requires a lot of hand-coding to make it work for the major astronomical journals where the html all looks different etc - that's probably beyond the scope of astropy.io.ascii and better done at e.g. ADS.

However, the low-hanging fruit is probably to point that out in the docs, something like "many tables in this format can also be found in that format which is much easier to parse - looks at the ADS and see if there is a link to format X" or something like that.

@taldcroft

@taldcroft thanks; really appreciate it. I had never gotten the captcha clicking around on data links inside an article before, but I very often get it picking an article to read (bc I audit/review 5-10 articles at a time, which I guess IOP thinks is abnormal).

tl;dr -- I see a won't fix label. Does that mean we should close this issue as won't fix?

While we won't fix the full issue at this time, there are still aspects of this discussion that we can work on (for example, I'm trying to see if I can make an ASCII reader that can read a large majority of the ApJ "ASCII" format tables.

Also, the AAS journals (see comments above) might change the way those tables are referenced, so some of the issues that stop us from fixing this (like the captchas or very short-lived S3 links) might go away in the forseeable future. If that happens, we'll remove the "won't fix" label. Thus, I think this is one o those (slightly annoying) cases that have a "won't fix" but should stay open.

Hi.

Around the time of my last comment we (the AAS) issued a request to IOP to remove the keyed S3 links and fix the CAPTCHA problems for data files.

IOP removed the keyed S3 links to all data files around January 12th of this year, and we met with IOPScience last week to get some additional details on how they implemented static links to the data files. I'm working on trying to summarize how the links now work, which I will post soon. Here are four articles I'm using as examples.

Article 1: https://doi.org/10.3847/1538-4365/aceda8 (Stubbed MRT and data behind the figure. Also versioning)
Article 2: https://doi.org/10.3847/1538-3881/aced3c (Tar file of other stuff)
Article 3: https://doi.org/10.3847/1538-4357/acb432 (Stubbed and unstubbed MRTs; also useful for the breadth of “tables” authors make)
Article 4: https://doi.org/10.1086/340554 (pre-IOP data links; yes, I'm using my article)

The CAPTCHA problem hit when crawling an article is not fixed; they are wondering if there are URL request hints that could be negotiated to sort of "white list" astropy access. We imagined that the CAPTCHA problem would manifest differently when developing a tool, running test/CI, and in a user context. We also wondered if with a stable link structure, crawling an article is still necessary.

What would a conversation between developers here and IOPScience look like -- to work around the captcha issue? I’m prob not doing the best job ferrying developer-level conversations across the Styx. I doubt they are going to come on GitHub to talk this through on an issue thread, but we could ask.

This is fantastic news and thank you for pushing on that. I'm happy to engage with IOP if you think that helps, but what form is the most efficient to do that? If they are willing to look here, that would allow all interested parties to be on the same page, but if you believe that, e.g. an email conversation or a zoom call with 2-3 users/astropy developers is more likely to convince them, then we can organize that, too (and ask here for volunteers to join that call or to provide any additional input).

@hamogu

@hamogu Cool! It is unclear to me how best to move forward on the CAPTCHA part. For instance, I just received the following request from IOP that has an efficiency I can't adjudicate:

Would you be able to get some more information for me, possibly the easiest next step would be for the people who are running the automated requests to do it again and let me know:

Time of the requests

URL being accessed (if possible)

IP address that the software is being run from

We can then inspect our log files and hopefully can find some kind of signature in them that we can use to disable the CAPTCHA for those requests

One way I could try to do this is to take @taldcroft's notebook https://gist.github.com/taldcroft/00911ca21cddf43f82c0c3365ea3b219 and run it locally until I hit a captcha and then send my information to IOP.

This presumes that the notebook will look anything like production code or even CI/test requests. But it is a start I'm happy to make with IOP to keep them moving forward.

@augustfly

@augustfly I would suggest to do exactly what you suggest: Run the notebook and see if that helps. My best guess is that the only distinguishing feature will be the user-agent in the header. For requests using e.g. the urllib from Python, it's going to be set to something like "python-requests/2.28.2"; if you use a real browser, it's going to be set to something like 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'. If IOP is willing to open it to all Python requests, then it does not matter how exactly the script is written, because all requests using the standard libraries will have the same user-agent (in that case, one might wonder what the Julia users will say, but Python is by far the largest community that has asked for this, so that would still be a good place to start.). But maybe there is something else in their log files that might turn out to be useful. I'm not a web-developer...

Super. I ran the test, triggered the captcha and sent the necessary information to IOP. I was able to get 1 data file before being blocked. Then then nothing.

Wow, great to see some real progress here, and I'm glad that my quick notebook ended up being useful!

About allowing a particular user-agent, I'm a little skeptical about this solution. This would amount to security through obscurity, guessing that folks generating unwanted downloads either don't use Python already or that they can't figure out that Python scraping is allowed.

Maybe more to the point is understanding the underlying reason for having a CAPTCHA. One presumes they don't want to allow people to just crawl and download the entire data archive? Or maybe this is protection from DOS attacks?

@augustfly

About allowing a particular user-agent, I'm a little skeptical about this solution.

It's up to IOP, not us to decide what they want to do. I was just speculating about what differences they may see in their logs and point out that at least in some characteristics (like the user-agent) the notebook will indeed look like production code or even CI/test requests, so it's not a futile exercise to answer IOP's request that @augustfly listed at #14556 (comment) at this stage.

@taldcroft

We are experimenting with referer passthrough. It's sort of working. I'll post my revision of @taldcroft's notebook as soon as IOP has done some more thinking if this passthrough is legit or not.

And yes, all forms of DDOS and other platform service slowdowns/interruptions are the reasons for the CAPTCHAs. That is what we are told anyway.

@taldcroft

A proposed work around is here (my butchering @taldcroft's notebook). It is a simple header referer pass through. The referer URL (proposed by IOP) to add to the (empty) example is "http://www.astropy.org/AAS".

I was able to walk any file or access any of the data links using that referer URL. I was able to get a variety of unforced errors in the data once retrieved, but that's not related to access question for these files.

Their answer to the question, "what happens if someone nefarious starts using that referer URL?" was "we are creating a small risk that a bad actor could do the same thing, but if that happens we can turn off the bypass rule very quickly".

Navigation Menu

Repository navigation

io.ascii to parse link and read in full version of tables #14556

EDIT from @taldcroft 2023-Dec-14

53 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

io.ascii to parse link and read in full version of tables #14556

Description

EDIT from @taldcroft 2023-Dec-14

Activity

pllim commented on Mar 20, 2023

bsipocz commented on Mar 20, 2023

augustfly commented on Mar 20, 2023

bsipocz commented on Mar 20, 2023

pllim commented on Mar 20, 2023

augustfly commented on Mar 20, 2023

augustfly commented on Mar 20, 2023

bsipocz commented on Mar 20, 2023

hamogu commented on Mar 20, 2023

taldcroft commented on Mar 28, 2023

bsipocz commented on Apr 12, 2023

gmzsebastian commented on Dec 13, 2023

bsipocz commented on Dec 13, 2023

53 remaining items

ManonMarchand commented on Dec 21, 2023

taldcroft commented on Dec 21, 2023

hamogu commented on Dec 21, 2023

augustfly commented on Dec 21, 2023

pllim commented on Jan 10, 2024

hamogu commented on Jan 10, 2024

augustfly commented on Jan 29, 2024

hamogu commented on Jan 29, 2024

augustfly commented on Jan 29, 2024

hamogu commented on Jan 29, 2024

augustfly commented on Jan 29, 2024

taldcroft commented on Jan 30, 2024

hamogu commented on Jan 30, 2024

augustfly commented on Jan 31, 2024

augustfly commented on Feb 6, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions