Repository navigation
IERS_A_URL is down, do we need alternative to fallback? #5194
Description
IERS_A_URL [0] seems to be down for a while now and throws a Forbidden 403 Error.
I wonder whether it makes any sense to introduce an alternative to fall back to in cases like this? E.g. [1] seems like a good one from https://www.iers.org/IERS/EN/DataProducts/EarthOrientationData/eop.html. Or the one from the ftp site listed there.
[0] http://maia.usno.navy.mil/ser7/finals2000A.all
[1] https://datacenter.iers.org/eop/-/somos/5Rgv/latest/7
Activity
bsipocz commentedon Jul 24, 2016
Run into this in astroplan astropy/astroplan#192
mhvk commentedon Jul 25, 2016
from the exchange with the IERS people that we had, the
iers.orgsite is meant as backup exactly in these kind of cases, so it would seem good to go with that. Also, a quick e-mail to themaia.usnopeople?bsipocz commentedon Jul 25, 2016
The site is back and again again.
However I'll try to come up with something to get the other url into our system, too.
taldcroft commentedon Jul 25, 2016
I made a comment on the astroplan issue that sort of applies here as well.
Has astroplan considered using the new IERS auto-download capability in astropy 1.2? I'm not sure if it would help with this test issue (when starting from a clean cache), but in general for users it should mostly make < few day downtimes of the upstream IERS site(s) invisible by means of the automated local caching.
The idea of the automated IERS handling in 1.2 is to grab the most recent IERS-A only when actually needed based on the date range involved and validity of the available (cached) IERS-A data. This means you hit the upstream site less often. Of course this doesn't really help with testing where one typically starts from a clean cache.
taldcroft commentedon Jul 25, 2016
Testing the automated IERS handling was a huge thorn, and it basically uses @remote_data everywhere to avoid pounding the IERS site with downloads and reducing astropy test times. With @remote_data on network-based tests then you don't suffer from Travis test fails when the IERS site is down.
wtgee commentedon Jul 26, 2016
Sorry if this is not the right place, but this line is where I am seeing the failures. Can we just specify a default
Falsevalue on this line?https://github.com/astropy/astropy/blob/master/astropy/tests/pytest_plugins.py#L137
crawfordsm commentedon Jan 23, 2017
I'm bumping this issue -- currently the URL is inaccessible from South Africa (and has been for a while) and the other URLs also have problems.
If we are going to support this type of calculation, I think astropy should have its own mirror or do the calculations itself. There is no guarantee that this URL or work will be available in the future and might in fact no longer be available very soon.
taldcroft commentedon Jan 24, 2017
@crawfordsm - yes, mirroring the IERS data is an idea that is overdue. As an immediate resource we have the astropy.stsci.edu/data server (data.astropy.org). This also begs the question of additional astropy data mirror sites.
@pllim or @astrofrog - who knows details of that machine? Can we set up a cron job to copy the IERS-A data after the weekly update of finals2000A.all? Likewise for IERS-B monthly?
astrofrog commentedon Jan 24, 2017
@taldcroft - the easiest thing would be to actually set up a cron job to push to the astropy-data repository. One way to do this would be to set up Travis on astropy-data and set it to run as a cron job to do the updates. I think this would be better than hiding the cron job away on the STScI server since we'd have no control over that.
taldcroft commentedon Jan 24, 2017
What is the current process for putting astropy data on the server? Doesn't astropy or some local representative have some sort of authentication to log in to a server and maintain that data?
Thinking about defining an infrastructure for other mirror sites, for example at CfA we require being on the VPN to log in to any machine. That's basically a non-starter for pushing from Travis.
So I can imagine providing a script that does the job of syncing with master data sources. Then we have a role as a remote-data mirror maintainer, with responsibility for keeping that script running. This would be a distributed model for mirrors instead of a centralized model where Travis pushes out to the mirrors.
crawfordsm commentedon Jan 24, 2017
I'd be happy to set up a mirror here (we mirror a few other things) but might need something that would push it here as I can't seem to access it.
astrofrog commentedon Jan 24, 2017
@taldcroft - just add data to this repository: https://github.com/astropy/astropy-data - there is a post-commit hook to sync to the STScI server.
astrofrog commentedon Jan 24, 2017
So the Travis job can push to the repository itself
taldcroft commentedon Jan 24, 2017
Ah I see, that's clever. However it still requires a CGI script on the other end that is listening.
I can tell you that because of web security issues, putting a publicly accessible CGI script on a CfA server is a pain. It needs to be formally reviewed by an external auditor for security. If other institutions are not that strict then they are being naive (and this comes from actual bad experience at CfA, not just theory). On the other hand, just serving static data files is no problem. So from my perspective it is much easier to have a cron job running that pulls data and puts it into a directory for a static data server.
I am imagining a simple cron script that essentially rsyncs a confiigurable "master" site (defaulting to data.astropy.org) into a local directory that is exposed to a static server on the mirror site. Anyone can set this up really easily, so we can have lots of mirror sites and it requires no further maintenance on our part. We might have the cron script do something like access a different CGI on data.astropy.org to register itself and provide notification that it has completed the sync. This could update a list of mirror sites which would be available itself on data.astropy.org (and the mirrors!).
In this way we have a distributed network of mirrors that requires no centralized maintenance (i.e. keeping the Travis CI webhooks up to date), and mirror owners only need to serve static data instead of setting up CGI script servers (which as I said is a whole new can of worms for web security).
taldcroft commentedon Jan 24, 2017
But getting back on topic, adding a Travis cron job to push weekly updates of the IERS-A file and monthly updates of IERS-B into astropy-data sounds like a great idea. This is independent of the more general idea of mirrors for data.astropy.org. Along with putting IERS data onto our data server would be the plan of changing the default IERS URLs and also generally allowing for arbitrary lists of mirror sites for the function(s) that get data.
I am also really 👍 on hitting astropy-affiliated servers for testing. We would feel less guilty and have higher reliability (hopefully).
pllim commentedon May 1, 2018
Closing this as a duplicate of #5059 as @bsipocz suggested. The other issue links to the same problem in
astroplan, which I think is a bit more helpful, but info from both issues should be taken into account if we ever get around to implementing this.