Repository navigation
IERS URL timing out #5059
Description
The IERS tables that describe Earth's orientation ordinarily get downloaded from this URL stored in the utils.iers module:
In [18]: from astropy.utils import iers
In [19]: iers.IERS_A_URL
Out[19]: 'http://maia.usno.navy.mil/ser7/finals2000A.all'
At present (and maybe for some time now), the above URL times out always. I found from the IERS webpage that there's an alternative server which is serving the table appropriately here, at
http://toshi.nofs.navy.mil/ser7/finals2000A.all
This is causing a bunch of mysterious IERS warnings and errors in astroplan. Should we switch the hard-coded URL in astropy to the backup URL until the normal server is restored?
I checked to see if the IERS webpage maintainers have an email address that we can contact, but they left only a phone number on the webpage (for the adventurous: 202-762-1518).
cc @eteq (@kvyh – heads up, astroplan tests may fail due to this problem even if the code you wrote is fine)
Activity
pllim commentedon Jun 10, 2016
Could I be seeing the same thing at #5051(comment)? c/c @taldcroft , @mhvk
bmorris3 commentedon Jun 10, 2016
That's possible, @pllim. The debugging discussion happening in that thread is similar to the one I was having in my own head a few minutes ago.
mhvk commentedon Jun 10, 2016
The URL seems to work again (just pasted it in my browser). The fact that you are getting "mysterious IERS warnings" does suggest though that the IERS code may need to handle the timeouts more graciously, or have warnings that make clearer what is happening.
pllim commentedon Jun 10, 2016
For unreliable test, perhaps mark it as
xfailunder certain conditions? In this case, it would be a timeout error orHTTPError.taldcroft commentedon Jun 10, 2016
Well I certainly hope we have not erred in automatically using the IERS web site.
However, please do note that it is easy to opt out of this behavior and return to handling IERS-related transforms manually. See http://astropy.readthedocs.io/en/latest/utils/iers.html. In particular the configuration parameters section and subsequent sections talk about how to disable auto-download or do things manually.
taldcroft commentedon Jun 10, 2016
It would be technically feasible to try a backup website if the primary fails. I don't know enough about their redundancy to know if that is actually valuable or if it would just mean taking twice as long to fail.
astrofrog commentedon Jun 10, 2016
We could always use data.astropy.org as a mirror?
taldcroft commentedon Jun 10, 2016
Can we set a cron job to keep data.astropy.org up to date?
astrofrog commentedon Jun 10, 2016
I'm sure we could - I guess the question is just on which server to set up the cron job.
eteq commentedon Jun 10, 2016
@bmorris3 - can you clarify where in the IERS web site it points you to there? All the links I saw there (in admittedly a cursory glance) are at
datacenter.iers.org... which @mhvk investigated and got an official answer from the IERS folks that it is a mirror that of themaiacopy. I wonder if that should be the fallback, rather thantoshi?taldcroft commentedon Jun 10, 2016
@eteq et al. - do you think adding support for fallback sites is release-critical? I think I know how to do it, but it's a question of whether we (1) should do this, and (2) if it is in 1.2.
bmorris3 commentedon Jun 10, 2016
@eteq – the IERS webpage directs you to this webpage at the
datacenter.iers.orgaddress, which provides these URLS as the "Source:"mhvk commentedon Jun 10, 2016
I think the
datacenter.iers.orgwould be a good enough backup. I don't think it makes much sense to add yet another copy atdata.astropy.org.That said, most important would seem to be to have a sufficiently informative error message; I'm fine with just sticking to one server.
astrofrog commentedon Jun 10, 2016
@taldcroft - I think a fallback mechanism and a better error if both fail would make sense. If you think you can do this in the next few hours, I can delay the release candidate - I would really like to get the RC out today or tomorrow morning at the very latest.
taldcroft commentedon Jun 11, 2016
@astrofrog - I agree with @mhvk that this can wait. I think that the existing warning would be sufficient for programmatic use, it's just that emitted warnings don't show up in testing. And the situation that you encountered today is really just a problem in testing. It failed because the real IERS-A file was already loaded and it didn't pick up the stubbed version for testing.
pllim commentedon May 1, 2018
@bmorris3 is this still an issue? The last update was in 2016, so just want to make sure.
bmorris3 commentedon May 1, 2018
@pllim It is, see all of the references to related issues above.
bsipocz commentedon May 1, 2018
However it's a duplicate with #5194. I would suggest to close one of them, maybe the other issue has more concrete suggestion of how to work around, but I really have no preferences which one to close.
pllim commentedon Dec 20, 2018
Does #8308 address this as well?
mhvk commentedon Dec 21, 2018
Agreed with @pllim that #8308 addresses at least the basic point here, so closing. Probably best to open a new issue if there are still problems.