HTTPError: Forbidden for download_IERS_A()

Run into this in #177. I've repushed the latest commit, but it seems to be more than just a fluke and failed again. ~~I hasn't managed to reproduce it locally.~~

A failing build, e.g. https://travis-ci.org/astropy/astroplan/jobs/147020338

WARNING: failed to download http://maia.usno.navy.mil/ser7/finals2000A.all, using local IERS-B: HTTP Error 403: Forbidden [astropy.utils.iers.iers]

/home/travis/miniconda/envs/test/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.

  warnings.warn(self.msg_depr % (key, alt_key))

WARNING: Tried to get polar motions for times after IERS data is valid. Defaulting to polar motion from the 50-yr mean for those. This may affect precision at the 10s of arcsec level [astropy.coordinates.builtin_frames.utils]

WARNING: (some) times are outside of range covered by IERS table. Assuming UT1-UTC=0 for coordinate transformations. [astropy.coordinates.builtin_frames.utils]

WARNING: OldEarthOrientationDataWarning: For best precision (on the order of arcseconds), you must download an up-to-date IERS Bulletin A table. To do so, run:

>>> from astroplan import download_IERS_A

>>> download_IERS_A()

 [astroplan.utils]

WARNING: failed to download http://maia.usno.navy.mil/ser7/finals2000A.all, using local IERS-B: HTTP Error 403: Forbidden [astropy.utils.iers.iers]

WARNING: OldEarthOrientationDataWarning: For best precision (on the order of arcseconds), you must download an up-to-date IERS Bulletin A table. To do so, run:

>>> from astroplan import download_IERS_A

>>> download_IERS_A()

 [astroplan.utils]

/home/travis/build/astropy/astroplan/docs/faq/iers.rst:75: WARNING: Exception occurred in plotting iers-1

 from /home/travis/build/astropy/astroplan/docs/faq/iers.rst:

Traceback (most recent call last):

  File "/home/travis/miniconda/envs/test/lib/python3.5/site-packages/matplotlib/sphinxext/plot_directive.py", line 517, in run_code

    six.exec_(code, ns)

  File "<string>", line 11, in <module>

  File "/home/travis/miniconda/envs/test/lib/python3.5/site-packages/astropy/utils/data.py", line 1097, in download_file

    raise e

  File "/home/travis/miniconda/envs/test/lib/python3.5/site-packages/astropy/utils/data.py", line 1030, in download_file

    remote_url, timeout=timeout)) as remote:

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 163, in urlopen

    return opener.open(url, data, timeout)

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 472, in open

    response = meth(req, response)

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 582, in http_response

    'http', request, response, code, msg, hdrs)

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 510, in error

    return self._call_chain(*args)

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 444, in _call_chain

    result = func(*args)

  File "/home/travis/miniconda/envs/test/lib/python3.5/urllib/request.py", line 590, in http_error_default

    raise HTTPError(req.full_url, code, msg, hdrs, fp)

urllib.error.HTTPError: HTTP Error 403: Forbidden

Actually this fails locally, too:

>>> from astroplan import download_IERS_A
>>> download_IERS_A()

This is upstream, http://maia.usno.navy.mil/ throws a big Forbidden 403, too.

Has astroplan considered using the new IERS auto-download capability in astropy 1.2? I'm not sure if it would help with this test issue (when starting from a clean cache), but in general for users it should mostly make < few day downtimes of the upstream IERS site(s) invisible by means of the automated local caching.

I've changed it at one place but as I see a few others are still around. Thanks for the heads up.

@taldcroft

@taldcroft We intended to update astroplan to require >=1.2, since we rely on the new get_moon and and observatory database features. @bsipocz – did your PR which updated the astropy version to stable effectively update us to require 1.2 in the tests?

It updated to "stable", and ci-helpers should notice and update if for some reasons older than 1.2 is installed for "stable". So I think we should be fine.

@bsipocz

@bsipocz I had similar problems and started a discussion on this problem in astropy/astropy#5059 also.

@bmorris3

@bmorris3 - Thanks, I missed that one.

This is upstream, http://maia.usno.navy.mil/ throws a big Forbidden 403, too.

This is happening again.

Can confirm ☝️

>>> import astroplan
>>> astroplan.download_IERS_A()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/astroplan/utils.py", line 127, in download_IERS_A
    show_progress=show_progress)
  File "/usr/lib64/python3.6/site-packages/astropy/utils/data.py", line 1126, in download_file
    raise e
  File "/usr/lib64/python3.6/site-packages/astropy/utils/data.py", line 1059, in download_file
    remote_url, timeout=timeout)) as remote:
  File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Here's how I fixed it, for anyone else looking for the fix:

>>> import astropy
>>> astropy.utils.iers.IERS_A_URL = 'http://toshi.nofs.navy.mil/ser7/finals2000A.all'
>>> astroplan.download_IERS_A()
Downloading http://toshi.nofs.navy.mil/ser7/finals2000A.all
|=========================================| 3.2M/3.2M (100.00%)         4s

Having the download_IERS_A function try the backup if the original fails seems to be the obvious solution.

Any chance you can see if astropy/astropy#8308 helps fix this over here?

EDIT: Hmm maybe not (see code snippet below). 👍 to suggestion at #192 (comment)

astroplan/astroplan/utils.py

Line 125 in 6bf2cc7

local_iers_a_path = download_file(iers.IERS_A_URL, cache=True,

...

Here's how I fixed it, for anyone else looking for the fix:

>>> import astropy
>>> astropy.utils.iers.IERS_A_URL = 'http://toshi.nofs.navy.mil/ser7/finals2000A.all'
>>> astroplan.download_IERS_A()
Downloading http://toshi.nofs.navy.mil/ser7/finals2000A.all
|=========================================| 3.2M/3.2M (100.00%)         4s

Having the download_IERS_A function try the backup if the original fails seems to be the obvious solution.

Hmmm. I'm up at Palomar Observatory at the moment and am having trouble accessing either site. But toshi is particularly problematic. Here is an excerpt of the response:

<h1> WEBSITE BLOCKED </h1>
<p>You have attempted to access a blocked website. Access to this website has been blocked for operational reasons by the DOD Enterprise-Level Protection System.</p>
<p> Category: <b> JFHQ-DoDIN_OGC_Deny </b></p>
<p> URL: <b> toshi.nofs.navy.mil/ser7/finals2000A.all </b></p>
<p>Contact your local Network Control Center for information on how to gain access to MISSION ESSENTIAL or otherwise authorized websites, or to report a mis-categorized website.</p>
<p>Use the following links for additional guidance and procedures for
requesting access to blocked content:</p>
<p>-Army users click <a href="https://esd-crm.csd.disa.mil/app/answers/detail/a_id/1628">here</a>.</p>
<p>-Navy users click <a href="https://usff.navy.deps.mil/sites/netwarcom/N3/Pages/UnBlock.aspx">here</a>.</p>
<p>-Air Force users click <a href="https://cs2.eis.af.mil/sites/10060/Wiki/AF%20PPS.aspx">here</a>.</p>
<p>-DHA users click <a href="https://kbs.nsoc.health.mil/Documents/IP%20or%20Domain%20Block%20-%20Unblock%20Request/Unblock%20Request%20Form.pdf">here</a>.</p>

Update your astropy to 3.2.3, both usno servers gone offline for the foreseeable future.

Update your astropy to 3.2.3, both usno servers gone offline for the foreseeable future.

Thanks for the quick response! 3.1.2 was in conda but updating just now got me 3.2.3 and it works! Sorry for the noise; I'd looked for a solution but didn't stumble across fixes in the newest version.

No worries, this was an unexpected situation from about a month ago.

I'm also seeing this now. I get errors using both the default server and the one specified above.

Here's what I get using the URL specified above:

In [5]: astropy.__version__
Out[5]: '3.2.3'

In [6]: astroplan.__version__
Out[6]: '0.5'

In [7]: astropy.utils.iers.IERS_A_URL = 'http://toshi.nofs.navy.mil/ser7/finals2000A.all'

In [8]: astroplan.download_IERS_A()
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-8-185a77c8f66e> in <module>
----> 1 astroplan.download_IERS_A()

~/anaconda/lib/python3.6/site-packages/astroplan/utils.py in download_IERS_A(show_progress)
    124
    125     local_iers_a_path = download_file(iers.IERS_A_URL, cache=True,
--> 126                                       show_progress=show_progress)
    127     # Undo monkey patch set up by get_IERS_A_or_workaround
    128     iers.IERS.iers_table = iers.IERS_A.open(local_iers_a_path)

~/anaconda/lib/python3.6/site-packages/astropy/utils/data.py in download_file(remote_url, cache, show_progress, timeout)
   1103             e.reason.strerror = e.reason.strerror + '. requested URL: ' + remote_url
   1104             e.reason.args = (e.reason.errno, e.reason.strerror)
-> 1105         raise e
   1106     except socket.timeout as e:
   1107         # this isn't supposed to happen, but occasionally a socket.timeout gets

~/anaconda/lib/python3.6/site-packages/astropy/utils/data.py in download_file(remote_url, cache, show_progress, timeout)
   1022                                 return url2hash[url_mirror]
   1023
-> 1024         with urllib.request.urlopen(remote_url, timeout=timeout) as remote:
   1025             # keep a hash to rename the local file to the hashed name
   1026             hash = hashlib.md5()

~/anaconda/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    221     else:
    222         opener = _opener
--> 223     return opener.open(url, data, timeout)
    224
    225 def install_opener(opener):

~/anaconda/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
    530         for processor in self.process_response.get(protocol, []):
    531             meth = getattr(processor, meth_name)
--> 532             response = meth(req, response)
    533
    534         return response

~/anaconda/lib/python3.6/urllib/request.py in http_response(self, request, response)
    640         if not (200 <= code < 300):
    641             response = self.parent.error(
--> 642                 'http', request, response, code, msg, hdrs)
    643
    644         return response

~/anaconda/lib/python3.6/urllib/request.py in error(self, proto, *args)
    568         if http_err:
    569             args = (dict, 'default', 'http_error_default') + orig_args
--> 570             return self._call_chain(*args)
    571
    572 # XXX probably also want an abstract factory that knows when it makes

~/anaconda/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    502         for handler in handlers:
    503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
    505             if result is not None:
    506                 return result

~/anaconda/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    648 class HTTPDefaultErrorHandler(BaseHandler):
    649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    651
    652 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 503: Service Unavailable

I get anHTTP Error 403: Forbidden if I don't change the default URL.

the navy server is also down, try the mirror one instead of the default, that should work.

(alternatively you can give astropy v4.0.rc1 a try, there was some more reshuffling of the machinery in there).

The v4 IERS_A_URL works !
However, I don't undestand why IERS_A_URL_MIRROR (https) is reachable with a browser but urllib.request.urlopen fails ...

@ppmim

@ppmim they block crawlers basically. You need to set a user agent header: https://stackoverflow.com/a/24226797/3838691

Sometimes, using requests works though.

Navigation Menu

Repository navigation

HTTPError: Forbidden for download_IERS_A() #192

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HTTPError: Forbidden for download_IERS_A() #192

Description

Activity

bsipocz commented on Jul 24, 2016

bsipocz commented on Jul 24, 2016

taldcroft commented on Jul 25, 2016

bsipocz commented on Jul 25, 2016

bmorris3 commented on Jul 27, 2016

bsipocz commented on Jul 27, 2016

bmorris3 commented on Jul 29, 2016

bsipocz commented on Jul 29, 2016

maxnoe commented on Dec 19, 2018

martinjohndyer commented on Dec 19, 2018

pllim commented on Dec 20, 2018

lockhart commented on Nov 15, 2019

bsipocz commented on Nov 15, 2019

lockhart commented on Nov 15, 2019

bsipocz commented on Nov 15, 2019

joshwalawender commented on Dec 3, 2019

bsipocz commented on Dec 3, 2019

ppmim commented on Dec 10, 2019

maxnoe commented on Dec 10, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions