Repository navigation
TST: Consider using a local copy of sites.json for testing #10298
Description
Description
Several coordinates tests now chronically fail. They pull sites.json from data server and are marked as "remote data" but the server is now denying access. I don't know if they would work again once the CI traffic decreases or not; i.e., if the blacklist is temporary or permanent. Because of these failures, maintainers might miss real "remote data" failures that need to be fixed (e.g., #10297 and #10299).
What are these tests really testing? Do they really need to be on the whim of internet? Can we use a local copy of sites.json and not mark them as "remote data"?
___________________________ test_earthlocation_site ____________________________
cls = <class 'astropy.coordinates.earth.EarthLocation'>
site_name = 'Keck Observatory'
@classmethod
def of_site(cls, site_name):
"""
Return an object of this class for a known observatory/site by name.
This is intended as a quick convenience function to get basic site
information, not a fully-featured exhaustive registry of observatories
and all their properties.
Additional information about the site is stored in the ``.info.meta``
dictionary of sites obtained using this method (see the examples below).
.. note::
When this function is called, it will attempt to download site
information from the astropy data server. If you would like a site
to be added, issue a pull request to the
`astropy-data repository <https://github.com/astropy/astropy-data>`_ .
If a site cannot be found in the registry (i.e., an internet
connection is not available), it will fall back on a built-in list,
In the future, this bundled list might include a version-controlled
list of canonical observatories extracted from the online version,
but it currently only contains the Greenwich Royal Observatory as an
example case.
Parameters
----------
site_name : str
Name of the observatory (case-insensitive).
Returns
-------
site : This class (a `~astropy.coordinates.EarthLocation` or subclass)
The location of the observatory.
Examples
--------
>>> from astropy.coordinates import EarthLocation
>>> keck = EarthLocation.of_site('Keck Observatory') # doctest: +REMOTE_DATA
>>> keck.geodetic # doctest: +REMOTE_DATA +FLOAT_CMP
GeodeticLocation(lon=<Longitude -155.47833333 deg>, lat=<Latitude 19.82833333 deg>, height=<Quantity 4160. m>)
>>> keck.info.meta # doctest: +REMOTE_DATA
{'source': 'IRAF Observatory Database', 'timezone': 'US/Aleutian'}
See Also
--------
get_site_names : the list of sites that this function can access
"""
registry = cls._get_site_registry()
try:
> el = registry[site_name]
../../.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py:353:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <astropy.coordinates.sites.SiteRegistry object at 0x7f24892be320>
site_name = 'Keck Observatory'
def __getitem__(self, site_name):
"""
Returns an EarthLocation for a known site in this registry.
Parameters
----------
site_name : str
Name of the observatory (case-insensitive).
Returns
-------
site : `~astropy.coordinates.EarthLocation`
The location of the observatory.
"""
if site_name.lower() not in self._lowercase_names_to_locations:
# If site name not found, find close matches and suggest them in error
close_names = get_close_matches(site_name, self._lowercase_names_to_locations)
close_names = sorted(close_names, key=len)
> raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
E astropy.coordinates.errors.UnknownSiteException: "Site 'Keck Observatory' not in database. Use the 'names' attribute to see available sites."
../../.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py:58: UnknownSiteException
During handling of the above exception, another exception occurred:
tmpdir = local('/tmp/pytest-of-travis/pytest-0/test_earthlocation_site0')
@pytest.mark.remote_data
def test_earthlocation_site(tmpdir):
> keck = EarthLocation.of_site('Keck Observatory')
../../.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_earthlocation.py:55:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'astropy.coordinates.earth.EarthLocation'>
site_name = 'Keck Observatory'
@classmethod
def of_site(cls, site_name):
"""
Return an object of this class for a known observatory/site by name.
This is intended as a quick convenience function to get basic site
information, not a fully-featured exhaustive registry of observatories
and all their properties.
Additional information about the site is stored in the ``.info.meta``
dictionary of sites obtained using this method (see the examples below).
.. note::
When this function is called, it will attempt to download site
information from the astropy data server. If you would like a site
to be added, issue a pull request to the
`astropy-data repository <https://github.com/astropy/astropy-data>`_ .
If a site cannot be found in the registry (i.e., an internet
connection is not available), it will fall back on a built-in list,
In the future, this bundled list might include a version-controlled
list of canonical observatories extracted from the online version,
but it currently only contains the Greenwich Royal Observatory as an
example case.
Parameters
----------
site_name : str
Name of the observatory (case-insensitive).
Returns
-------
site : This class (a `~astropy.coordinates.EarthLocation` or subclass)
The location of the observatory.
Examples
--------
>>> from astropy.coordinates import EarthLocation
>>> keck = EarthLocation.of_site('Keck Observatory') # doctest: +REMOTE_DATA
>>> keck.geodetic # doctest: +REMOTE_DATA +FLOAT_CMP
GeodeticLocation(lon=<Longitude -155.47833333 deg>, lat=<Latitude 19.82833333 deg>, height=<Quantity 4160. m>)
>>> keck.info.meta # doctest: +REMOTE_DATA
{'source': 'IRAF Observatory Database', 'timezone': 'US/Aleutian'}
See Also
--------
get_site_names : the list of sites that this function can access
"""
registry = cls._get_site_registry()
try:
el = registry[site_name]
except UnknownSiteException as e:
> raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
E astropy.coordinates.errors.UnknownSiteException: "Site 'Keck Observatory' not in database. Use EarthLocation.get_site_names to see available sites."
../../.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py:355: UnknownSiteException
_____________________ [doctest] docs/coordinates/index.rst _____________________
317
318 ..
319 EXAMPLE END
320
321 For sites (primarily observatories) on the Earth, `astropy.coordinates` provides
322 a quick way to get an `~astropy.coordinates.EarthLocation` - the
323 `~astropy.coordinates.EarthLocation.of_site` method::
324
325 >>> from astropy.coordinates import EarthLocation
326 >>> EarthLocation.of_site('Apache Point Observatory') # doctest: +REMOTE_DATA +FLOAT_CMP
UNEXPECTED EXCEPTION: UnknownSiteException("Site 'Apache Point Observatory' not in database. Use EarthLocation.get_site_names to see available sites.")
Traceback (most recent call last):
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 353, in of_site
el = registry[site_name]
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py", line 58, in __getitem__
raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'Apache Point Observatory' not in database. Use the 'names' attribute to see available sites."
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/3.7.1/lib/python3.7/doctest.py", line 1329, in __run
compileflags, 1), test.globs)
File "<doctest index.rst[54]>", line 1, in <module>
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 355, in of_site
raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'Apache Point Observatory' not in database. Use EarthLocation.get_site_names to see available sites."
/home/travis/build/astropy/astropy/docs/coordinates/index.rst:326: UnexpectedException
________________ [doctest] docs/coordinates/remote_methods.rst _________________
015 >>> SkyCoord.from_name("PSR J1012+5307") # doctest: +REMOTE_DATA +FLOAT_CMP
016 <SkyCoord (ICRS): (ra, dec) in deg
017 ( 153.1393271, 53.117343)>
018
019 The second is the :class:`~astropy.coordinates.EarthLocation` :meth:`~astropy.coordinates.EarthLocation.of_site` method, which
020 provides a similar quick way to get an
021 :class:`~astropy.coordinates.EarthLocation` from an observatory name::
022
023 >>> from astropy.coordinates import EarthLocation
024 >>> EarthLocation.of_site('Apache Point Observatory') # doctest: +REMOTE_DATA +FLOAT_CMP
UNEXPECTED EXCEPTION: UnknownSiteException("Site 'Apache Point Observatory' not in database. Use EarthLocation.get_site_names to see available sites.")
Traceback (most recent call last):
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 353, in of_site
el = registry[site_name]
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py", line 58, in __getitem__
raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'Apache Point Observatory' not in database. Use the 'names' attribute to see available sites."
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/3.7.1/lib/python3.7/doctest.py", line 1329, in __run
compileflags, 1), test.globs)
File "<doctest remote_methods.rst[3]>", line 1, in <module>
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 355, in of_site
raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'Apache Point Observatory' not in database. Use EarthLocation.get_site_names to see available sites."
/home/travis/build/astropy/astropy/docs/coordinates/remote_methods.rst:24: UnexpectedException
__________________ [doctest] docs/coordinates/satellites.rst ___________________
090 >>> location = itrs.earth_location # doctest: +REMOTE_DATA
091 >>> location.geodetic # doctest: +REMOTE_DATA +FLOAT_CMP
092 GeodeticLocation(lon=<Longitude 160.34199789 deg>, lat=<Latitude -24.6609379 deg>, height=<Quantity 420.17927591 km>)
093
094 Or, if you want to find the altitude and azimuth of the satellite from a particular location::
095
096 .. doctest-requires:: sgp4
097
098 >>> from astropy.coordinates import EarthLocation, AltAz
099 >>> siding_spring = EarthLocation.of_site('aao') # doctest: +REMOTE_DATA
UNEXPECTED EXCEPTION: UnknownSiteException("Site 'aao' not in database. Use EarthLocation.get_site_names to see available sites.")
Traceback (most recent call last):
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 353, in of_site
el = registry[site_name]
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py", line 58, in __getitem__
raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'aao' not in database. Use the 'names' attribute to see available sites."
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/3.7.1/lib/python3.7/doctest.py", line 1329, in __run
compileflags, 1), test.globs)
File "<doctest satellites.rst[19]>", line 1, in <module>
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 355, in of_site
raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'aao' not in database. Use EarthLocation.get_site_names to see available sites."
/home/travis/build/astropy/astropy/docs/coordinates/satellites.rst:99: UnexpectedException
_________________ [doctest] docs/coordinates/spectralcoord.rst _________________
212 To use the more advanced funtionality in |SpectralCoord|, including the ability
213 to easily transform between different well-defined velocity frames, you will
214 need to give it information about the location (and optionally velocity) of
215 the observer and target. This is done by passing either coordinate frame objects
216 or |SkyCoord| objects. To take a concrete example, let's assume that we are now
217 observe the source T Tau using the ALMA telescope. To create an observer object
218 corresponding to this, we can make use of the |EarthLocation| class::
219
220 >>> from astropy.coordinates import EarthLocation
221 >>> location = EarthLocation.of_site('ALMA') # doctest: +REMOTE_DATA
UNEXPECTED EXCEPTION: UnknownSiteException("Site 'ALMA' not in database. Use EarthLocation.get_site_names to see available sites.")
Traceback (most recent call last):
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 353, in of_site
el = registry[site_name]
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py", line 58, in __getitem__
raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'ALMA' not in database. Use the 'names' attribute to see available sites."
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/3.7.1/lib/python3.7/doctest.py", line 1329, in __run
compileflags, 1), test.globs)
File "<doctest spectralcoord.rst[25]>", line 1, in <module>
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 355, in of_site
raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'ALMA' not in database. Use EarthLocation.get_site_names to see available sites."
/home/travis/build/astropy/astropy/docs/coordinates/spectralcoord.rst:221: UnexpectedException
_______________________ [doctest] docs/whatsnew/4.1.rst ________________________
063 >>> sc.to(u.eV)
064 <SpectralCoord [1.89520328, 1.89462406, 1.89404519] eV>
065 >>> sc.to(u.THz)
066 <SpectralCoord [458.25811373, 458.11805929, 457.97809044] THz>
067
068 It can also hold information about the observer and target::
069
070 >>> from astropy.time import Time
071 >>> from astropy.coordinates import EarthLocation
072 >>> location = EarthLocation.of_site('ALMA') # doctest: +REMOTE_DATA
UNEXPECTED EXCEPTION: UnknownSiteException("Site 'ALMA' not in database. Use EarthLocation.get_site_names to see available sites.")
Traceback (most recent call last):
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 353, in of_site
el = registry[site_name]
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/sites.py", line 58, in __getitem__
raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'ALMA' not in database. Use the 'names' attribute to see available sites."
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/3.7.1/lib/python3.7/doctest.py", line 1329, in __run
compileflags, 1), test.globs)
File "<doctest 4.1.rst[10]>", line 1, in <module>
File "/home/travis/build/astropy/astropy/.tox/py37-test-devdeps/lib/python3.7/site-packages/astropy/coordinates/earth.py", line 355, in of_site
raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
astropy.coordinates.errors.UnknownSiteException: "Site 'ALMA' not in database. Use EarthLocation.get_site_names to see available sites."
/home/travis/build/astropy/astropy/docs/whatsnew/4.1.rst:72: UnexpectedException
Activity
bsipocz commentedon May 4, 2020
We have one location stored in the local sites.json file, and with the notion of astropy shouldn't be the maintainer of an observatory location database, we shouldn't really have any of these contents in it. Maybe rework the docstring example to have Greenwich in it rather than Keck, and the narrative docs then can say a bit more that the sites info lives in a different location so
EarthLocation.of_site()relies on this file being available locally, but then the tests could remainremote_data.An ugly workaround on the other hand is to use a
testsetupdirective, and define the location for the sites used in the example, hidden from the docs page. The tests would then pick that up, but users would still need the sites.json from the internet or their cache.mhvk commentedon May 4, 2020
The other solution might be to use @aarchiba's nice work that allows pre-populating the cache from an archive (which could be on the astropy data server perhaps) - we could do this for the whole travis test run.
I do think @pllim is right that most of these tests should not rely on the server working (though probably good to have at least one left just in case it disappears completely!).
bsipocz commentedon May 4, 2020
it's already lives on the data server, so having these blacklist issues is not a good sign.
mhvk commentedon May 4, 2020
Oops...
mhvk commentedon May 4, 2020
And here I was about to suggest to do this with not just the
sites.jsondata...bsipocz commentedon May 4, 2020
Using an inter-build Travis cache on the other hand may work, though I don't know whether it's possible. I suppose all blacklist would go away once we only trigger the server only once a day
mhvk commentedon May 4, 2020
That was part of the motivation for @aarchiba - e.g., in PINT, it can help to avoid JPL ephemerides multiple times.
pllim commentedon May 4, 2020
I really don't think it needs to hammer the data server so much, cache or no cache. I might have a cunning plan based on the ideas above... 💭