Feature Request: Functional of_site for offline use

@stargaser

In sites.py, get_downloaded_sites() accepts a jsonurl for offline access (default is online). It is called by _get_site_registry() in earth.py with no way to change the default. _get_site_registry() in turn is called by of_site() also in earth.py, also with no way to change the default.

Since observatories don't move around every day, it is not unreasonable to make it easier for user to pass in jsonurl to of_site() if they wish to avoid network connection.

This was a follow-up for Scipy 2018. cc @stargaser , @eteq , and @adrn

@pllim

I think I get this (and I think I agree), but can I just confirm: it would be sufficient to have a configuration item that essentially lets you set the default that _get_site_registry goes to, right, @pllim? That would let you do a local file and call it all-good?

@stargaser

Yes. Configuration item works too, as long as that is acceptable for @stargaser 's use case.

Could it not just fall back to an internal/downloaded-earlier file as we do for IERS?

@stargaser

That really depends on how @stargaser intends to use this. Hopefully he can chime in.

Since I was just looking at this: LSST would like to be able to "download and cache" this data and have it transparently usable. There's already some caching code in utils (e.g. download_file()), so I'm curious why we can't use that?

@parejkoj

@parejkoj , just to be clear, how are you using it currently? In what way that the cache is not working for you?

>>> from astropy.coordinates import EarthLocation                                               
>>> from astropy.utils import data
>>> keck = EarthLocation.of_site('Keck Observatory')                                            
Downloading http://data.astropy.org/coordinates/sites.json
|==============================================================|  21k/ 21k (100.00%)         0s
>>> keck = EarthLocation.of_site('Keck Observatory')  # No download
>>> data.get_cached_urls()                                                                      
['http://data.astropy.org/coordinates/sites.json']

If you quit your python session, it has not saved the file, so it has to download it again. I thought the cache was for between-session caching, not within-session.

@parejkoj

@parejkoj , I still don't see it. I quit my session and did this in a new session (and used a new terminal for good measure):

In [1]: from astropy.utils import data                                                              

In [2]: data.get_cached_urls()                                                                      
Out[2]: ['http://data.astropy.org/coordinates/sites.json']

In [3]: from astropy.coordinates import EarthLocation                                               

In [4]: keck = EarthLocation.of_site('Keck Observatory')  # No download

In [5]:

What do you see on your end? Are you trying with Astropy 3.1RC1?

Huh. I see the same thing. I'm on 3.0.3. I was previously trying it on my mac laptop, and now I'm on my Linux desktop. Let me try the laptop again later today. The behavior I saw there was that if I turned off wifi, of_site would fail in the second python session.

Ok, there's definitely a bug somewhere. This is with an up-to-date astropy master, 3.2.dev23397, with my laptop's wifi turned off:

In [1]: from astropy.utils import data

In [2]: data.get_cached_urls()
   ...:
Out[2]:
['http://maia.usno.navy.mil/ser7/finals2000A.all',
 'http://data.astropy.org/coordinates/sites.json']

In [3]: from astropy.coordinates import EarthLocation

In [4]: EarthLocation.of_site('Keck Observatory')
WARNING: Could not access the online site list. Falling back on the built-in version, which is rather limited. If you want to retry the download, do EarthLocation._get_site_registry(force_download=True) [astropy.coordinates.earth]
---------------------------------------------------------------------------
UnknownSiteException                      Traceback (most recent call last)
~/lsst/astropy/astropy/coordinates/earth.py in of_site(cls, site_name)
    355         try:
--> 356             el = registry[site_name]
    357         except UnknownSiteException as e:

~/lsst/astropy/astropy/coordinates/sites.py in __getitem__(self, site_name)
     57
---> 58             raise UnknownSiteException(site_name, "the 'names' attribute", close_names=close_names)
     59

UnknownSiteException: "Site 'Keck Observatory' not in database. Use the 'names' attribute to see available sites."

During handling of the above exception, another exception occurred:

UnknownSiteException                      Traceback (most recent call last)
<ipython-input-4-490eb7f3efec> in <module>
----> 1 EarthLocation.of_site('Keck Observatory')

~/lsst/astropy/astropy/coordinates/earth.py in of_site(cls, site_name)
    356             el = registry[site_name]
    357         except UnknownSiteException as e:
--> 358             raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
    359
    360         if cls is el.__class__:

UnknownSiteException: "Site 'Keck Observatory' not in database. Use EarthLocation.get_site_names to see available sites."

@parejkoj

Thanks, @parejkoj . I am able to reproduce and have concluded that it is indeed a bug, introduced by #6987 (my bad). That PR inserted a block of code that access internet to cache mirror URL regardless of whether data is already cached or not. I'll think of a fix.

The issue of came up for ZTF when we ran lots of pipelines in parallel for ZTF. I listed it in my talk at SciPy 2018. Our workaround was to hard-code the location of Palomar.

It may well be that #8163 in conjunction with what LSST is doing, will address the need for ZTF.

@parejkoj

Just to follow-up with @parejkoj and @stargaser , is this still a problem at all for your use cases? If so, please state what else is missing so we can slate the work for 3.2. Otherwise, I guess we can close. Thanks.

I'm going to see if we can investigate whether the new changes work for our cases in LSST. Hopefully we can report back sometime next week.

With version 3.2.dev23801, I can turn off my laptop's Internet connection and the lookup of EarthLocation.of_site('palomar') picks up the cached version fine. This addresses the issue as far as I'm concerned.

@stargaser

Great to hear, @stargaser ! Just waiting for @parejkoj in case there is something else.

@parejkoj

This seems to be resolved so I'm adding the label, @parejkoj please confirm.

Please give me until the end of March. If you haven't heard back from me by then, close it.

Hopefully the bot will be resurrected soon, so it will close it in about a month unless we remove the label.

Hi humans 👋 - this issue was labeled as Close? approximately 7 days ago. If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I will close this issue in a month.

If you believe I commented on this issue incorrectly, please report this here

I have not gotten anyone else from LSST to help me explore this, and I don't know exactly what our use case is. #8163 fixed the specific issue I described above, so you might as well close this and we'll open a new issue once LSST has a better description of their needs.

Navigation Menu

Repository navigation

Feature Request: Functional of_site for offline use #7653

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature Request: Functional of_site for offline use #7653

Description

Activity

eteq commented on Jul 26, 2018

pllim commented on Jul 26, 2018

mhvk commented on Jul 27, 2018

pllim commented on Jul 28, 2018

parejkoj commented on Nov 18, 2018

pllim commented on Nov 19, 2018

parejkoj commented on Nov 19, 2018

pllim commented on Nov 19, 2018

parejkoj commented on Nov 19, 2018

parejkoj commented on Nov 20, 2018

pllim commented on Nov 20, 2018

stargaser commented on Dec 5, 2018

pllim commented on Feb 8, 2019

parejkoj commented on Feb 8, 2019

stargaser commented on Feb 8, 2019

pllim commented on Feb 11, 2019

bsipocz commented on Mar 18, 2019

parejkoj commented on Mar 21, 2019

bsipocz commented on Mar 21, 2019

astropy-bot commented on Mar 25, 2019

parejkoj commented on Mar 28, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions