get_cached_urls() return is empty in test_name_resolve_cache

Description

Both in 4.1rc1 and on master the above test fails:

> python3.8 -c "import astropy; astropy.test(package='coordinates', args='-vk test_name_resolve', remote_data=True, open_files=True)"                                       
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.3, pytest-5.3.0, py-1.8.1, pluggy-0.13.0 -- /sw/bin/python3.8
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/derek/Downloads/.hypothesis/examples')

Running tests with Astropy version 4.1rc1.
Running tests in /sw/lib/python3.8/site-packages/astropy/coordinates.

Date: 2020-05-26T19:34:58

Platform: macOS-10.12.6-x86_64-i386-64bit

Executable: /sw/bin/python3.8

Full Python Version: 
3.8.3 (default, May 14 2020, 22:17:35) 
[Clang 9.0.0 (clang-900.0.39.2)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.19.0rc1
Scipy: 1.4.1
Matplotlib: 3.2.1
h5py: 2.9.0
Pandas: not available
Cython: 0.29.16
Scikit-image: not available
asdf: 2.6.0

Using Astropy options: remote_data: any.

Matplotlib: 3.2.1
Freetype: 2.6.1
PyQt5 5.12.2 -- Qt runtime 5.7.1 -- Qt compiled 5.7.1
rootdir: /sw/lib/python3.8/site-packages/astropy/coordinates
plugins: asdf-2.6.0, flaky-3.5.3, hypothesis-4.36.0, arraydiff-0.3, astropy-header-0.1.2, cov-2.6.1, doctestplus-0.5.0, expect-1.1.0, forked-1.1.3, mock-1.10.4, mpl-0.11, openfiles-0.3.2, qt-3.3.0, remotedata-0.3.1, xdist-1.30.0, requests-mock-1.7.0
collected 1338 items / 1333 deselected / 5 selected                                                                                                                                      

../../../sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_names PASSED                                                                          [ 20%]
../../../sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_name_resolve_cache FAILED                                                             [ 40%]
../../../sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_names_parse PASSED                                                                    [ 60%]
../../../sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_database_specify[NGC 3642-db_dict0] PASSED                                            [ 80%]
../../../sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_database_specify[castor-db_dict1] PASSED                                              [100%]

======================================================================================== FAILURES ========================================================================================
________________________________________________________________________________ test_name_resolve_cache _________________________________________________________________________________

tmpdir = local('/private/var/folders/7l/j6rbh6fw8xjc4w0059bv63_h00027w/T/pytest-of-derek/pytest-46/test_name_resolve_cache0')

    @pytest.mark.remote_data
    def test_name_resolve_cache(tmpdir):
        from astropy.utils.data import _get_download_cache_locs, get_cached_urls
        import shelve
    
        target_name = "castor"
    
        temp_cache_dir = str(tmpdir.mkdir('cache'))
        with paths.set_temp_cache(temp_cache_dir, delete=True):
            download_dir, urlmapfn = _get_download_cache_locs()
    
            with shelve.open(urlmapfn) as url2hash:
                assert len(url2hash) == 0
    
            icrs1 = get_icrs_coordinates(target_name, cache=True)
    
            # This is a weak test: we just check to see that a url is added to the
            #  cache!
            with shelve.open(urlmapfn) as url2hash:
                assert len(url2hash) == 1
>               url = get_cached_urls()[0]
E               IndexError: list index out of range

/sw/lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py:164: IndexError
==================================================================================== warnings summary ====================================================================================
/sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124
  /sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124: PytestCacheWarning: could not create cache path /sw/lib/python3.8/site-packages/astropy/coordinates/.pytest_cache/v/cache/stepwise
    self.warn("could not create cache path {path}", path=path)

/sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124
  /sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124: PytestCacheWarning: could not create cache path /sw/lib/python3.8/site-packages/astropy/coordinates/.pytest_cache/v/cache/nodeids
    self.warn("could not create cache path {path}", path=path)

/sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124
  /sw/lib/python3.8/site-packages/_pytest/cacheprovider.py:124: PytestCacheWarning: could not create cache path /sw/lib/python3.8/site-packages/astropy/coordinates/.pytest_cache/v/cache/lastfailed
    self.warn("could not create cache path {path}", path=path)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===Flaky Test Report===


===End Flaky Test Report===
=============================================================== 1 failed, 4 passed, 1333 deselected, 3 warnings in 13.73s ================================================================

System Details

Same for Python 3.6.10 and 3.7.7

Probably because of PytestCacheWarning: could not create cache path? Are you able to run this somewhere you have write access?

Probably because of PytestCacheWarning: could not create cache path? Are you able to run this somewhere you have write access?

The cache path in the warning is a system directory and not writable, but running from my user installation throws the same exception:

PYTHONPATH=~/lib/python3.8/site-packages python3.8 -c "import astropy; astropy.test(package='coordinates', args='-vk test_name_resolve_cache', remote_data=True, open_fil>
================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.8.3, pytest-5.3.0, py-1.8.1, pluggy-0.13.0 -- /sw/bin/python3.8
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/derek/Downloads/.hypothesis/examples')

Running tests with Astropy version 4.2.dev112+g595484597.
Running tests in lib/python3.8/site-packages/astropy/coordinates.

Date: 2020-05-26T20:08:33

Platform: macOS-10.12.6-x86_64-i386-64bit

Executable: /sw/bin/python3.8

Full Python Version: 
3.8.3 (default, May 14 2020, 22:17:35) 
[Clang 9.0.0 (clang-900.0.39.2)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.19.0rc1
Scipy: 1.4.1
Matplotlib: 3.2.1
h5py: 2.9.0
Pandas: not available
Cython: 0.29.16
Scikit-image: not available
asdf: 2.6.1.dev51+gcc88c58.d20200522

Using Astropy options: remote_data: any.

Matplotlib: 3.2.1
Freetype: 2.6.1
PyQt5 5.12.2 -- Qt runtime 5.7.1 -- Qt compiled 5.7.1
rootdir: /Users/derek
plugins: doctestplus-0.5.0, asdf-2.6.1.dev51+gcc88c58.d20200522, flaky-3.5.3, hypothesis-4.36.0, arraydiff-0.3, astropy-header-0.1.2, cov-2.6.1, expect-1.1.0, forked-1.1.3, mock-1.10.4, mpl-0.11, openfiles-0.3.2, qt-3.3.0, remotedata-0.3.1, xdist-1.30.0, requests-mock-1.7.0
collected 1338 items / 1337 deselected / 1 selected                                                                                                                                      

../lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py::test_name_resolve_cache FAILED                                                                      [100%]

======================================================================================== FAILURES ========================================================================================
________________________________________________________________________________ test_name_resolve_cache _________________________________________________________________________________

tmpdir = local('/private/var/folders/7l/j6rbh6fw8xjc4w0059bv63_h00027w/T/pytest-of-derek/pytest-52/test_name_resolve_cache0')

    @pytest.mark.remote_data
    def test_name_resolve_cache(tmpdir):
        from astropy.utils.data import _get_download_cache_locs, get_cached_urls
        import shelve
    
        target_name = "castor"
    
        temp_cache_dir = str(tmpdir.mkdir('cache'))
        with paths.set_temp_cache(temp_cache_dir, delete=True):
            download_dir, urlmapfn = _get_download_cache_locs()
    
            with shelve.open(urlmapfn) as url2hash:
                assert len(url2hash) == 0
    
            icrs1 = get_icrs_coordinates(target_name, cache=True)
    
            # This is a weak test: we just check to see that a url is added to the
            #  cache!
            with shelve.open(urlmapfn) as url2hash:
                assert len(url2hash) == 1
>               url = get_cached_urls()[0]
E               IndexError: list index out of range

../lib/python3.8/site-packages/astropy/coordinates/tests/test_name_resolve.py:164: IndexError
===Flaky Test Report===


===End Flaky Test Report===
=========================================================================== 1 failed, 1337 deselected in 2.64s ===========================================================================

That is weird. It passed assert len(url2hash) == 1 but fails at url = get_cached_urls()[0]. Are you able to reproduce it outside of testing environment?

Where is the tmpdir function set up? When using a manually created new cache dir, I get this:

>>> with paths.set_temp_cache(temp_cache_dir, delete=False):
...   download_dir, urlmapfn = _get_download_cache_locs()
...   with shelve.open(urlmapfn) as url2hash:
...     print(len(url2hash), url2hash)
...     print(get_cached_urls())
...   icrs1 = get_icrs_coordinates("castor", cache=True)
...   with shelve.open(urlmapfn) as url2hash:
...     print(len(url2hash), url2hash)
...     print(get_cached_urls())
... 
0 <shelve.DbfilenameShelf object at 0x10c593040>
[]
1 <shelve.DbfilenameShelf object at 0x10dc97c10>
[]

but after the icrs call cache/astropy/download/py3 contains a file in addition to urlmap containing the Vizier table; apparently get_cached_urls does not find this:

total 40
-rw-------  1 derek  stellar    811 26 May 21:16 4e4fe953e5afc74b579a9ed1cac93791
-rw-r-----  1 derek  stellar  16384 26 May 21:16 urlmap

I cannot reproduce that on Linux. But I did notice a behavior: For a given temp_cache_dir (doesn't have to be pytest dir, I use /home/user/zzz), if zzz is not created yet, it complains about FileNotFoundError: [Errno 2] No such file or directory: '/home/user/zzz/astropy', but if zzz is there but not zzz/astropy, then it creates the astropy subdir and succeeds.

import shelve
from astropy.coordinates.name_resolve import get_icrs_coordinates
from astropy.config import paths
from astropy.utils.data import _get_download_cache_locs, get_cached_urls
target_name = "castor"
temp_cache_dir = '/home/user/zzz'

with paths.set_temp_cache(temp_cache_dir, delete=False): 
    download_dir, urlmapfn = _get_download_cache_locs() 
    print(download_dir) 
    with shelve.open(urlmapfn) as url2hash: 
        print('Should be 0:', len(url2hash)) 
    icrs1 = get_icrs_coordinates(target_name, cache=True) 
    with shelve.open(urlmapfn) as url2hash: 
        print('Should be 1:', len(url2hash)) 
        print('What is cached:', get_cached_urls())

When it succeeds:

/home/user/zzz/astropy/download/py3
Should be 0: 0
Should be 1: 1
What is cached: ['http://cdsweb.u-strasbg.fr/cgi-bin/nph-sesame/A?castor']

Yes, the cache dir is expected to be created by the tmpdir.mkdir('cache'), all subdirectories are then created automatically, but delete=True was removing the base directory as well, therefore I changed it to delete=False to check the contents.
The url2hash from the context apparently has the correct url, but it is not communicated to get_cached_urls:

>>> with paths.set_temp_cache(temp_cache_dir, delete=False):
...   download_dir, urlmapfn = _get_download_cache_locs()
...   with shelve.open(urlmapfn) as url2hash:
...     print(len(url2hash), list(url2hash.keys()))
...   icrs1 = get_icrs_coordinates("castor", cache=True)
...   with shelve.open(urlmapfn) as url2hash:
...     print(len(url2hash), list(url2hash.keys()))
...     get_cached_urls()
... 
0 []
1 ['http://cdsweb.u-strasbg.fr/cgi-bin/nph-sesame/A?castor']
[]

Do you get any warnings at all? Looking at _cache implementation, it yields an empty dict with a warning if something goes wrong. That would explain why you are seeing an empty list, which is the output of listing keys for empty dict.

Also, I am starting to get curious if newer OSX sees this or it is just 10.12.

Yes, it looks like it is failing on getting the file lock; when changing the warning to an error I see in

astropy/astropy/utils/data.py

Line 1723 in 0a46d33

except dbm.error as e:

    raise WrongDBMModule(wrong_dbm_message + str(e))
astropy.utils.data.WrongDBMModule: Existing astropy cache is in an unsupported format, either install the appropriate package or use astropy.utils.data.clear_download_cache() to delete the whole cache; [Errno 35] Resource temporarily unavailable: '/var/folders/7l/j6rbh6fw8xjc4w0059bv63_h00027w/T/tmpi4atlz_1/cache/astropy/download/py3/urlmap'

@aarchiba

Let's see if @aarchiba knows of cache support in OSX 10.12. Unfortunately, I am not a Mac user.

It seems to work allright with the default ~/.astropy/cache directory, but using any paths.set_temp_cache() directory (even inside ~/.astropy) always results in the [Errno 35] Resource temporarily unavailable exception.

Are you able to turn that warning into exception (say, using the filtering provided by warnings) to see where exactly it is coming from?

Yes, it looks like it is failing on getting the file lock; when changing the warning to an error I see in

astropy/astropy/utils/data.py

Line 1723 in 0a46d33

except dbm.error as e:
    raise WrongDBMModule(wrong_dbm_message + str(e))
astropy.utils.data.WrongDBMModule: Existing astropy cache is in an unsupported format, either install the appropriate package or use astropy.utils.data.clear_download_cache() to delete the whole cache; [Errno 35] Resource temporarily unavailable: '/var/folders/7l/j6rbh6fw8xjc4w0059bv63_h00027w/T/tmpi4atlz_1/cache/astropy/download/py3/urlmap'

Sorry, just getting up to speed on this. This particular error means what it says - the cache exists, but the index is in a database format the current python does not understand. The most common situation is that the cache was created by a Python that supports gdbm but the current python does not (for example it is often not part of Anaconda).

Getting back to the original bug: do any of the tests in astropy.utils fail? There is a very extensive cache testing suite in there, and it would help to see whether any of those fail.

Because the test as written pokes around in the guts of the cache mechanism, it's possible there isn't actually a problem. But the test is checking that the downloaded data is stored in the cache, so if something about your configuration makes caching not work, download_file will just re-download on every request, but this test will fail. Now it does its testing with a cache set to a temporary directory, so I can't see how the cache could fail to be writable, so that shouldn't be a problem.

The test could be simplified - there isn't really any need to check what is in the cache, since the property being tested is really just that after the first download the file is available even if the internet is not.

Navigation Menu

Repository navigation

get_cached_urls() return is empty in test_name_resolve_cache #10401

Description

System Details

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

get_cached_urls() return is empty in test_name_resolve_cache #10401

Description

Description

System Details

Activity

pllim commented on May 26, 2020

dhomeier commented on May 26, 2020

pllim commented on May 26, 2020

dhomeier commented on May 26, 2020

pllim commented on May 26, 2020

dhomeier commented on May 26, 2020

pllim commented on May 26, 2020

pllim commented on May 26, 2020

dhomeier commented on May 26, 2020

pllim commented on May 26, 2020

dhomeier commented on May 26, 2020

pllim commented on May 27, 2020

aarchiba commented on Jun 2, 2020

aarchiba commented on Jun 2, 2020

aarchiba commented on Jun 2, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions