Repository navigation
clear_download_cache() does not work in Python 2.7 #4427
Description
In Python 2.7, clearing the download cache runs into two problems, related to #4426. First:
In [1]: astro
1.1
In [2]: from astropy.utils import data
In [3]: !rm -rf /Users/aldcroft/.astropy/cache
In [4]: f = data.download_file('http://google.com', cache=True)
Downloading http://google.com [Done]
In [5]: data.clear_download_cache()
In [6]: f = data.download_file('http://google.com', cache=True)
In [7]: import os
In [8]: os.path.exists(f)
Out[8]: False
This is because the shelve file name without the .db extension is removed if it exists (which it doesn't, but that is not caught in the code). This relates to a fundamental issue with shelve on Python 2, which is that you don't know (and cannot determine in general) the name of the backing database file(s). This could be fixed by putting them into a sub-directory with a known name and then clobbering that.
Next, if I try harder (with the actual URL) it generates an exception:
In [9]: data.clear_download_cache('http://google.com')
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-9-1e289daec783> in <module>()
----> 1 data.clear_download_cache('http://google.com')
/Users/aldcroft/anaconda/lib/python2.7/site-packages/astropy/utils/data.pyc in clear_download_cache(hashorurl)
1268 filepath = url2hash[hash_key]
1269 del url2hash[hash_key]
-> 1270 os.unlink(filepath)
1271 else:
1272 msg = 'Could not find file or url {0}'
OSError: [Errno 2] No such file or directory: '/Users/aldcroft/.astropy/cache/download/42afb277772da0f1a943e37fd64ec317'
cc: @eteq
Activity
embray commentedon Jan 4, 2016
The subdirectory idea sounds viable.
taldcroft commentedon Jan 4, 2016
Yes, and maybe having something like
download_db/py2/anddownload_db/py3/directories would work for #4426. Unfortunately this means that files can get downloaded twice (as previously mentioned in discussions of using a different persistence method for the cache names / data).embray commentedon Jan 4, 2016
I think that for now the double download problem will only be an issue for a minority of users though. It's mostly just those of us doing heavy development that are using multiple Python versions on a regular basis, I imagine...