You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add way to have utils.data functions w/ cache have their cache expire #1162
Right now, the various functions in utils.data that have a cache option hold onto their cached copy of a download until the file is deleted. There are some use cases that require the cache expire after a certain time, and get re-downloaded (e.g. #1145).
So these functions (which all depend on download_file I believe), should allow cache to give a specific timeframe after which the file should be re-downloaded. This is pretty straightforward, but there is a question how best to define the expiration time. My instinct is to use stdlib datetime.datetime and datetime.timedelta objects, but an alternative might be astropy.time.Time and astropy.time.DeltaTime.
Using astropy.time.Time to define the expiration time on the tables used for Time... very meta! Hopefully you don't need leap seconds to define the expiration time? ;)
For the IERS tables considered for Time, one will only know after the file is downloaded up to when the file is good. Also, @taldcroft has suggested that refreshing only happen if the contents do not cover the date for which information is needed; if so, this may be less essential. On the other hand, for these files at least, they just get longer, so once a copy is there, (the equivalent of) wget -c will do the trick and will carry rather little overhead.
Activity
astrofrog commentedon Jun 6, 2013
👍 to this idea!
Using
astropy.time.Timeto define the expiration time on the tables used forTime... very meta! Hopefully you don't need leap seconds to define the expiration time? ;)mhvk commentedon Jun 6, 2013
For the IERS tables considered for Time, one will only know after the file is downloaded up to when the file is good. Also, @taldcroft has suggested that refreshing only happen if the contents do not cover the date for which information is needed; if so, this may be less essential. On the other hand, for these files at least, they just get longer, so once a copy is there, (the equivalent of)
wget -cwill do the trick and will carry rather little overhead.hamogu commentedon Jul 13, 2013
It might be worth noting that the Sunpy project has a GSoC student who implements a database for organize store and cache datafiles that are downloaded from archives.
http://www.sunpy.org/2013/06/19/google-summer-of-code-the-database-class/
So far, I am not quite sure if and how much that overlaps with this, but we might be able to just use that class here as well. It might be worthwhile to check which features are implemented by the end of the summer.
derdon commentedon Mar 15, 2014
Hi, this student is me :) The docs for the database can be found at http://sunpy.readthedocs.org/en/stable/guide/acquiring_data/database.html and perhaps the most interesting section is Caching. It's also easily possible to implement and use a custom cache type. This is not documented yet, though. Here you can find a unit test for a custom cache to see an example of how to do it: https://github.com/sunpy/sunpy/blob/master/sunpy/database/tests/test_caching.py#L15
You can also find me on #astropy or #sunpy if you want to discuss this idea further outside this thread.