Repository navigation
Cache for test files #8748
Description
Following discussions in #8727: the cache for test files is set to a temporary directory, but I think that it would be better to find a way to avoid downloading all the files every time, to reduce bandwidth usage. And these files are not supposed to change often, so it should not be a problem to cache them.
It would also be good to put the files in a specific directory, not in the user cache. And a fix is needed for the test_download_cache test which deletes the cache (which is not an issue currently since the cache directory is temporary). I have a fix for this in a branch.
Activity
pllim commentedon Oct 30, 2019
With #9182 merged, it is now possible to export the cache into a ZIP file, ship it into the CI system, and then import that file into the cache.
aarchiba commentedon Nov 12, 2019
At least within Travis, this is probably more easily addressed by setting the astropy cache to somewhere that Travis will cache: then it automatically updates as needed - for example if someone writes a test needing some additional file, the first run will pull it off the Net, then it will sit in the Travis cache after that. Travis handles pulling the cache from master into new PRs, which are allowed to change it and keep their own version after that.
Using a ZIP file does make sharing easy but (again, at least for Travis) there doesn't seem to be a really good way to get the ZIP file onto the CI machine - the only reasonable mechanism I can see is the Travis cache (git LFS has problems), at which point there's no point ZIPping and unZIPping.
mhvk commentedon Jun 3, 2021
Having just re-raised a similar issue (#11811, now closed as duplicate), how do we move forward? Some kind of config item seems best, with maybe the following options:
remote_data- or at least notremote_data=any).Not quite sure what the best default would be. Maybe the under .tox option?
mhvk commentedon Dec 7, 2021
And it seems I again forgot about this: it would be really good to avoid downloading the DE* ephemerides, also on one's own computer. (See now closed #12564.)
pllim commentedon Dec 16, 2021
At 2021-12-16 infrastructure tag-up, @saimn said we can probably use the same cache for all the jobs, though now that I think a little more, maybe we want different caches per remote data setting (due to differing cache sizes).
neutrinoceros commentedon Apr 27, 2026
Having worked on #19575 for two entire weeks, I must say this idea is dreadful to me right now; it was already very hard to fix all issues (especially in CI), and I cannot imagine how much harder it would have been if files were also cached in runners (especially if there was any kind of defect in invalidation mechanisms). That said, rewriting the entire system from scratch should really be a singular event and I don't expect these conditions to be met again (that's kind of the point of going back to the drawing board), so, maybe if the new system proves stable enough, we could revive this issue. An alternative approach would be to simply reduce the number of jobs that exercise network requests, which I think is worth considering.