Repository navigation
Caching of files for testing with remote_data #11811
Description
Our test suite is very extensive, which also means it takes so long that running all tests is longer compatible with development. Fortunately, -P <something> really helps. But one thing that still is very painful is having to run remote-data tests, especially those that download large files, like the solar system ephemeris. Would it be possible to have a system where the download cache is shared between pytest invocations? Or perhaps is this possible already?
Activity
pllim commentedon Jun 3, 2021
I think we can use https://github.com/actions/cache but will need some thoughts on how to set it up properly for so many jobs.
pllim commentedon Jun 3, 2021
p.s. Will also have to consider the security risk.
mhvk commentedon Jun 3, 2021
I was actually wondering more about how to do it on my own machine, e.g., if I'm running
pytest -P coordinates --remote-data=anymultiple times.pllim commentedon Jun 3, 2021
Oh... heh heh. Locally, if the tests use
download_file(..., cache=True), I think it still goes into~/.astropy/cache. If it usestmpdir, you can overwrite with https://docs.pytest.org/en/6.2.x/tmpdir.html#the-default-base-temporary-directory , though I am not sure if that solves all the caching woes (also opens a can of worms because we never use that option).saimn commentedon Jun 3, 2021
There are some issues about this (#8748, #9557) and probably some changes to make (e.g. #8727 (comment)). Currently the cache is set to a temporary directory:
astropy/conftest.py
Line 55 in 3cbb775
pllim commentedon Jun 3, 2021
Oh ops, forgot about those. Thanks, @saimn ! So, is this a duplicate of #8748 ?
mhvk commentedon Jun 3, 2021
Yes, sorry, I would say this is a duplicate of #8748 and #9557. I guess this is now just a reminder that it would be useful to solve those...