Repository navigation
astropy.utils.data download race condition from multiple processes #609
Description
If multiple processes (for example in an mpi4py job) attempt to use astropy.utils.data to download and cache the same file, there exists a race condition where the processes can stomp on each other. Here is a tarball containing a minimal working example that demonstrates the problem at NERSC. Any computing center with more than one node and a shared home directory should see the same problem:
One possible solution would be:
-
Provide a script with healpy that can pre-download all auxilliary data files to a central location
-
Support using an environment variable or other mechanism to communicate where healpy should find these files.
Running the Test
At NERSC, load any python stack that has a compatible mpi4py (either the NERSC
supported one or cmbenv). For example:
module load python
pip install --upgrade --user astropy
pip install --user healpy
OR
module use /global/common/software/cmb/cori/default/modulefiles
module load cmbenv
source cmbenv
Next, ensure that the astropy download cache is empty, or move the
directory out of the way if you prefer:
rm -rf ~/.astropy/cache/download
Now submit the example included in this tarball:
sbatch run.slurm
Test Results
The central problem is a race condition. It is possible that by sheer luck one
process will download the file before the others try and no error will occur. I
have purposely chosen the job concurrency to be high, so that the collision
between processes is more likely. If your job happens to work, remove the
astropy download directory and try again.
Here is a typical error from one process:
Proc 40: Traceback (most recent call last):
Proc 40: File "/global/u1/k/kisner/git/toast/examples/astropy_cache/./data_test.py", line 56, in <module>
main()
Proc 40: File "/global/u1/k/kisner/git/toast/examples/astropy_cache/./data_test.py", line 44, in main
data = rot.rotate_map_alms(data, use_pixel_weights=True)
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/healpy/rotator.py", line 417, in rotate_map_alms
m, use_pixel_weights=use_pixel_weights, lmax=lmax, mmax=mmax
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/healpy/sphtfunc.py", line 216, in map2alm
package="healpy",
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/astropy/utils/data.py", line 627, in get_pkg_data_filename
conf.dataurl_mirror + data_name])
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/astropy/utils/data.py", line 1147, in download_file
http_headers=http_headers)
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/astropy/utils/data.py", line 992, in _download_file_from_source
with _cache(pkgname) as (dldir, url2hash):
Proc 40: File "/usr/common/software/python/3.7-anaconda-2019.10/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/astropy/utils/data.py", line 1694, in _cache
with _cache_lock(pkgname), shelve.open(urlmapfn, flag="r") as url2hash:
Proc 40: File "/usr/common/software/python/3.7-anaconda-2019.10/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
Proc 40: File "/global/homes/k/kisner/.local/cori/3.7-anaconda-2019.10/lib/python3.7/site-packages/astropy/utils/data.py", line 1595, in _cache_lock
raise RuntimeError(msg)
Proc 40: RuntimeError: Cache is locked after 5.02 s. This may indicate an astropy bug or that kill -9 was used. If you want to unlock the cache remove the directory /global/homes/k/kisner/.astropy/cache/download/py3/lock. Lock claims to be held by process 22640.
Activity
zonca commentedon Apr 16, 2020
Related to astropy/astropy#9970
tskisner commentedon Apr 16, 2020
While I agree that it is related to the upstream astropy issue, it would be great to have an alternate solution that did not use astropy data caching at all. This way we have more control over exactly when data is downloaded and where it is placed. Falling back to astropy is fine, and very convenient for users running on their personal systems. However, download-on-demand of big data files seems incompatible with large parallel workflows.
mreineck commentedon Apr 17, 2020
I'm not sure, but would it help setting
datapathwhen calling the SHT functions? If the provided directory contains all required files, I expect that this should work ... but I also have dim recollections that I tried it at some point and ran into problems ...Sorry, no time right now to look more closely at this...
tskisner commentedon May 5, 2020
Just to note that this cache locking issue happens even if the data is already downloaded locally. Even in this read-only scenario, astropy is locking the hash table just to make a copy of it in memory:
https://github.com/astropy/astropy/blob/a26a6613133ea18b8ae0bfcbfd4d013bcf5adcb1/astropy/utils/data.py#L1719
It would be awesome to have a way to point healpy to a pre-downloaded set of pixel weight files and point it to those at runtime in order to skip
astropy.utils.data.zonca commentedon May 5, 2020
ok, @tskisner, then this is way more urgent than I though, I'll work on it this week
zonca commentedon May 6, 2020
@tskisner can you use master? or do you need a release?
tskisner commentedon May 6, 2020
I can use master for testing now. It will likely be a few weeks before anyone besides me is going to "pip install healpy" and then try to run a large benchmark job. I can ping you later if no release happens before that time.
zonca commentedon May 6, 2020
hivon commentedon May 7, 2020