Cache directory on read-only filesystem

Description

Hi,

We run astropy in a Docker container on an HPC system using the Shifter container-running platform. In Shifter, the container's filesystems are all read-only. We have pre-loaded a number of cached files required by astropy (eg, IERS). But even though the file already exists in the cache directory, it fails because the filesystem is read-only! This seems not ideal.

$ python -c "from astropy.time import Time; Time.now()"
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/root/.astropy/cache/download/py3/lock' [astropy.utils.data]
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/root/.astropy/cache/download/py3/lock' [astropy.utils.data]
Downloading https://www.ietf.org/timezones/data/leap-seconds.list [Done]

Additional context

ps, weird thing is, strace shows it looks for an "urlmap" file, and then trying to create a lock

stat("/root/.astropy/cache/download/py3/urlmap", 0x7fffffff3d70) = -1 ENOENT (No such file or directory)
mkdir("/root/.astropy/cache/download/py3/lock", 0777) = -1 EROFS (Read-only file system)

but in the cache dir there are files urlmap.dat and urlmap.dir....

EDIT: Workaround

Set astropy.utils.iers.conf.auto_download = False. Also see #10003 (comment)

I feel this is the same as #9970, if it's I suggest to keep the discussions under that issue and close this as a duplicate.

@aarchiba

cc @aarchiba

Not sure I agree -- locking on NFS is a particularly dicey issue. This is a much simpler request -- please can cache dirs that already contain the desired file work on a read-only filesystem.

This is worrying - what version of astropy are you using? As of this version, cache on read-only filesystems is supposed to work, and without any locking. In fact there are reasonably extensive tests that are supposed to verify that it does work. Possibly the problem is that astropy does not recognize the exception that is raised by a read-only filesystem as opposed to setting the read-only flag? This would be an easy fix.

Not sure I agree -- locking on NFS is a particularly dicey issue. This is a much simpler request -- please can cache dirs that already contain the desired file work on a read-only filesystem.

Indeed, this is a separate issue, though related as a pre-loaded read-only cache is probably the best solution to NFS caching headaches, so it had better work.

Hi,

Thanks for your reply.

We're using astropy 4.0.

It looks like the IERS code is being a little obnoxious in its use of the cache: it wants to clear the cache before even checking the cached file?!

https://github.com/astropy/astropy/blob/v4.0/astropy/utils/iers/iers.py#L1001

This is the code path we're hitting, and it's the 'https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat' URL that wants to clear the cache (so the shipped data file astropy/astropy/utils/iers/data/Leap_Second.dat is considered to be expired...)

thanks,
--dustin

FWIW, I just tried grabbing the latest version of the astropy/utils/iers/data/Leap_Second.dat and copying it into the container, and that seems to work... at least for now, until it expires again.....

Theoretically, according to

astropy/astropy/utils/iers/iers.py

Lines 993 to 994 in cad8f07

    
           if conf.auto_download: 
        
               trials += [(f, False) for f in files if urlparse(f).netloc]

If you have

astropy.utils.iers.conf.auto_download = False  # default is True

then it would not hit

astropy/astropy/utils/iers/iers.py

Lines 1000 to 1002 in cad8f07

    
           for f, allow_cache in trials: 
        
               if not allow_cache: 
        
                   clear_download_cache(f)

🤔

@lpsinger

I wonder if setting astropy.utils.iers.conf.auto_download = False would help @lpsinger too even though the situation is a bit different. Maybe the IERS auto-download mechanism is not playing well with cluster computing.

I don't understand the approach of deleting the cache before even checking whether the file exists. That just seems like a straight-up bug.

Checking whether the files exist happened a little earlier at

astropy/astropy/utils/iers/iers.py

Lines 986 to 990 in cad8f07

    
           # Our trials start with normal files and remote ones that are 
        
           # already in cache.  The bools here indicate that the cache 
        
           # should be used. 
        
           trials = [(f, True) for f in files 
        
                     if not urlparse(f).netloc or is_url_in_cache(f)]

The line you encountered is meant to invalidate the cache, as far as I can understand.

Ahhh yes, I see, apologies.

HOWEVER, the is_url_in_cache() fails, NOT because the file isn't there -- it is -- but because it tries to lock the cache before READING!

Specifically, here,

astropy/astropy/utils/data.py

Line 1681 in cad8f07

except dbm.error as e:

I am getting:

dbm.error [Errno 30] Read-only file system: '/tmp/astropy-cache/astropy/download/py3/lock'

So indeed, the cache does not work on a read-only filesystem.

But that line you mentioned is under if write. For read-only, wouldn't it go to the other logic block? 🤔

@aarchiba

Ah, okay. I think the original intend is to lock the cache, so no one can accidentally write over it when it is being read. However, this can cause problem if another read happens while this lock is in place. I think @aarchiba mentioned that maybe there should be two different locks? Feel free to correct me if I misunderstood.

@dstndstn

@dstndstn , but for now, as a workaround, does astropy.utils.iers.conf.auto_download = False fix your particular problem with IERS data access?

Yes, that works around the problem.

For ease of reproducing this, I have made a Docker container with one-liners that demonstrate success/failure with a rw/ro cache directory.

Unzip the attached, then

tar xvf cache2.tar
docker run --mount type=bind,src=$(pwd)/astropy-cache,dst=/tmp/astropy-cache,ro -it dstndstn/astropy-testcase
docker run --mount type=bind,src=$(pwd)/astropy-cache,dst=/tmp/astropy-cache -it dstndstn/astropy-testcase

The first should fail, the second should succeed;

> docker run --mount type=bind,src=$(pwd)/astropy-cache,dst=/tmp/astropy-cache,ro -it dstndstn/astropy-testcase
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/tmp/astropy-cache/astropy/download/py3/lock' [astropy.utils.data]
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|=========================================================================================================================| 1.3k/1.3k (100.00%)         0s
Downloading https://www.ietf.org/timezones/data/leap-seconds.list [Done]
2020-03-03 16:15:34.212033
> docker run --mount type=bind,src=$(pwd)/astropy-cache,dst=/tmp/astropy-cache -it dstndstn/astropy-testcase
2020-03-03 16:15:38.584997

demo.zip

@dstndstn

Thank you, @dstndstn !

I don't understand the approach of deleting the cache before even checking whether the file exists. That just seems like a straight-up bug.

Before the current version, there was no way to ask the cache to obtain a new version if there was one available. This can now be changed to use cache='update' on the download. It still won't address the code's desire to write to the cache if a new version is available (and the old version is considered out of date).

@pllim

(note that this comment of mine was wrong, due to my mis-reading of the code; as @pllim pointed out, it does try to check the cache before doing this; unfortunately, to check the cache it tries to create a lock file, which fails on a r/o filesystem!)

@aarchiba

Ah, okay. I think the original intend is to lock the cache, so no one can accidentally write over it when it is being read. However, this can cause problem if another read happens while this lock is in place. I think @aarchiba mentioned that maybe there should be two different locks? Feel free to correct me if I misunderstood.

The problem is that if someone is trying to write to the cache - specifically to modify the cache index to reflect a new file that has already been downloaded - then the index cannot cope with someone trying to read it. So there's just one lock, and I couldn't figure out how to make a reader/writer lock work with the tools I had (os.mkdir as the assumed-atomic operation).

That said, if a function tries to acquire the lock for reading only, and the cache is read-only, there should be no problem - the read-only-ness should be detected and since there is no need for a lock the read should progress normally without it.

I can see three things, any or all of which might be misbehaving here:

Astropy might be failing to recognize the exception being raised as indicating read-only-ness; the test suite is not able to create a temporary read-only file system. This would be a fairly easy fix, just an addition to the except block in _cache_lock.
is_url_in_cache might be requesting a writeable lock. Obviously unnecessary and would lead to the problems you see.
The IERS code might be demanding writeablility even when it doesn't need it (e.g. no update is actually available). This should be a pretty easy fix, using cache='update' should work fine on a read-only filesystem if no update is available.

Finally, there is one change that would dramatically help this sort of situation:

the IERS code, and other higher-level functions that use the cache under the hood, should fail more gracefully if the cache is read-only; specifically they should notify the user that an update is needed but cannot be obtained and they should explain to the user how to tell them not to worry about having current data if that is acceptable.

@aarchiba

Hi @aarchiba , What's happening is:

The IERS code is correctly calling is_url_in_cache(), which calls _cache(..., write=False).

At this line,

astropy/astropy/utils/data.py

Line 1691 in cad8f07

except dbm.error as e:

I am getting

dbm.error [Errno 30] Read-only file system: '/tmp/astropy-cache/astropy/download/py3/lock'

so then _cache returns an EMPTY ReadOnlyDict().

And then the IERS code doesn't find what's looking for in the (empty) dictionary, then it tries to re-fetch and cache the file, which again fails.

Also, if I may make a suggestion: os.rename() is an atomic operation on Unix systems (this is a POSIX requirement). So one possible way to avoid the need to lock on read is, when updating the cache index, write the cache index to a temp file in the same directory, and then (atomically) rename it to the correct cache index filename, overwriting the old version. Unix systems will correctly handle this situation, even if a reader already has the original version of the database index file open. It's really quite a powerful technique.

(ps, I am pointing to the v4.0 tag of the code, which is what I'm running in production; if you'd like me to check what happens in master, happy to do so.)

Also, if I may make a suggestion: os.rename() is an atomic operation on Unix systems (this is a POSIX requirement). So one possible way to avoid the need to lock on read is, when updating the cache index, write the cache index to a temp file in the same directory, and then (atomically) rename it to the correct cache index filename, overwriting the old version. Unix systems will correctly handle this situation, even if a reader already has the original version of the database index file open. It's really quite a powerful technique.

Unfortunately

rename is not atomic on Windows, which we support
the index is stored in a shelve object, which could be one of several backends and for which we do not even know the name of the underlying file (it is different, and may even be multiple files, on different backends)
incautious use of the above technique risks losing updates unless there is another lock for writing, and the combination does not give a full reader/writer lock; it is not clear to me that that combination would be sufficient

Astropy might be failing to recognize the exception being raised as indicating read-only-ness; the test suite is not able to create a temporary read-only file system. This would be a fairly easy fix, just an addition to the except block in _cache_lock.

Okay I fixed this in PR #10007 ; I'll see whether any of the others can be resolved as well.

I think the IERS code does a reasonably credible job of not updating unless it needs to. So once #10007 is applied I think it handles things fairly well, in the sense that if no update is needed it should work fine, while if an update is needed you get an error message about a read-only file system. So please let me know whether #10007 solves the problem for you.

@aarchiba

@aarchiba Thank you very much for this fix!!

Navigation Menu

Repository navigation

Cache directory on read-only filesystem #10003

Description

Additional context

EDIT: Workaround

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if conf.auto_download:
	trials += [(f, False) for f in files if urlparse(f).netloc]

	for f, allow_cache in trials:
	if not allow_cache:
	clear_download_cache(f)

	# Our trials start with normal files and remote ones that are
	# already in cache. The bools here indicate that the cache
	# should be used.
	trials = [(f, True) for f in files
	if not urlparse(f).netloc or is_url_in_cache(f)]

Cache directory on read-only filesystem #10003

Description

Description

Additional context

EDIT: Workaround

Activity

bsipocz commented on Mar 2, 2020

bsipocz commented on Mar 2, 2020

dstndstn commented on Mar 2, 2020

aarchiba commented on Mar 2, 2020

aarchiba commented on Mar 2, 2020

dstndstn commented on Mar 2, 2020

dstndstn commented on Mar 2, 2020

pllim commented on Mar 2, 2020

pllim commented on Mar 2, 2020

dstndstn commented on Mar 2, 2020

pllim commented on Mar 2, 2020

dstndstn commented on Mar 2, 2020

pllim commented on Mar 3, 2020

1 remaining item

pllim commented on Mar 3, 2020

pllim commented on Mar 3, 2020

dstndstn commented on Mar 3, 2020

pllim commented on Mar 3, 2020

aarchiba commented on Mar 4, 2020

dstndstn commented on Mar 4, 2020

aarchiba commented on Mar 4, 2020

dstndstn commented on Mar 4, 2020

dstndstn commented on Mar 4, 2020

dstndstn commented on Mar 4, 2020

aarchiba commented on Mar 4, 2020

aarchiba commented on Mar 4, 2020

aarchiba commented on Mar 4, 2020

dstndstn commented on Mar 5, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions