Skip to content

Commit 363499a

Browse files
weatherfrogSBA
authored andcommitted
avoid race condition during chunk write
When the chunk file is first removed before the new version is moved into place, racing reads may encounter a missing chunk. Using rename() or replace() without remove() avoids the issue on Posix-Systems as the methods are atomic. The fallback of remove() -> rename() is included for Windows pre Python 3.3. Fixes zarr-developers#263
1 parent cc9f4e1 commit 363499a

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

‎zarr/storage.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -771,9 +771,18 @@ def __setitem__(self, key, value):
771771
f.write(value)
772772

773773
# move temporary file into place
774-
if os.path.exists(file_path):
775-
os.remove(file_path)
776-
os.rename(temp_path, file_path)
774+
if hasattr(os, 'replace'):
775+
# Python >= 3.3 has replace() which can replace existing
776+
# files on all platforms.
777+
os.replace(temp_path, file_path)
778+
else:
779+
# In Python < 3.3 use rename().
780+
if os.name == 'nt' and os.path.exists(file_path):
781+
# On windows, rename() can't overwrite files. So
782+
# the file is removed first.
783+
os.remove(file_path)
784+
785+
os.rename(temp_path, file_path)
777786

778787
finally:
779788
# clean up if temp file still exists for whatever reason

0 commit comments

Comments
 (0)