Skip to content

Commit 1372b2e

Browse files
weatherfrogSBA
authored andcommitted
avoid race condition during chunk write
When the chunk file is first removed before the new version is moved into place, racing reads may encounter a missing chunk. Using rename() or replace() without remove() avoids the issue on Posix-Systems as the methods are atomic. The fallback of remove() -> rename() is included for Windows pre Python 3.3. Fixes zarr-developers#263
1 parent dc90a20 commit 1372b2e

1 file changed

Lines changed: 12 additions & 3 deletions

File tree

‎zarr/storage.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -770,9 +770,18 @@ def __setitem__(self, key, value):
770770
f.write(value)
771771

772772
# move temporary file into place
773-
if os.path.exists(file_path):
774-
os.remove(file_path)
775-
os.rename(temp_path, file_path)
773+
if hasattr(os, 'replace'):
774+
# Python >= 3.3 has replace() which can replace existing
775+
# files on all platforms.
776+
os.replace(temp_path, file_path)
777+
else:
778+
# In Python < 3.3 use rename().
779+
if os.name == 'nt' and os.path.exists(file_path):
780+
# On windows, rename() can't overwrite files. So
781+
# the file is removed first.
782+
os.remove(file_path)
783+
784+
os.rename(temp_path, file_path)
776785

777786
finally:
778787
# clean up if temp file still exists for whatever reason

0 commit comments

Comments
 (0)