Skip to content

Commit 4c61dcf

Browse files
d-v-bclaude
andauthored
refactor(store): polish the LocalStore rename retry from #4358 (#4359)
Follow-up cleanup to the retry landed in #4358, no behaviour change: - drop the leading 0.0 sentinel, the bare `last_error` annotation, and the possibly-unbound re-raise from `_move_with_retry`; the final attempt now runs after the loop and propagates its own traceback - correct the comment on ERROR_ACCESS_DENIED: Windows also reports it for destinations that will never clear (directory, read-only, ACL), so those surface the same error after the bounded delay - align the changelog with the code: the exclusive path is routed through the retry, only FileExistsError is excluded; rename the fragment to the PR number and describe #3522 as mitigated, since a second process holding the destination open past the budget still fails - tests record `time.sleep` instead of sleeping (~1.2 s per run before, now instant) and assert the actual delay schedule; the closure test double becomes a small callable class, removing four type-ignores - add a regression test for `_atomic_write` onto an existing directory Assisted-by: ClaudeCode:claude-fable-5-1 Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 6947dd8 commit 4c61dcf

4 files changed

Lines changed: 78 additions & 62 deletions

File tree

‎changes/3522.bugfix.md‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎changes/4358.bugfix.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
`LocalStore` now retries the rename that publishes a written file when Windows
2+
reports the destination as transiently busy (`ERROR_ACCESS_DENIED` or
3+
`ERROR_SHARING_VIOLATION`). Replacing a name that was itself replaced moments
4+
earlier intermittently fails this way in a single process, which aborted
5+
otherwise ordinary writes. The retry is bounded to well under a second, is a
6+
no-op off Windows, and never retries the `FileExistsError` that the `exclusive`
7+
path uses to report an existing node. This mitigates but does not fully resolve
8+
#3522: a second process holding the destination open for longer than the retry
9+
budget will still fail. Zarr v2 had the equivalent retry from #698 and it was
10+
not carried over when atomic writes arrived in #3412.

‎src/zarr/storage/_local.py‎

Lines changed: 19 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -59,51 +59,49 @@ def _safe_move(src: Path, dst: Path) -> None:
5959
os.unlink(src)
6060

6161

62-
# Windows error codes meaning the destination could not be superseded *right
63-
# now*, as opposed to a permission problem that will not clear. Replacing a name
64-
# that was itself replaced moments earlier intermittently fails this way, with no
65-
# second process and no open handle involved, and a retry clears it in well under
66-
# a millisecond. Zarr v2 hit the same thing and fixed it in #698.
62+
# Windows error codes that MoveFileEx reports when the destination could not be
63+
# superseded. Replacing a name that was itself replaced moments earlier
64+
# intermittently fails this way with no second process and no open handle
65+
# involved, and a retry clears it in well under a millisecond. Zarr v2 hit the
66+
# same thing and fixed it in #698.
67+
#
68+
# ERROR_ACCESS_DENIED is also what Windows reports for conditions that will not
69+
# clear (the destination is a directory, is read-only, or is ACL-denied). Those
70+
# are retried too and surface the same error after the bounded delay below.
6771
#
6872
# Nothing else is retried. In particular the FileExistsError that the exclusive
6973
# path relies on to report an existing node is ERROR_ALREADY_EXISTS (183), so it
70-
# is excluded here by construction and still propagates on the first attempt.
74+
# still propagates on the first attempt.
7175
_TRANSIENT_WINERRORS = frozenset(
7276
{
7377
5, # ERROR_ACCESS_DENIED
7478
32, # ERROR_SHARING_VIOLATION
7579
}
7680
)
7781

78-
# Delay before each successive attempt; the leading 0.0 is the original attempt.
79-
# Measured on Windows 11 over 20,000 replaces onto an existing destination: 720
80-
# failed with no retry and none with, 475 of those clearing on the second attempt
81-
# and the worst on the fourth. The tail is headroom for a busier machine, and the
82-
# whole sequence sums to under a second so a genuine failure still surfaces
83-
# promptly.
84-
_RETRY_DELAYS = (0.0, 0.001, 0.005, 0.02, 0.05, 0.2)
82+
# Delay before each retry. The sequence sums to well under a second so that a
83+
# genuine failure still surfaces promptly; in practice most transient failures
84+
# clear on the first retry.
85+
_RETRY_DELAYS = (0.001, 0.005, 0.02, 0.05, 0.2)
8586

8687

8788
def _move_with_retry(tmp_path: Path, path: Path, move: Callable[[Path, Path], object]) -> None:
88-
"""Run ``move(tmp_path, path)``, retrying while the destination is busy.
89+
"""Run `move(tmp_path, path)`, retrying while the destination is busy.
8990
9091
This is a single attempt on every platform but Windows, without needing to
91-
test for one: only ``winerror`` values are ever retried, and off Windows an
92-
``OSError`` does not carry one.
92+
test for one: only `winerror` values are ever retried, and off Windows an
93+
`OSError` does not carry one.
9394
"""
94-
last_error: OSError
9595
for delay in _RETRY_DELAYS:
96-
if delay:
97-
time.sleep(delay)
9896
try:
9997
move(tmp_path, path)
10098
except OSError as e:
10199
if getattr(e, "winerror", None) not in _TRANSIENT_WINERRORS:
102100
raise
103-
last_error = e
101+
time.sleep(delay)
104102
else:
105103
return
106-
raise last_error
104+
move(tmp_path, path)
107105

108106

109107
@contextlib.contextmanager

‎tests/test_store/test_local.py‎

Lines changed: 49 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import pathlib
44
import re
5-
from typing import TYPE_CHECKING
5+
import time
66

77
import numpy as np
88
import pytest
@@ -15,9 +15,6 @@
1515
from zarr.testing.store import StoreTests
1616
from zarr.testing.utils import assert_bytes_equal
1717

18-
if TYPE_CHECKING:
19-
from collections.abc import Callable
20-
2118

2219
class TestLocalStore(StoreTests[LocalStore, cpu.Buffer]):
2320
store_cls = LocalStore
@@ -177,75 +174,95 @@ def _oserror(winerror: int) -> OSError:
177174
return error
178175

179176

180-
def _flaky_move(failures: int, error: OSError) -> Callable[[pathlib.Path, pathlib.Path], None]:
181-
"""A move that raises ``error`` the first ``failures`` times it is called."""
182-
attempts = 0
177+
class _FlakyMove:
178+
"""A move that raises `error` the first `failures` times it is called."""
179+
180+
def __init__(self, failures: int, error: OSError) -> None:
181+
self.failures = failures
182+
self.error = error
183+
self.attempts = 0
183184

184-
def move(src: pathlib.Path, dst: pathlib.Path) -> None:
185-
nonlocal attempts
186-
attempts += 1
187-
if attempts <= failures:
188-
raise error
185+
def __call__(self, src: pathlib.Path, dst: pathlib.Path) -> None:
186+
self.attempts += 1
187+
if self.attempts <= self.failures:
188+
raise self.error
189189
src.replace(dst)
190190

191-
move.attempts = lambda: attempts # type: ignore[attr-defined]
192-
return move
191+
192+
@pytest.fixture
193+
def sleeps(monkeypatch: pytest.MonkeyPatch) -> list[float]:
194+
"""Record the delays `_move_with_retry` would sleep for instead of sleeping."""
195+
recorded: list[float] = []
196+
monkeypatch.setattr(time, "sleep", recorded.append)
197+
return recorded
193198

194199

195200
@pytest.mark.parametrize("winerror", [5, 32])
196201
@pytest.mark.parametrize("failures", [1, 2, 3])
197-
def test_move_with_retry_recovers(tmp_path: pathlib.Path, winerror: int, failures: int) -> None:
202+
def test_move_with_retry_recovers(
203+
tmp_path: pathlib.Path, sleeps: list[float], winerror: int, failures: int
204+
) -> None:
198205
"""A destination that is briefly busy is retried, not reported."""
199206
src = tmp_path / "src"
200207
dst = tmp_path / "dst"
201208
src.write_bytes(b"abc")
202-
move = _flaky_move(failures, _oserror(winerror))
209+
move = _FlakyMove(failures, _oserror(winerror))
203210

204211
_move_with_retry(src, dst, move)
205212

206213
assert dst.read_bytes() == b"abc"
207-
assert move.attempts() == failures + 1 # type: ignore[attr-defined]
214+
assert move.attempts == failures + 1
215+
assert sleeps == list(_RETRY_DELAYS[:failures])
208216

209217

210-
def test_move_with_retry_gives_up(tmp_path: pathlib.Path) -> None:
218+
def test_move_with_retry_gives_up(tmp_path: pathlib.Path, sleeps: list[float]) -> None:
211219
"""A destination that never frees still raises, after a bounded wait."""
212220
src = tmp_path / "src"
213221
dst = tmp_path / "dst"
214222
src.write_bytes(b"abc")
215-
move = _flaky_move(len(_RETRY_DELAYS), _oserror(5))
223+
move = _FlakyMove(len(_RETRY_DELAYS) + 1, _oserror(5))
216224

217225
with pytest.raises(OSError, match="Access is denied"):
218226
_move_with_retry(src, dst, move)
219227

220-
assert move.attempts() == len(_RETRY_DELAYS) # type: ignore[attr-defined]
221-
assert not dst.exists()
228+
assert move.attempts == len(_RETRY_DELAYS) + 1
229+
assert sleeps == list(_RETRY_DELAYS)
230+
assert sum(sleeps) < 1.0
222231

223232

224-
@pytest.mark.parametrize("winerror", [2, 3, 183, None])
233+
@pytest.mark.parametrize("winerror", [2, 183, None])
225234
def test_move_with_retry_does_not_retry_other_errors(
226-
tmp_path: pathlib.Path, winerror: int | None
235+
tmp_path: pathlib.Path, sleeps: list[float], winerror: int | None
227236
) -> None:
228237
"""Only a busy destination is transient; everything else fails at once.
229238
230-
183 is the case that matters: ``ERROR_ALREADY_EXISTS`` is how the
231-
``exclusive`` path reports that a node is already there, and retrying it
232-
would overwrite what ``_safe_move`` refused to touch.
239+
183 is the case that matters: `ERROR_ALREADY_EXISTS` is how the
240+
`exclusive` path reports that a node is already there, and retrying it
241+
would overwrite what `_safe_move` refused to touch.
233242
"""
234243
src = tmp_path / "src"
235244
dst = tmp_path / "dst"
236245
src.write_bytes(b"abc")
237246
error = OSError(17, "boom")
238247
if winerror is not None:
239248
error.winerror = winerror # type: ignore[attr-defined]
240-
move = _flaky_move(1, error)
249+
move = _FlakyMove(1, error)
241250

242251
with pytest.raises(OSError, match="boom"):
243252
_move_with_retry(src, dst, move)
244253

245-
assert move.attempts() == 1 # type: ignore[attr-defined]
254+
assert move.attempts == 1
255+
assert sleeps == []
246256

247257

248-
def test_retry_delays_are_bounded() -> None:
249-
"""A stuck destination must not turn a fast failure into a long hang."""
250-
assert sum(_RETRY_DELAYS) < 1.0
251-
assert _RETRY_DELAYS[0] == 0.0 # the original attempt is not delayed
258+
@pytest.mark.parametrize("exclusive", [True, False])
259+
def test_atomic_write_onto_directory(
260+
tmp_path: pathlib.Path, sleeps: list[float], exclusive: bool
261+
) -> None:
262+
"""Writing a key whose destination is a directory fails and leaves no temp file."""
263+
path = tmp_path / "node"
264+
path.mkdir()
265+
with pytest.raises(OSError), _atomic_write(path, "wb", exclusive=exclusive) as f:
266+
f.write(b"abc")
267+
assert path.is_dir()
268+
assert list(tmp_path.iterdir()) == [path] # no temp files

0 commit comments

Comments
 (0)