Skip to content

Commit 9d903cd

Browse files
jpascucci-nvjpascucci-nv
authored
Remove flaky test_concurrent_eviction_does_not_delete_replaced_file (#3150)
Refs #2745 The failure is a race in the test’s shutdown protocol, not clear evidence of a cache bug. A valid failing interleaving is: 1. The rewriter writes survivor. 2. The churner’s final size-cap pass legitimately evicts survivor; the 5,000-byte cap only holds two ~2,000-byte entries. 3. The churner exits. 4. The parent sets done_event. 5. The rewriter observes the event before its next write and exits. 6. The final lookup returns None. The stat guard only prevents deletion when the file changes between the eviction snapshot and its second stat(). It does not protect survivor from ordinary LRU eviction. There is also an explicitly documented residual race between the second stat() and unlink(). Supporting evidence: - Both subprocesses exited successfully; only the final survival assertion failed. - Repetitions 1–3 and 5 passed; repetition 4 failed under Python 3.14t with parallel tests and --count=5. - The same failure was previously reported on an unrelated PR and identified as racy. - A deterministic test already forces replacement between the scan and re-stat and verifies the intended guard at [test_program_cache.py (line 1697)](/data/src/cuda-python/cuda_core/tests/test_program_cache.py:1697). I recommend removing the stochastic multiprocess test at [test_program_cache_multiprocess.py (line 148)](/data/src/cuda-python/cuda_core/tests/test_program_cache_multiprocess.py:148), since the deterministic test covers the property reliably. Simply serializing it or adding a final rewrite would only hide the race, not validate the stat guard.
1 parent 66387f1 commit 9d903cd

1 file changed

Lines changed: 0 additions & 67 deletions

File tree

‎cuda_core/tests/test_program_cache_multiprocess.py‎

Lines changed: 0 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -123,70 +123,3 @@ def test_concurrent_reader_never_sees_torn_file(tmp_path):
123123
# mismatches.
124124
assert hits == 200
125125
assert mismatches == 0
126-
127-
128-
def _worker_size_cap_writer(root: str, prefix: bytes, payload: bytes, count: int, max_size_bytes: int) -> None:
129-
"""Hammer a small-cap cache with churning writes so eviction fires often."""
130-
from cuda.core.utils import FileStreamProgramCache
131-
132-
with FileStreamProgramCache(root, max_size_bytes=max_size_bytes) as cache:
133-
for i in range(count):
134-
cache[prefix + str(i).encode()] = payload
135-
136-
137-
def _worker_size_cap_rewriter(root: str, key: bytes, payload: bytes, max_size_bytes: int, done_event) -> None:
138-
"""Repeatedly rewrite ``key`` with a fresh value until ``done_event`` fires."""
139-
from cuda.core.utils import FileStreamProgramCache
140-
141-
with FileStreamProgramCache(root, max_size_bytes=max_size_bytes) as cache:
142-
i = 0
143-
while not done_event.is_set():
144-
cache[key] = payload + str(i).encode()
145-
i += 1
146-
147-
148-
def test_concurrent_eviction_does_not_delete_replaced_file(tmp_path):
149-
"""Eviction is stat-guarded: while one process is evicting an entry to
150-
bring the cache under its size cap, another process may have already
151-
``os.replace``-d a fresh value into the same path. The evictor must
152-
refuse to unlink in that case, otherwise the racing rewriter's
153-
just-committed entry vanishes.
154-
155-
The rewriter does NO post-race uncontested write -- the entry's
156-
survival has to come from the in-race rewrites being preserved by
157-
the stat-guarded eviction path."""
158-
from cuda.core.utils import FileStreamProgramCache
159-
160-
root = str(tmp_path / "fc")
161-
payload = b"X" * 2000
162-
cap = 5000 # fits 2 raw 2000B entries; the third write triggers eviction
163-
164-
ctx = _mp.get_context("spawn")
165-
done_event = ctx.Event()
166-
rewriter = ctx.Process(
167-
target=_worker_size_cap_rewriter,
168-
args=(root, b"survivor", payload, cap, done_event),
169-
)
170-
# Churning writer creates new keys faster than eviction can drain them,
171-
# forcing _enforce_size_cap to consider 'survivor' for eviction many times.
172-
churner = ctx.Process(
173-
target=_worker_size_cap_writer,
174-
args=(root, b"churn-", payload, 80, cap),
175-
)
176-
rewriter.start()
177-
churner.start()
178-
churner.join(timeout=60)
179-
done_event.set()
180-
rewriter.join(timeout=60)
181-
assert rewriter.exitcode == 0
182-
assert churner.exitcode == 0
183-
184-
# If eviction blindly unlinked replaced files, the rewriter's last
185-
# in-race write would be gone -- there is no post-race write to mask
186-
# the failure. The value must also start with ``payload`` (not a
187-
# ``churn-`` value) because each rewriter iteration writes
188-
# ``payload + str(i).encode()``.
189-
with FileStreamProgramCache(root, max_size_bytes=cap) as cache:
190-
got = cache.get(b"survivor")
191-
assert got is not None, "rewriter's entry was evicted by racing churner"
192-
assert got.startswith(payload), f"survivor value not from rewriter: {got[:32]!r}..."

0 commit comments

Comments
 (0)