Skip to content

Commit b715078

Browse files
jpascucci-nvrwgk
jpascucci-nv
and
rwgk
authored
fix(cuda.core): serialize pinned-memory kernel-access test on Windows (#3436)
* fix(cuda.core): serialize pinned-memory kernel-access test on Windows closes #3426 Add @thread_unsafe_on_windows to the pinned-memory kernel-access test. The test accesses host memory while parallel workers may execute GPU work; synchronizing its own stream does not coordinate those workers. This follows the guard used by similar tests on Windows and WSL. Windows CI is needed to confirm it resolves the reported crashes. The separate OOM failures remain under investigation. Validation: Ruff and git diff --check pass. * fix(cuda.core): report CUDA synchronization errors during test teardown Check the cuCtxSynchronize result in init_cuda so asynchronous errors surface during the triggering test's teardown instead of going unnoticed until a later test. Always attempt context cleanup via finally. Related to #3426. * fix(cuda.core): synchronize the bound context during test teardown Use device.sync() because the test wrapper may pop the current context before init_cuda teardown. This avoids spurious INVALID_CONTEXT errors while preserving synchronization error reporting and context cleanup. Add regression coverage for a missing current context and cleanup after synchronization failures. Related to #3426. * fix(cuda.core): serialize memory-pool watermark reset tests Mark test_mempool_watermark_reset thread_unsafe, matching test_mempool_trim. Both exercise the pool teardown and synchronization sequence associated with the suspected driver race in #3350. * fix(cuda.core): bypass access revocation when destroying host pools Skip peer-access revocation for locally created HOST and HOST_NUMA pools to investigate pinned-pool teardown OOMs and crashes. Continue destroying the pools and preserve cleanup for device, managed, and imported pools. This is a diagnostic workaround pending Windows CI validation. Related to #3426. * fix(cuda.core): preserve the first exception during fixture cleanup Keep setup or synchronization failures primary and attach subsequent cleanup failures as exception notes. Always attempt to pop the context. Add regression coverage for combined failures, including interrupts. * fix(cuda.core): limit host-pool cleanup workaround to pinned allocations HOST and HOST_NUMA also describe managed-pool preferences. Check the allocation type so the pinned-memory workaround preserves managed-pool peer-access revocation. Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> * test(cuda.core): expose pinned allocation OOM and retain its traceback Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> * ci: diagnose PR3436 pinned pool teardown on Windows Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> * ci: move watermark investigation out of PR3436 Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> * test(cuda.core): restore expected pinned allocation OOM handling Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> * revert(cuda.core): defer pinned host pool cleanup workaround Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> --------- Signed-off-by: Ralf W. Grosse-Kunstleve <[email protected]> Co-authored-by: Ralf W. Grosse-Kunstleve <[email protected]>
1 parent b975a83 commit b715078

3 files changed

Lines changed: 134 additions & 3 deletions

File tree

‎cuda_core/tests/conftest.py‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,7 @@ def init_cuda(request):
244244
# live_at_setup stays None until the setup snapshot is recorded.
245245
device.set_current()
246246
live_at_setup = None
247+
primary_error = None
247248
try:
248249
# Set option to avoid spin-waiting on synchronization.
249250
if int(os.environ.get("CUDA_CORE_TEST_BLOCKING_SYNC", 0)) != 0:
@@ -275,6 +276,9 @@ def init_cuda(request):
275276
# not counted, so no cached-default bookkeeping is needed.
276277
live_at_setup = _live_resource_counts()
277278
yield device
279+
except BaseException as exc:
280+
primary_error = exc
281+
raise
278282
finally:
279283
# pytest holds fixture values in item.funcargs (a pytest-internal dict)
280284
# until all teardown has run; we deliberately drop them so already
@@ -298,8 +302,23 @@ def init_cuda(request):
298302
# With pytest-run-parallel this runs after worker join. This also runs
299303
# when setup raises, so an escalated leftover warning cannot leave the
300304
# context current.
301-
driver.cuCtxSynchronize()
302-
_ = _device_unset_current()
305+
# The test wrapper may already have popped the calling thread's
306+
# context; synchronize the fixture device's bound context explicitly.
307+
# Keep the first failure primary; cleanup still runs for interrupts.
308+
try:
309+
device.sync()
310+
except BaseException as exc:
311+
if primary_error is None:
312+
primary_error = exc
313+
raise
314+
primary_error.add_note(f"CUDA fixture synchronization failed: {type(exc).__name__}: {exc}")
315+
finally:
316+
try:
317+
_ = _device_unset_current()
318+
except BaseException as exc:
319+
if primary_error is None:
320+
raise
321+
primary_error.add_note(f"CUDA fixture context pop failed: {type(exc).__name__}: {exc}")
303322
# After the cleanup, so an escalated warning cannot skip it.
304323
if live_after is not None and any(now < snap for now, snap in zip(live_after, live_at_setup)):
305324
released = tuple(s - n for s, n in zip(live_at_setup, live_after))

‎cuda_core/tests/memory/test_init_cuda_collect_gate.py‎

Lines changed: 111 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@
3333
UnreleasedMemoryResourceWarning,
3434
)
3535

36-
from cuda.core._utils.cuda_utils import driver, handle_return
36+
from cuda.core import Device
37+
from cuda.core._utils.cuda_utils import CUDAError, driver, handle_return
3738

3839

3940
class _ScriptedCounts:
@@ -250,3 +251,112 @@ def test_teardown_warns_on_baseline_release(request, monkeypatch):
250251
pytest.raises(StopIteration),
251252
):
252253
next(gen) # teardown: live_after == (0, 0) -> baseline-release warning
254+
255+
256+
@_THREAD_UNSAFE_MARKER
257+
@pytest.mark.agent_authored(model="gpt-6")
258+
@pytest.mark.parametrize("clear_current", [False, True])
259+
def test_teardown_synchronizes_fixture_context(request, monkeypatch, clear_current):
260+
"""Teardown works even after the test wrapper has popped the current context."""
261+
func = _init_cuda_func(request)
262+
_patch_conftest_global(monkeypatch, func, "_mr_expected_live", None)
263+
_patch_conftest_global(monkeypatch, func, "_live_resource_counts", _ScriptedCounts((0, 0)))
264+
265+
saved = _pop_context_stack()
266+
gen = func(_FakeRequest())
267+
try:
268+
next(gen)
269+
if clear_current:
270+
handle_return(driver.cuCtxPopCurrent())
271+
assert current_context_handle() == 0
272+
_finish_teardown(gen)
273+
assert current_context_handle() == 0
274+
finally:
275+
gen.close()
276+
_push_context_stack(saved)
277+
278+
279+
@_THREAD_UNSAFE_MARKER
280+
@pytest.mark.agent_authored(model="gpt-6")
281+
def test_teardown_sync_failure_still_pops_context(request, monkeypatch):
282+
"""A synchronization error propagates without leaving the context current."""
283+
func = _init_cuda_func(request)
284+
_patch_conftest_global(monkeypatch, func, "_mr_expected_live", None)
285+
_patch_conftest_global(monkeypatch, func, "_live_resource_counts", _ScriptedCounts((0, 0)))
286+
failure = CUDAError("injected synchronization failure")
287+
288+
class FailingSyncDevice:
289+
device_id = 0
290+
291+
def set_current(self):
292+
Device(0).set_current()
293+
294+
def sync(self):
295+
raise failure
296+
297+
_patch_conftest_global(monkeypatch, func, "Device", lambda _: FailingSyncDevice())
298+
saved = _pop_context_stack()
299+
gen = func(_FakeRequest())
300+
try:
301+
next(gen)
302+
with pytest.raises(CUDAError) as excinfo:
303+
next(gen)
304+
assert excinfo.value is failure
305+
assert current_context_handle() == 0
306+
finally:
307+
gen.close()
308+
_push_context_stack(saved)
309+
310+
311+
@_THREAD_UNSAFE_MARKER
312+
@pytest.mark.agent_authored(model="gpt-6")
313+
@pytest.mark.parametrize(
314+
"fail_setup, fail_sync, fail_pop",
315+
[(True, True, True), (True, False, True), (True, True, False), (False, True, True), (False, False, True)],
316+
)
317+
@pytest.mark.parametrize("error_type", [RuntimeError, KeyboardInterrupt])
318+
def test_cleanup_preserves_first_failure(request, monkeypatch, fail_setup, fail_sync, fail_pop, error_type):
319+
"""Later cleanup errors annotate the first failure, and pop is always attempted."""
320+
func = _init_cuda_func(request)
321+
_patch_conftest_global(monkeypatch, func, "_mr_expected_live", None)
322+
monkeypatch.setenv("CUDA_CORE_TEST_BLOCKING_SYNC", "0")
323+
setup_error = error_type("injected setup failure")
324+
sync_error = error_type("injected synchronization failure")
325+
pop_error = error_type("injected context pop failure")
326+
state = {"current": False}
327+
328+
class FakeDevice:
329+
def set_current(self):
330+
state["current"] = True
331+
332+
def sync(self):
333+
if fail_sync:
334+
raise sync_error
335+
336+
def counts():
337+
if fail_setup:
338+
raise setup_error
339+
return (0, 0)
340+
341+
def pop():
342+
state["current"] = False
343+
if fail_pop:
344+
raise pop_error
345+
346+
_patch_conftest_global(monkeypatch, func, "Device", lambda _: FakeDevice())
347+
_patch_conftest_global(monkeypatch, func, "_live_resource_counts", counts)
348+
_patch_conftest_global(monkeypatch, func, "_device_unset_current", pop)
349+
gen = func(_FakeRequest())
350+
if not fail_setup:
351+
next(gen)
352+
with pytest.raises(error_type) as excinfo:
353+
next(gen)
354+
expected = setup_error if fail_setup else sync_error if fail_sync else pop_error
355+
assert excinfo.value is expected
356+
assert not state["current"]
357+
notes = getattr(expected, "__notes__", [])
358+
assert len(notes) == int(fail_setup and fail_sync) + int(fail_pop and (fail_setup or fail_sync))
359+
if fail_setup and fail_sync:
360+
assert any(str(sync_error) in note for note in notes)
361+
if fail_pop and (fail_setup or fail_sync):
362+
assert any(str(pop_error) in note for note in notes)

‎cuda_core/tests/test_memory.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1246,6 +1246,7 @@ def test_pinned_memory_resource_initialization(init_cuda):
12461246
buffer.close()
12471247

12481248

1249+
@thread_unsafe_on_windows
12491250
@pytest.mark.human_reviewed
12501251
@pytest.mark.parametrize("pool", ["current", "new", "new_numa"])
12511252
def test_pinned_memory_resource_kernel_access(init_cuda, pool):
@@ -2972,6 +2973,7 @@ def test_mempool_release_threshold_accepts_index_types(memory_resource_factory,
29722973
mr.close()
29732974

29742975

2976+
@pytest.mark.thread_unsafe(reason="CUDA legacy-default-stream mempool teardown/sync race (gh-3350)")
29752977
@pytest.mark.agent_authored(model="claude-opus-5-5")
29762978
@pytest.mark.parametrize("kind", ["reserved", "used"])
29772979
def test_mempool_watermark_reset(memory_resource_factory, kind):

0 commit comments

Comments
 (0)