Skip to content

Commit c3dc370

Browse files
Andy-Jostclaude
andcommitted
cuda.core docs: which reporting channel to use, and what pw_ really does
Review follow-ups on NVIDIA#2759: - DESIGN.md gains a table that picks the reporting channel by situation (a path that can raise, an except block whose rollback failed, a deleter, a Python destructor path, a CUDA callback thread), with the thread-local mechanisms marked transitional pending NVIDIA#2760, and a section on p_ versus pw_: a pw_ wrapper acquires the GIL on failure and runs user Python, so it is never used while a C++ lock is held. Python exceptions raised by that code never become C++ exceptions; nothing on the report path may allocate or throw. CUDA callback threads do nothing that needs the GIL; Py_AddPendingCall is how work leaves them. - AGENTS.md gets the same two rules in short form. The header comments on the reporting functions and on WarnOnFailure say what pw_ runs. - note_or_report_cuda_error is renamed attach_rollback_failure: callers are in one situation (a rollback failed while an exception is in flight) and should not have to know the note-or-warning mechanism. The test hook follows. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent e0e03d9 commit c3dc370

10 files changed

Lines changed: 89 additions & 26 deletions

‎cuda_core/AGENTS.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,8 +132,23 @@ below are for contributors. Reviewers and agents should flag violations.
132132
which emits `cuda.core.CUDAWarning`. No `print(file=sys.stderr)` and no
133133
`fprintf` outside that helper. `CUDA_ERROR_DEINITIALIZED` is filtered by the
134134
helper because it means the driver is shutting down.
135+
- **Pick the channel by where you are**: a path that can raise uses
136+
`HANDLE_RETURN`; an `except` block whose rollback failed uses
137+
`attach_rollback_failure()`; a deleter or cleanup path uses a `pw_*`
138+
wrapper or `report_cuda_error()`; the same situation in Cython or Python
139+
uses `warnings.warn(..., CUDAWarning)`; a CUDA callback thread does nothing
140+
that needs the GIL and hands its work to the deferred-cleanup queue
141+
(`Py_AddPendingCall` is GIL-free and allowed there). The table in
142+
`_cpp/DESIGN.md` ("Which channel to use") spells this out.
143+
- **`pw_*` runs user Python**: a `p_` pointer only calls the driver; its `pw_`
144+
twin also acquires the GIL on failure and runs the warning filters,
145+
`showwarning`, or `sys.unraisablehook`, any of which may call back into
146+
cuda.core. Never call a `pw_*` wrapper or `report_*` while holding a C++
147+
lock. Take the GIL as the outermost lock, release it before taking a C++
148+
lock, and when a lock must stay held call `p_`, keep the status, and report
149+
after the lock is released (`deviceptr_import_ipc` is the model).
135150
- **Rollback failure**: the original exception propagates; the failed rollback
136-
is attached to it with `note_or_report_cuda_error()` (a PEP 678 note on
151+
is attached to it with `attach_rollback_failure()` (a PEP 678 note on
137152
Python 3.11+, reported out-of-band on 3.10), or chained with
138153
`raise ... from` when a second exception must be raised. Catching everything
139154
(bare `except:` or `except BaseException:`) is acceptable only for

‎cuda_core/cuda/core/_cpp/DESIGN.md‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,10 +334,47 @@ terminate the process; see `docs/source/error_handling.rst` and the "Failure han
334334
section of `AGENTS.md` for the policy.
335335

336336
A rollback that fails inside a Cython `except` block is not a non-propagating
337-
path: `note_or_report_cuda_error()` attaches it as a note to the exception being
337+
path: `attach_rollback_failure()` attaches it as a note to the exception being
338338
handled (`PyErr_GetHandledException`, Python 3.11+) and falls back to a report
339339
only when there is no such exception or notes are unavailable.
340340

341+
### Which channel to use
342+
343+
Pick the channel by where the failure happens. Every failure goes through
344+
exactly one of these; none is ever dropped.
345+
346+
| Where you are | Use | Result |
347+
|---|---|---|
348+
| Cython, on a path that can raise | `HANDLE_RETURN(status)` | Raises `CUDAError`. A restoration detail recorded by the C++ helper becomes a note on the exception. |
349+
| Cython, after a handle constructor returned an empty handle | `HANDLE_RETURN(get_last_error())`, immediately | Same. Transitional: #2760 makes constructors return the status instead. |
350+
| C++, a helper that runs an operation in another context | Return the `CUresult`; `exit_context` records the restoration detail | Cython raises it. Transitional: #2760 returns the restoration status as a second out-parameter. |
351+
| Cython, inside an `except` block whose rollback failed | `attach_rollback_failure(op, status, detail)` | Adds a note to the exception being handled. Reports instead if nothing is being handled or notes do not exist (Python 3.10). |
352+
| C++, a deleter or deferred cleanup | A `pw_*` wrapper, or `report_cuda_error()` / `report_message()` | Emits `CUDAWarning`. Never raises. |
353+
| Cython or Python, a `__dealloc__` or destructor-path callback | `warnings.warn(msg, CUDAWarning, stacklevel=2)` | Same. |
354+
| A CUDA callback thread | Nothing that needs the GIL. Hand the work to the deferred-cleanup queue with `Py_AddPendingCall` | CUDA forbids driver calls there, and acquiring the GIL there can deadlock with a GIL holder blocked in a driver call. GIL-free C API that only schedules work is fine. |
355+
356+
### `p_` versus `pw_`
357+
358+
A `p_` function pointer calls the driver and nothing else. Its `pw_` twin calls
359+
the driver and, if the call fails, acquires the GIL and runs Python: the warning
360+
filters, `showwarning`, or `sys.unraisablehook`. Any of those can be user code,
361+
and user code can call back into cuda.core. This is the one place where the
362+
handle layer runs code it does not control, and it is the entry point through
363+
which a thread holding a C++ lock can deadlock (see "GIL Management").
364+
365+
Python exceptions raised by that code never become C++ exceptions: the C API
366+
reports them as return codes, and `report_message` hands them to
367+
`sys.unraisablehook`. Nothing on the report path may allocate or throw, since a
368+
deleter is `noexcept`.
369+
370+
So: use `pw_` only in deleters and cleanup paths that hold no C++ lock and have
371+
finished updating the layer's own state. Where a lock must stay held, call
372+
`p_`, keep the status, and report after the lock is released, as
373+
`deviceptr_import_ipc` does. CUDA callback threads need no extra rule for
374+
`pw_`: the driver call is forbidden there, so the wrapper is too. The general
375+
rule for those threads is no GIL and no Python objects; GIL-free scheduling
376+
calls such as `Py_AddPendingCall` are how work leaves them.
377+
341378
## Usage from Cython
342379

343380
```cython

‎cuda_core/cuda/core/_cpp/resource_handles.cpp‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,7 @@ bool add_note_to_handled_exception(const char* message) noexcept {
360360

361361
} // namespace
362362

363-
void note_or_report_cuda_error(const char* operation, CUresult status, const char* detail) noexcept {
363+
void attach_rollback_failure(const char* operation, CUresult status, const char* detail) noexcept {
364364
if (status == CUDA_SUCCESS || status == CUDA_ERROR_DEINITIALIZED) {
365365
return;
366366
}
@@ -600,6 +600,13 @@ CUresult cleanup_in_context(const ContextHandle& h_context, const char* name,
600600
// Decorate a status-returning cleanup call to report whenever it fails. CUDA
601601
// calls (CUresult) are reported with the error name and description; NVRTC,
602602
// NVVM and nvJitLink calls (integer status codes) with the raw code.
603+
//
604+
// A pw_ wrapper is not a p_ pointer with logging. On failure it acquires the
605+
// GIL and runs Python: the warning filters, showwarning, or sys.unraisablehook,
606+
// any of which may be user code that calls back into cuda.core. Never invoke
607+
// one while holding a C++ lock; the GIL must be the outermost lock. Where a
608+
// lock must stay held, call the p_ pointer, keep the status, and report after
609+
// the lock is released (see deviceptr_import_ipc and DESIGN.md).
603610
template <auto& Function>
604611
class WarnOnFailure {
605612
public:
@@ -638,7 +645,8 @@ class WarnOnFailure {
638645
const char* operation_;
639646
};
640647

641-
// Warning-decorated CUDA operations used by non-throwing cleanup paths.
648+
// Warning-decorated CUDA operations for deleters and cleanup paths. Each one
649+
// may run user Python on failure (see WarnOnFailure above): no C++ lock held.
642650
const WarnOnFailure<p_cuStreamDestroy> pw_cuStreamDestroy{"cuStreamDestroy"};
643651
const WarnOnFailure<p_cuEventDestroy> pw_cuEventDestroy{"cuEventDestroy"};
644652
const WarnOnFailure<p_cuMemFree> pw_cuMemFree{"cuMemFree"};

‎cuda_core/cuda/core/_cpp/resource_handles.hpp‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,13 @@ void clear_last_error() noexcept;
6060
// ============================================================================
6161
// Non-propagating error reporting
6262
//
63-
// Paths that cannot raise (shared_ptr deleters, CUDA callbacks, __dealloc__)
64-
// report failures through these functions instead of discarding them. They
65-
// emit a cuda.core.CUDAWarning when the interpreter can be used and write to
66-
// stderr otherwise; they never raise. See docs/source/error_handling.rst.
63+
// Paths that cannot raise (shared_ptr deleters, __dealloc__) report failures
64+
// through these functions instead of discarding them. They emit a
65+
// cuda.core.CUDAWarning when the interpreter can be used and write to stderr
66+
// otherwise; they never raise. Emitting the warning acquires the GIL and runs
67+
// user Python (warning filters, showwarning, sys.unraisablehook), so never call
68+
// them while holding a C++ lock. See docs/source/error_handling.rst and the
69+
// "Which channel to use" table in DESIGN.md.
6770
// ============================================================================
6871

6972
// Register the Python warning category used by report_* (cuda.core.CUDAWarning).
@@ -84,7 +87,7 @@ void report_status_code(const char* operation, long code) noexcept;
8487
// (PEP 678 note, Python 3.11+): for rollback failures inside `except` blocks
8588
// whose original exception is about to be re-raised. When no exception is
8689
// being handled or notes are unavailable, falls back to report_cuda_error().
87-
void note_or_report_cuda_error(const char* operation, CUresult status, const char* detail = nullptr) noexcept;
90+
void attach_rollback_failure(const char* operation, CUresult status, const char* detail = nullptr) noexcept;
8891

8992
// Detail recorded by a context-scoped helper for the CUresult it is about to
9093
// return, e.g. that the caller's context could not be restored. The Cython

‎cuda_core/cuda/core/_resource_handles.pxd‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ cdef void report_cuda_error(
176176
const char* operation, cydriver.CUresult status, const char* detail) noexcept nogil
177177
cdef void report_message(const char* message) noexcept nogil
178178
cdef void report_status_code(const char* operation, long code) noexcept nogil
179-
cdef void note_or_report_cuda_error(
179+
cdef void attach_rollback_failure(
180180
const char* operation, cydriver.CUresult status, const char* detail) noexcept nogil
181181
cdef const char* take_last_error_detail(cydriver.CUresult status) noexcept nogil
182182
cdef void clear_last_error_detail() noexcept nogil

‎cuda_core/cuda/core/_resource_handles.pyi‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,10 @@ def _set_context_restore_fault_for_testing(status: int):
4949
injected failure leaves the target context current, exactly as a failing
5050
``cuCtxSetCurrent`` would, so callers must restore the context themselves.
5151
"""
52-
def _note_or_report_cuda_error_for_testing(status: int):
52+
def _attach_rollback_failure_for_testing(status: int):
5353
"""Attach a failed CUDA call to the exception being handled, or report it.
5454
55-
Test hook for ``note_or_report_cuda_error()``. Called inside an ``except``
55+
Test hook for ``attach_rollback_failure()``. Called inside an ``except``
5656
block it adds a note to the exception being handled (Python 3.11+); anywhere
5757
else it emits a ``CUDAWarning``.
5858
"""

‎cuda_core/cuda/core/_resource_handles.pyx‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,11 @@ cdef extern from "_cpp/resource_handles.hpp" namespace "cuda_core":
4545
void report_message "cuda_core::report_message" (const char* message) noexcept nogil
4646
void report_status_code "cuda_core::report_status_code" (
4747
const char* operation, long code) noexcept nogil
48-
void note_or_report_cuda_error "cuda_core::note_or_report_cuda_error" (
48+
void attach_rollback_failure "cuda_core::attach_rollback_failure" (
4949
const char* operation, cydriver.CUresult status, const char* detail) noexcept nogil
5050
# Alias for calls made from this module: calling the pxd-declared name here
5151
# would make Cython emit a conflicting static prototype for it.
52-
void _note_or_report_cuda_error_local "cuda_core::note_or_report_cuda_error" (
52+
void _attach_rollback_failure_local "cuda_core::attach_rollback_failure" (
5353
const char* operation, cydriver.CUresult status, const char* detail) noexcept nogil
5454
const char* take_last_error_detail "cuda_core::take_last_error_detail" (
5555
cydriver.CUresult status) noexcept nogil
@@ -597,14 +597,14 @@ def _set_context_restore_fault_for_testing(int status):
597597
set_context_restore_fault_for_testing(<cydriver.CUresult>status)
598598

599599

600-
def _note_or_report_cuda_error_for_testing(int status):
600+
def _attach_rollback_failure_for_testing(int status):
601601
"""Attach a failed CUDA call to the exception being handled, or report it.
602602
603-
Test hook for ``note_or_report_cuda_error()``. Called inside an ``except``
603+
Test hook for ``attach_rollback_failure()``. Called inside an ``except``
604604
block it adds a note to the exception being handled (Python 3.11+); anywhere
605605
else it emits a ``CUDAWarning``.
606606
"""
607-
_note_or_report_cuda_error_local(
607+
_attach_rollback_failure_local(
608608
b"cuTestOperation", <cydriver.CUresult>status, b"failed while testing")
609609

610610
# =============================================================================

‎cuda_core/cuda/core/graph/_graph_builder.pyx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ from cuda.core.graph._subclasses cimport (
2020
ExecutableGraphNode,
2121
create_executable_node_view,
2222
)
23-
from cuda.core._resource_handles cimport note_or_report_cuda_error, report_cuda_error
23+
from cuda.core._resource_handles cimport attach_rollback_failure, report_cuda_error
2424
from cuda.core._resource_handles cimport (
2525
GraphExecHandle,
2626
GraphHandle,
@@ -864,7 +864,7 @@ cdef class GraphBuilder:
864864
else:
865865
# The original exception propagates with the failed rollback
866866
# attached as a note (error handling policy).
867-
note_or_report_cuda_error(
867+
attach_rollback_failure(
868868
b"cuGraphDestroyNode", rollback_status,
869869
b"failed while rolling back a child graph node; the node remains in the graph")
870870
raise

‎cuda_core/cuda/core/graph/_graph_node.pyx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ from cuda.core.graph._subclasses cimport (
4545
SwitchNode,
4646
WhileNode,
4747
)
48-
from cuda.core._resource_handles cimport note_or_report_cuda_error
48+
from cuda.core._resource_handles cimport attach_rollback_failure
4949
from cuda.core._resource_handles cimport (
5050
GraphHandle,
5151
GraphNodeHandle,
@@ -1118,7 +1118,7 @@ cdef inline ChildGraphNode GN_embed(GraphNode self, GraphDefinition child_def):
11181118
else:
11191119
# The original exception propagates with the failed rollback
11201120
# attached as a note (error handling policy).
1121-
note_or_report_cuda_error(
1121+
attach_rollback_failure(
11221122
b"cuGraphDestroyNode", rollback_status,
11231123
b"failed while rolling back a child graph node; the node remains in the graph")
11241124
raise

‎cuda_core/tests/test_error_handling.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131
)
3232
from cuda.core._memory._synchronous_memory_resource import _SynchronousMemoryResource
3333
from cuda.core._resource_handles import (
34-
_note_or_report_cuda_error_for_testing,
34+
_attach_rollback_failure_for_testing,
3535
_set_context_restore_fault_for_testing,
3636
)
3737
from cuda.core._stream import default_stream
@@ -305,12 +305,12 @@ def test_rollback_failure_is_attached_to_the_propagating_exception():
305305
except RuntimeError:
306306
if HAS_NOTES:
307307
with assert_no_cuda_warning():
308-
_note_or_report_cuda_error_for_testing(INVALID_VALUE)
308+
_attach_rollback_failure_for_testing(INVALID_VALUE)
309309
else:
310310
with pytest.warns(CUDAWarning, match="cuTestOperation failed while testing"):
311-
_note_or_report_cuda_error_for_testing(INVALID_VALUE)
311+
_attach_rollback_failure_for_testing(INVALID_VALUE)
312312
with assert_no_cuda_warning():
313-
_note_or_report_cuda_error_for_testing(DEINITIALIZED)
313+
_attach_rollback_failure_for_testing(DEINITIALIZED)
314314
raise
315315
exc = excinfo.value
316316
assert str(exc) == "primary failure"
@@ -321,4 +321,4 @@ def test_rollback_failure_is_attached_to_the_propagating_exception():
321321
assert not hasattr(exc, "__notes__")
322322
# With no exception being handled there is nothing to attach to.
323323
with pytest.warns(CUDAWarning, match="cuTestOperation failed while testing"):
324-
_note_or_report_cuda_error_for_testing(INVALID_VALUE)
324+
_attach_rollback_failure_for_testing(INVALID_VALUE)

0 commit comments

Comments
 (0)