Skip to content

Commit f84a1cc

Browse files
Andy-Jostclaude
andcommitted
cuda.core: name the resource in cleanup reports, export CUDAError, note the detail limitation
Review follow-ups on NVIDIA#2759: - Cleanup reports name the resource handle ("cuMemFreeAsync(0x...) failed ..."). Python's warning registry collapses repeated warnings with identical text from one call site, so two independent resources failing the same call from the same line produced a single CUDAWarning. The pw_* wrappers name their first argument, cleanup_in_context() takes the handle explicitly, and the Buffer deallocation callback names the pointer. A test releases two buffers under an injected restoration failure and expects two reports. - CUDAError and NVRTCError are importable from cuda.core; the error handling page told users to catch CUDAError but it lived in a private module. Both classes gained docstrings. - DESIGN.md and the docs no longer claim that keying the thread-local detail to its status prevents misattribution; a caller that drops the status leaves it behind for a later error with the same code. NVIDIA#2760 removes the thread-local state. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent 80a3d2f commit f84a1cc

11 files changed

Lines changed: 162 additions & 36 deletions

File tree

‎cuda_core/cuda/core/__init__.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,10 +102,12 @@ class _PatchedProperty(metaclass=_PatchedPropMeta):
102102
from cuda.core._stream import __all__ as _stream_all
103103
from cuda.core._tensor_map import *
104104
from cuda.core._tensor_map import __all__ as _tensor_map_all
105-
from cuda.core._utils.cuda_utils import CUDAWarning
105+
from cuda.core._utils.cuda_utils import CUDAError, CUDAWarning, NVRTCError
106106

107107
__all__ = [
108+
"CUDAError",
108109
"CUDAWarning",
110+
"NVRTCError",
109111
*_context_all,
110112
*_device_all,
111113
*_device_resources_all,

‎cuda_core/cuda/core/_cpp/DESIGN.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -300,16 +300,20 @@ detail keyed to the returned status (`take_last_error_detail(status)`) that
300300
`_check_driver_error` attaches to the raised `CUDAError` as a PEP 678 note
301301
(appended to the message on Python 3.10), so the user learns that the caller's
302302
context was not restored, which context is current and, for a double failure,
303-
why restoration failed. Keying the detail to its status keeps it from attaching
304-
to an unrelated error if the caller never raises that status; `enter_context`
305-
clears any stale detail. Tests inject restoration failures with
303+
why restoration failed. Keying the detail to its status narrows, but does not
304+
remove, misattribution: a caller that drops the status (an empty handle raised
305+
as a generic error) leaves the detail behind, and a later error on the same
306+
thread with the same status code picks it up. `enter_context` clears stale
307+
detail at the next context-scoped operation. Issue #2760 removes this
308+
thread-local state in favor of explicit status returns. Tests inject restoration failures with
306309
`set_context_restore_fault_for_testing()`.
307310

308311
### Reporting from non-propagating paths
309312

310313
Deleters and CUDA callbacks cannot raise. They report through
311314
`report_cuda_error()` / `report_message()` (the `pw_*` wrappers decorate
312-
destroy calls with it), which emit a `cuda.core.CUDAWarning` through
315+
destroy calls with it and name the resource handle in the message, so Python's
316+
warning registry does not collapse independent failures of one call), which emit a `cuda.core.CUDAWarning` through
313317
the Python warnings machinery when the interpreter is usable, deliver an
314318
escalated warning as an unraisable exception, and fall back to stderr when the
315319
GIL cannot be taken (for example during finalization). `CUDA_ERROR_DEINITIALIZED`

‎cuda_core/cuda/core/_cpp/resource_handles.cpp‎

Lines changed: 63 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,36 @@ void format_cuda_error(char* buffer, size_t size, const char* operation, CUresul
237237
}
238238
}
239239

240+
// Bits of a resource handle for diagnostics: a pointer as its address, an
241+
// integer handle (CUdeviceptr, CUtexObject, ...) as its value, and a pointer
242+
// to a handle (nvrtcDestroyProgram(&prog) and friends) as the handle it points
243+
// to.
244+
template <typename T>
245+
unsigned long long handle_bits(const T& value) noexcept {
246+
using U = std::remove_cv_t<std::remove_reference_t<T>>;
247+
if constexpr (std::is_pointer_v<U>) {
248+
using P = std::remove_cv_t<std::remove_pointer_t<U>>;
249+
if constexpr (std::is_pointer_v<P>) {
250+
return value ? handle_bits(*value) : 0ull;
251+
} else {
252+
return static_cast<unsigned long long>(reinterpret_cast<uintptr_t>(value));
253+
}
254+
} else if constexpr (std::is_integral_v<U> || std::is_enum_v<U>) {
255+
return static_cast<unsigned long long>(value);
256+
} else {
257+
return 0ull;
258+
}
259+
}
260+
261+
// "<operation>(<handle>)": naming the resource keeps independent failures of
262+
// the same call distinct. Python's warning registry collapses repeated warnings
263+
// with identical text from one call site, so without the handle only the first
264+
// of several leaked resources would be reported.
265+
void format_operation(char* buffer, size_t size, const char* operation,
266+
unsigned long long handle) noexcept {
267+
std::snprintf(buffer, size, "%s(%#llx)", operation, handle);
268+
}
269+
240270
} // namespace
241271

242272
// Report a message that could not be raised. Emits cuda.core.CUDAWarning via
@@ -538,8 +568,11 @@ CUresult invoke_in_context_or_undo(const ContextHandle& h_context, Fn&& operatio
538568
// operation or activation status; restoration never changes the return value.
539569
template <typename Fn, typename... Args>
540570
CUresult cleanup_in_context(const ContextHandle& h_context, const char* name,
541-
Fn&& operation, Args&&... args) noexcept {
571+
unsigned long long handle, Fn&& operation,
572+
Args&&... args) noexcept {
542573
ASSERT_NOTHROW_INVOCABLE(Fn&&, Args&&...);
574+
char operation_name[160];
575+
format_operation(operation_name, sizeof(operation_name), name, handle);
543576
CUcontext previous = nullptr;
544577
int changed = 0;
545578
const char* detail = nullptr;
@@ -551,12 +584,12 @@ CUresult cleanup_in_context(const ContextHandle& h_context, const char* name,
551584
}
552585
CUresult restore = exit_context(previous, changed, CUDA_SUCCESS);
553586
if (status != CUDA_SUCCESS) {
554-
report_cuda_error(name, status, detail);
587+
report_cuda_error(operation_name, status, detail);
555588
}
556589
if (restore != CUDA_SUCCESS) {
557590
// Nothing is raised here, so the detail exit_context recorded has no
558591
// exception to attach to: report it and drop the detail.
559-
report_cuda_error(name, restore, "failed while restoring the caller's context");
592+
report_cuda_error(operation_name, restore, "failed while restoring the caller's context");
560593
clear_last_error_detail();
561594
}
562595
return status;
@@ -572,22 +605,33 @@ class WarnOnFailure {
572605
public:
573606
explicit WarnOnFailure(const char* operation) noexcept : operation_(operation) {}
574607

575-
template <typename... Args>
576-
auto operator()(Args&&... args) const noexcept {
577-
auto status = Function(std::forward<Args>(args)...);
578-
report(status);
608+
// The first argument is the resource being released; it is named in the
609+
// report so that independent failures are not collapsed by the warning
610+
// registry (see format_operation).
611+
template <typename First, typename... Rest>
612+
auto operator()(First&& first, Rest&&... rest) const noexcept {
613+
const unsigned long long handle = handle_bits(first);
614+
auto status = Function(std::forward<First>(first), std::forward<Rest>(rest)...);
615+
report(status, handle);
579616
return status;
580617
}
581618

582619
private:
583-
void report(CUresult status) const noexcept {
584-
report_cuda_error(operation_, status);
620+
void report(CUresult status, unsigned long long handle) const noexcept {
621+
if (status == CUDA_SUCCESS || status == CUDA_ERROR_DEINITIALIZED) {
622+
return;
623+
}
624+
char operation[160];
625+
format_operation(operation, sizeof(operation), operation_, handle);
626+
report_cuda_error(operation, status);
585627
}
586628

587629
template <typename Status>
588-
void report(Status status) const noexcept {
630+
void report(Status status, unsigned long long handle) const noexcept {
589631
if (static_cast<long>(status) != 0) {
590-
report_status_code(operation_, static_cast<long>(status));
632+
char operation[160];
633+
format_operation(operation, sizeof(operation), operation_, handle);
634+
report_status_code(operation, static_cast<long>(status));
591635
}
592636
}
593637

@@ -1544,7 +1588,7 @@ DevicePtrHandle deviceptr_alloc_from_pool(size_t size, const MemoryPoolHandle& h
15441588
GILReleaseGuard gil;
15451589
const DeallocationStream& stream = b->deallocation;
15461590
cleanup_in_context(
1547-
deallocation_context(stream), "cuMemFreeAsync",
1591+
deallocation_context(stream), "cuMemFreeAsync", handle_bits(b->resource),
15481592
[&]() noexcept {
15491593
return p_cuMemFreeAsync(
15501594
b->resource, as_cu(stream.h_stream));
@@ -1574,7 +1618,7 @@ DevicePtrHandle deviceptr_alloc_async(size_t size, const StreamHandle& h_stream)
15741618
GILReleaseGuard gil;
15751619
const DeallocationStream& stream = b->deallocation;
15761620
cleanup_in_context(
1577-
deallocation_context(stream), "cuMemFreeAsync",
1621+
deallocation_context(stream), "cuMemFreeAsync", handle_bits(b->resource),
15781622
[&]() noexcept {
15791623
return p_cuMemFreeAsync(
15801624
b->resource, as_cu(stream.h_stream));
@@ -1659,7 +1703,7 @@ DevicePtrHandle deviceptr_create_mapped_graphics(
16591703
CUgraphicsResource resource = as_cu(h_resource);
16601704
const DeallocationStream& stream = b->deallocation;
16611705
cleanup_in_context(
1662-
deallocation_context(stream), "cuGraphicsUnmapResources",
1706+
deallocation_context(stream), "cuGraphicsUnmapResources", handle_bits(resource),
16631707
[&]() noexcept {
16641708
return p_cuGraphicsUnmapResources(
16651709
1, &resource, as_cu(stream.h_stream));
@@ -1698,7 +1742,7 @@ DevicePtrHandle deviceptr_create_with_mr(CUdeviceptr ptr, size_t size, PyObject*
16981742
if (mr_dealloc_cb) {
16991743
const DeallocationStream& stream = b->deallocation;
17001744
cleanup_in_context(
1701-
deallocation_context(stream), "MemoryResource.deallocate",
1745+
deallocation_context(stream), "MemoryResource.deallocate", handle_bits(b->resource),
17021746
[&]() noexcept {
17031747
mr_dealloc_cb(mr, b->resource, size, stream.h_stream);
17041748
return CUDA_SUCCESS;
@@ -1803,7 +1847,7 @@ DevicePtrHandle deviceptr_import_ipc(const MemoryPoolHandle& h_pool, const void*
18031847
GILReleaseGuard gil;
18041848
const DeallocationStream& stream = b->deallocation;
18051849
cleanup_in_context(
1806-
deallocation_context(stream), "cuMemFreeAsync",
1850+
deallocation_context(stream), "cuMemFreeAsync", handle_bits(b->resource),
18071851
[&]() noexcept {
18081852
return p_cuMemFreeAsync(
18091853
b->resource, as_cu(stream.h_stream));
@@ -1834,7 +1878,7 @@ DevicePtrHandle deviceptr_import_ipc(const MemoryPoolHandle& h_pool, const void*
18341878
GILReleaseGuard gil;
18351879
const DeallocationStream& stream = b->deallocation;
18361880
cleanup_in_context(
1837-
deallocation_context(stream), "cuMemFreeAsync",
1881+
deallocation_context(stream), "cuMemFreeAsync", handle_bits(b->resource),
18381882
[&]() noexcept {
18391883
return p_cuMemFreeAsync(
18401884
b->resource, as_cu(stream.h_stream));
@@ -3258,7 +3302,7 @@ TexObjectHandle make_tex_object_handle(const CUDA_RESOURCE_DESC& res,
32583302
new TexObjectBox{TexObjectValue{obj}, std::move(h_backing), h_context},
32593303
[](const TexObjectBox* b) {
32603304
GILReleaseGuard gil;
3261-
cleanup_in_context(b->h_context, "cuTexObjectDestroy", [&]() noexcept {
3305+
cleanup_in_context(b->h_context, "cuTexObjectDestroy", handle_bits(b->resource.raw), [&]() noexcept {
32623306
return p_cuTexObjectDestroy(b->resource.raw);
32633307
});
32643308
delete b;
@@ -3306,7 +3350,7 @@ SurfObjectHandle create_surf_object_handle(const ContextHandle& h_context,
33063350
new SurfObjectBox{SurfObjectValue{obj}, h_backing, h_context},
33073351
[](const SurfObjectBox* b) {
33083352
GILReleaseGuard gil;
3309-
cleanup_in_context(b->h_context, "cuSurfObjectDestroy", [&]() noexcept {
3353+
cleanup_in_context(b->h_context, "cuSurfObjectDestroy", handle_bits(b->resource.raw), [&]() noexcept {
33103354
return p_cuSurfObjectDestroy(b->resource.raw);
33113355
});
33123356
delete b;

‎cuda_core/cuda/core/_memory/_buffer.pyx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,8 @@ cdef void _mr_dealloc_callback(
7777
mr.deallocate(int(ptr), size, stream=stream)
7878
except Exception as exc:
7979
warnings.warn(
80-
f"mr.deallocate() failed during Buffer destruction; the allocation may have leaked: {exc}",
80+
f"mr.deallocate({int(ptr):#x}) failed during Buffer destruction; "
81+
f"the allocation may have leaked: {exc}",
8182
CUDAWarning,
8283
stacklevel=2,
8384
)

‎cuda_core/cuda/core/_utils/cuda_utils.pyi‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,16 @@ _keep_nvrtc_in_stub: nvrtc.nvrtcResult
1212
_keep_runtime_in_stub: runtime.cudaError_t
1313
_fork_warning_checked = False
1414

15-
class CUDAError(Exception): ...
15+
class CUDAError(Exception):
16+
"""Raised when a CUDA driver or runtime call fails.
17+
18+
The message names the CUDA error and, when one is known, explains it. A
19+
secondary failure observed while the error was being raised, such as a
20+
failed restoration of the caller's CUDA context, is attached as a note
21+
(``__notes__``) on Python 3.11 and newer and appended to the message on
22+
older interpreters. See the error handling page of the ``cuda.core``
23+
documentation for the guarantees an exception provides.
24+
"""
1625

1726
class CUDAWarning(RuntimeWarning):
1827
"""Warning issued when ``cuda.core`` hits a CUDA error it cannot raise.
@@ -35,7 +44,8 @@ class CUDAWarning(RuntimeWarning):
3544
.. versionadded:: 1.3.0
3645
"""
3746

38-
class NVRTCError(CUDAError): ...
47+
class NVRTCError(CUDAError):
48+
"""Raised when an NVRTC call fails; the compiler log is appended when available."""
3949

4050
class ComputeCapability(NamedTuple):
4151
"""A named tuple of (major, minor) CUDA compute capability version numbers."""

‎cuda_core/cuda/core/_utils/cuda_utils.pyx‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,15 @@ from cuda.core._utils.runtime_cuda_error_explanations import RUNTIME_CUDA_ERROR_
3737

3838

3939
class CUDAError(Exception):
40-
pass
40+
"""Raised when a CUDA driver or runtime call fails.
41+
42+
The message names the CUDA error and, when one is known, explains it. A
43+
secondary failure observed while the error was being raised, such as a
44+
failed restoration of the caller's CUDA context, is attached as a note
45+
(``__notes__``) on Python 3.11 and newer and appended to the message on
46+
older interpreters. See the error handling page of the ``cuda.core``
47+
documentation for the guarantees an exception provides.
48+
"""
4149

4250

4351
class CUDAWarning(RuntimeWarning):
@@ -67,7 +75,7 @@ register_warning_category(<PyObject*>CUDAWarning)
6775

6876

6977
class NVRTCError(CUDAError):
70-
pass
78+
"""Raised when an NVRTC call fails; the compiler log is appended when available."""
7179

7280

7381

‎cuda_core/docs/source/api.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,6 +324,8 @@ reported as a warning instead.
324324
.. autosummary::
325325
:toctree: generated/
326326

327+
CUDAError
328+
NVRTCError
327329
CUDAWarning
328330

329331

‎cuda_core/docs/source/error_handling.rst‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,11 @@ Exceptions
1515
----------
1616

1717
A CUDA driver, runtime, NVRTC, NVVM or nvJitLink call that fails raises an
18-
exception (``CUDAError`` for driver and runtime failures) whose message contains
19-
the CUDA error name and its description. Invalid arguments and misuse raise the
18+
exception whose message contains the error name and its description:
19+
:class:`CUDAError` for driver and runtime failures, its subclass
20+
:class:`NVRTCError` for NVRTC failures, and the ``cuda.bindings`` error types
21+
for NVVM and nvJitLink failures. Both ``cuda.core`` classes are importable from
22+
the top-level ``cuda.core`` namespace. Invalid arguments and misuse raise the
2023
usual Python exception types (``TypeError``, ``ValueError``, ``RuntimeError``).
2124

2225
When a ``cuda.core`` call raises, the following hold:
@@ -55,7 +58,10 @@ Some ``cuda.core`` code runs where no Python exception can propagate:
5558
- callbacks invoked by CUDA.
5659

5760
A CUDA error in one of these places is reported as a :class:`CUDAWarning`. The
58-
message names the failed driver call and the CUDA error. The warning means the
61+
message names the failed driver call, the handle of the resource involved, and
62+
the CUDA error. Python shows a warning with a given text once per call site by
63+
default; because the handle differs per resource, independent failures are not
64+
collapsed into one report. The warning means the
5965
affected resource may have leaked; ``cuda.core`` never leaves a resource in use
6066
by CUDA with its memory released (it prefers a leak to a dangling pointer).
6167

‎cuda_core/docs/source/release/1.3.0-notes.rst‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,10 @@ New features
2727
cannot be raised are reported, and how context restoration failures and
2828
sticky CUDA errors are handled.
2929

30+
- :class:`CUDAError` and :class:`NVRTCError` are now importable from the
31+
top-level ``cuda.core`` namespace, alongside :class:`CUDAWarning`. Previously
32+
they were only available from a private module.
33+
3034
Fixes and enhancements
3135
----------------------
3236

@@ -66,8 +70,10 @@ Fixes and enhancements
6670
``cuLinkDestroy``, ``cuArrayDestroy``, ``cuMipmappedArrayDestroy``,
6771
``cuTexObjectDestroy``, ``cuSurfObjectDestroy``, user-object releases and the
6872
NVRTC, NVVM and nvJitLink destroy calls made from destructors were previously
69-
discarded; they are now reported. ``CUDA_ERROR_DEINITIALIZED`` (the driver is
70-
shutting down) is not reported. Test code that matched the old ``stderr``
73+
discarded; they are now reported, and each message names the failed call,
74+
the handle of the affected resource and the CUDA error, so Python's
75+
once-per-call-site warning filter does not collapse independent failures.
76+
``CUDA_ERROR_DEINITIALIZED`` (the driver is shutting down) is not reported. Test code that matched the old ``stderr``
7177
text uses ``pytest.warns(CUDAWarning)`` instead.
7278

7379
- When a :class:`Device` method has to run in the device's context and the

0 commit comments

Comments
 (0)