Repository navigation
fix(cuda): do not leak contexts when construction fails - #10872
Conversation
|
@claude review this pr in depth |
|
Claude finished @0ax1's task in 41s —— View job Review of #10872:
|
| Path | Before | After |
|---|---|---|
new |
cuDevicePrimaryCtxRetain → get_attribute(...)? → struct exists |
get_attribute → retain → struct exists |
new_non_primary |
cuCtxCreate_v2 (pushes the context onto the calling thread) → get_attribute(...)? → struct exists |
get_attribute → create → struct exists |
If the attribute query failed on the old order, new kept a primary-context refcount that nothing released. new_non_primary left a live context on top of the thread's context stack. The PR description is accurate.
Why the new order is safe
cuDeviceGetAttributeneeds no context. It only takes aCUdevice, so it can run before any context exists.result::device::get_attribute(result.rs:145) is a thin wrapper and has no context requirement, so the SAFETY commentdevice is validstill holds.- Nothing fallible runs between acquiring and wrapping. After
cuDevicePrimaryCtxRetain/cuCtxCreate_v2succeed, the next step isSelf::wrap. There,Arc::newcomes before the only remaining fallible call,bind_to_thread(). - A
bind_to_threadfailure inwrapis cleaned up. ReturningErrdrops the onlyArc, soDropruns:- Its own
bind_to_threadfailure goes throughrecord_err, which is harmless because the object is being torn down. eventsis empty, so nothing else needs cleanup.cuDevicePrimaryCtxRelease_v2(device)takes only the device.cuCtxDestroy_v2(ctx)does not require the context to be current, and the driver pops it from the calling thread's stack if it is current there. Either way, the retain or create is undone once.
- Its own
- Error order is unchanged.
init, then the device lookup (includingINVALID_VALUEfor an ordinal that does not fit in ani32), then the attribute query. The only difference is that the attribute error now comes before the retain or create, not after. - Call sites.
deviceandwrapare private, with just these two call sites, so the signature changes go no further.
Nits (optional)
context.rs:103-104: the doc comment ondeviceoverstates things. "Everything that can fail is queried here, before a context is retained or created, so that no failure leaves one behind" isn't literally true, since the retain or create can still fail, and so canbind_to_threadinwrap. Those cases are safe for other reasons: nothing is acquired yet, orDrophandles it. A tighter version:Queries everything that does not need a context here, so that nothing fallible runs between acquiring the context and wrapping it in a
CudaContextwhoseDropreleases it.context.rs:120-126:wrapnow takes two adjacentbools,has_async_allocandprimary, withcontext/ordinalbetween them. A call that swaps them would still compile. With only two private call sites the risk is low, so this is fine as is. If you'd like it a bit clearer, you could put the two bools next to each other or name them at the call site.
Tests
There's no regression test. That's reasonable here: triggering this path means making cuDeviceGetAttribute fail after cuDeviceGet succeeds, which needs driver fault injection that this crate doesn't have.
Checks
I reviewed the code by reading it and did not compile it or run any checks, matching the PR's note that CI runs them. The change only reorders calls and threads one extra bool through, so there is little compile risk.
Merging this PR will improve performance by 11.7%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
`CudaContext::new` retained the primary context, and `new_non_primary` created a context, before querying whether the device supports memory pools. If that query failed, the `?` returned before the `CudaContext` existed, so the retain was never released and the created context was never destroyed, and stayed current on the calling thread. Query the attribute together with the device, before retaining or creating the context. Signed-off-by: Alexander Droste <[email protected]>
Signed-off-by: Alexander Droste <[email protected]>
b9ca7df to
509ef36
Compare
## Summary Follow-up to #11110. Keep the remaining noisy cases out of automatic CodSpeed runs while preserving them for local benchmarks. The Python scalar exclusions in #11110 never took effect because the runner exports simulation as `instrumentation`, while the fixture checked for `simulation`. Use the plugin's resolved mode so these cases are actually skipped. ## Changes - Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature macro appends its options. The multiline bitpacked take definition covers this case in CI builds. - Share the resolved simulation check between the scalar and filter fixtures. Also skip the 2 integer one-shot filtered scans that still report system calls under simulation. - Keep AVX2 bitpacked take with 80 selections local after batching still produced a regression alert. Retain the 64-selection case and both AVX-512 cases in CI. - Keep AVX-512 nullable integer comparison local because its minimum changes by more than 10% while its median stays stable. - Keep the small random power-law boolean filter and the large correlated power-law boolean filter local under simulation. The former has repeated unexplained changes, and the latter repeatedly makes system calls that simulation cannot measure. <details> <summary>Evidence from recent PR reports</summary> Reviewed 40 CodSpeed report comments updated between November 2 at 19:28 UTC and November 3 at 19:28 UTC, including their comparison revisions and the affected measurement modes. | Case | Evidence | | --- | --- | | Python integer scalar scans | [#11154](#11154 (comment)) still reports the repeated scan's 10 system calls. [A develop run after #11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160) executes both integer scalar cases instead of skipping them. The [runner exports the legacy `instrumentation` value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs), which pytest-codspeed 5.0.3 normalizes to simulation. | | Python integer filtered scans | [#11130](#11130 (comment)) reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`. [#11138](#11138 (comment)) reports 30 calls totaling 16.3 ms in `test_scan_filter_to_arrow_array[int]`. | | AVX2 `threshold[80]` | [#11004](#11004 (comment)) compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as a 13.75% regression. Both use 1000 samples of 16 iterations. The head median is 2.556 µs, so this is not just a single outlying minimum. The PR does not change the fastlanes code or benchmark. | | AVX-512 `lt_i64_nullable` | [#11168](#11168 (comment)) reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the opposite direction, including [#10872](#10872 (comment)), at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. | | Boolean `filter_powerlaw_by_random[10000]` | [#11167](#11167 (comment)) reports a 16.2% regression, [#11168](#11168 (comment)) a 23.56% improvement, and [#11169](#11169 (comment)) a 13.06% regression. The cause is unknown. | | Boolean `filter_powerlaw_by_correlated_runs[100000]` | [#10871](#10871 (comment)) and [#11165](#11165 (comment)) each report 2 system calls totaling about 44 µs. | The NEON chunked-take alerts on #11090 are on a PR that changes execution substantially, so those cases remain enabled. Other dictionary simulation alerts need more attribution before excluding them. </details> CI passes formatting, Python lint and script tests, Rust lint, default-feature and all-feature Rust builds, all 3 CPU walltime benchmark jobs, and Python benchmarks (36 passed, 4 intentionally skipped). The previously failed simulation shard also passes. Some remaining CI jobs and CodSpeed performance analysis are still pending. No local checks were run. The pre-commit and pre-push hooks were bypassed as requested. --------- Signed-off-by: Connor Tsui <[email protected]>
Summary
Fixes a CUDA context leak when querying device capabilities fails during
construction.
Previously, CudaContext::new retained the device's primary context, and
new_non_primary created a context, before querying memory-pool support. If
that query failed, construction returned before a Rust owner existed to
release the retain or destroy the context. A newly created non-primary
context also remained current on the calling thread.
Query device capabilities before acquiring the context instead. After
acquisition, wrap establishes ownership before the remaining fallible bind,
so a binding failure runs Drop and attempts cleanup. The successful path
makes the same driver calls, just in a different order.
Changes
Benchmark
Historical develop baseline: 246a397 (saved results; not rerun)
HEAD: 509ef36
HEAD rebased onto: eed0212
2 cold + 2 hot samples per dataset/revision, after one excluded warmup.
Times are means in ms. Negative change = faster on HEAD.
GPU: GH200. Vortex block rows: 16,777,216.
I/O: native. GPU allocator: cuda_async.
Cold/hot refers to OS file-cache residency.
No clear regression signal. Two samples and a historical baseline do not
isolate this fix's performance impact from intervening develop changes.