Skip to content

fix(cuda): do not leak contexts when construction fails - #10872

Merged
0ax1 merged 2 commits into
developfrom
ad/cuda-context-construction-leak
Nov 3, 2026
Merged

0ax1 merged 2 commits into
developfrom
ad/cuda-context-construction-leak

Conversation

@0ax1

@0ax1 0ax1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes a CUDA context leak when querying device capabilities fails during
construction.

Previously, CudaContext::new retained the device's primary context, and
new_non_primary created a context, before querying memory-pool support. If
that query failed, construction returned before a Rust owner existed to
release the retain or destroy the context. A newly created non-primary
context also remained current on the calling thread.

Query device capabilities before acquiring the context instead. After
acquisition, wrap establishes ownership before the remaining fallible bind,
so a binding failure runs Drop and attempts cleanup. The successful path
makes the same driver calls, just in a different order.

Changes

  • fix(cuda): do not leak contexts when construction fails
  • docs(cuda): clarify context construction failure safety

Benchmark

Historical develop baseline: 246a397 (saved results; not rerun)
HEAD: 509ef36
HEAD rebased onto: eed0212

2 cold + 2 hot samples per dataset/revision, after one excluded warmup.
Times are means in ms. Negative change = faster on HEAD.

+-------+-------------------------+----------+----------+---------+
| Cache | Dataset                 | develop  | HEAD     | Change  |
+-------+-------------------------+----------+----------+---------+
| Cold  | Synthetic, 2M rows      |    2.657 |    2.591 |  -2.49% |
| Cold  | Numeric, 2M rows        |    1.831 |    1.757 |  -4.02% |
| Cold  | Lineitem SF10, decimal  |  100.261 |   99.135 |  -1.12% |
| Cold  | Lineitem SF10, float64  |  102.007 |  102.906 |  +0.88% |
+-------+-------------------------+----------+----------+---------+
| Hot   | Synthetic, 2M rows      |    1.497 |    1.492 |  -0.36% |
| Hot   | Numeric, 2M rows        |    0.737 |    0.717 |  -2.74% |
| Hot   | Lineitem SF10, decimal  |   73.779 |   70.870 |  -3.94% |
| Hot   | Lineitem SF10, float64  |   71.591 |   69.884 |  -2.38% |
+-------+-------------------------+----------+----------+---------+

GPU: GH200. Vortex block rows: 16,777,216.
I/O: native. GPU allocator: cuda_async.
Cold/hot refers to OS file-cache residency.

No clear regression signal. Two samples and a historical baseline do not
isolate this fix's performance impact from intervening develop changes.

@0ax1 0ax1 added the changelog/fix A bug fix label Oct 27, 2026
@0ax1

0ax1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@claude review this pr in depth

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @0ax1's task in 41s —— View job


Review of #10872: fix(cuda): do not leak contexts when construction fails

Verdict: correct, and ready to merge. I found no bugs. There are two optional wording nits below.

The fix

Before this PR, both constructors had a fallible step between taking ownership of the context and building the CudaContext whose Drop would clean it up:

Path Before After
new cuDevicePrimaryCtxRetain → get_attribute(...)? → struct exists get_attribute → retain → struct exists
new_non_primary cuCtxCreate_v2 (pushes the context onto the calling thread) → get_attribute(...)? → struct exists get_attribute → create → struct exists

If the attribute query failed on the old order, new kept a primary-context refcount that nothing released. new_non_primary left a live context on top of the thread's context stack. The PR description is accurate.

Why the new order is safe

  • cuDeviceGetAttribute needs no context. It only takes a CUdevice, so it can run before any context exists. result::device::get_attribute (result.rs:145) is a thin wrapper and has no context requirement, so the SAFETY comment device is valid still holds.
  • Nothing fallible runs between acquiring and wrapping. After cuDevicePrimaryCtxRetain / cuCtxCreate_v2 succeed, the next step is Self::wrap. There, Arc::new comes before the only remaining fallible call, bind_to_thread().
  • A bind_to_thread failure in wrap is cleaned up. Returning Err drops the only Arc, so Drop runs:
    • Its own bind_to_thread failure goes through record_err, which is harmless because the object is being torn down.
    • events is empty, so nothing else needs cleanup.
    • cuDevicePrimaryCtxRelease_v2(device) takes only the device. cuCtxDestroy_v2(ctx) does not require the context to be current, and the driver pops it from the calling thread's stack if it is current there. Either way, the retain or create is undone once.
  • Error order is unchanged. init, then the device lookup (including INVALID_VALUE for an ordinal that does not fit in an i32), then the attribute query. The only difference is that the attribute error now comes before the retain or create, not after.
  • Call sites. device and wrap are private, with just these two call sites, so the signature changes go no further.

Nits (optional)

  1. context.rs:103-104: the doc comment on device overstates things. "Everything that can fail is queried here, before a context is retained or created, so that no failure leaves one behind" isn't literally true, since the retain or create can still fail, and so can bind_to_thread in wrap. Those cases are safe for other reasons: nothing is acquired yet, or Drop handles it. A tighter version:

    Queries everything that does not need a context here, so that nothing fallible runs between acquiring the context and wrapping it in a CudaContext whose Drop releases it.

  2. context.rs:120-126: wrap now takes two adjacent bools, has_async_alloc and primary, with context/ordinal between them. A call that swaps them would still compile. With only two private call sites the risk is low, so this is fine as is. If you'd like it a bit clearer, you could put the two bools next to each other or name them at the call site.

Tests

There's no regression test. That's reasonable here: triggering this path means making cuDeviceGetAttribute fail after cuDeviceGet succeeds, which needs driver fault injection that this crate doesn't have.

Checks

I reviewed the code by reading it and did not compile it or run any checks, matching the PR's note that CI runs them. The change only reorders calls and threads one extra bool through, so there is little compile risk.

@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 11.7%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 1876 untouched benchmarks
⏩ 468 skipped benchmarks1
🗄️ 47 archived benchmarks run2

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime lt_i64_nullable_avx512 8.2 µs 7.3 µs +11.7%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ad/cuda-context-construction-leak (509ef36) with develop (eed0212)

Open in CodSpeed

Footnotes

  1. 468 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 47 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

0ax1 added 2 commits November 3, 2026 12:43
`CudaContext::new` retained the primary context, and `new_non_primary`
created a context, before querying whether the device supports memory
pools. If that query failed, the `?` returned before the `CudaContext`
existed, so the retain was never released and the created context was
never destroyed, and stayed current on the calling thread.

Query the attribute together with the device, before retaining or
creating the context.

Signed-off-by: Alexander Droste <[email protected]>
@0ax1
0ax1 force-pushed the ad/cuda-context-construction-leak branch from b9ca7df to 509ef36 Compare November 3, 2026 12:48
@0ax1
0ax1 marked this pull request as ready for review November 3, 2026 13:06
@0ax1
0ax1 requested review from moshap-firebolt, mprammer and robert3005 and removed request for moshap-firebolt November 3, 2026 13:06
@0ax1
0ax1 merged commit cafee53 into develop Nov 3, 2026
129 checks passed
@0ax1
0ax1 deleted the ad/cuda-context-construction-leak branch November 3, 2026 14:06
connortsui20 added a commit that referenced this pull request Nov 3, 2026
## Summary

Follow-up to #11110. Keep the remaining noisy cases out of automatic
CodSpeed runs while preserving them for local benchmarks.

The Python scalar exclusions in #11110 never took effect because the
runner exports simulation as `instrumentation`, while the fixture
checked for `simulation`. Use the plugin's resolved mode so these cases
are actually skipped.

## Changes

- Allow a trailing comma in `#[divan::bench(...)]` when the CPU-feature
macro appends its options. The multiline bitpacked take definition
covers this case in CI builds.
- Share the resolved simulation check between the scalar and filter
fixtures. Also skip the 2 integer one-shot filtered scans that still
report system calls under simulation.
- Keep AVX2 bitpacked take with 80 selections local after batching still
produced a regression alert. Retain the 64-selection case and both
AVX-512 cases in CI.
- Keep AVX-512 nullable integer comparison local because its minimum
changes by more than 10% while its median stays stable.
- Keep the small random power-law boolean filter and the large
correlated power-law boolean filter local under simulation. The former
has repeated unexplained changes, and the latter repeatedly makes system
calls that simulation cannot measure.

<details>
<summary>Evidence from recent PR reports</summary>

Reviewed 40 CodSpeed report comments updated between November 2 at 19:28
UTC and November 3 at 19:28 UTC, including their comparison revisions
and the affected measurement modes.

| Case | Evidence |
| --- | --- |
| Python integer scalar scans |
[#11154](#11154 (comment))
still reports the repeated scan's 10 system calls. [A develop run after
#11110](https://github.com/vortex-data/vortex/actions/runs/41677647530/job/126062463160)
executes both integer scalar cases instead of skipping them. The [runner
exports the legacy `instrumentation`
value](https://github.com/CodSpeedHQ/codspeed/blob/main/crates/runner-shared/src/measurement_mode.rs),
which pytest-codspeed 5.0.3 normalizes to simulation. |
| Python integer filtered scans |
[#11130](#11130 (comment))
reports 30 system calls totaling 14.3 ms in `test_scan_filter[int]`.
[#11138](#11138 (comment))
reports 30 calls totaling 16.3 ms in
`test_scan_filter_to_arrow_array[int]`. |
| AVX2 `threshold[80]` |
[#11004](#11004 (comment))
compares `f7b5b1d864` with `e59b236d7b`: 2.183 to 2.531 µs, reported as
a 13.75% regression. Both use 1000 samples of 16 iterations. The head
median is 2.556 µs, so this is not just a single outlying minimum. The
PR does not change the fastlanes code or benchmark. |
| AVX-512 `lt_i64_nullable` |
[#11168](#11168 (comment))
reports 7.342 to 8.217 µs. Several unrelated CUDA PRs report the
opposite direction, including
[#10872](#10872 (comment)),
at 8.208 to 7.348 µs. Their head medians remain around 8.3 to 8.5 µs. |
| Boolean `filter_powerlaw_by_random[10000]` |
[#11167](#11167 (comment))
reports a 16.2% regression,
[#11168](#11168 (comment))
a 23.56% improvement, and
[#11169](#11169 (comment))
a 13.06% regression. The cause is unknown. |
| Boolean `filter_powerlaw_by_correlated_runs[100000]` |
[#10871](#10871 (comment))
and
[#11165](#11165 (comment))
each report 2 system calls totaling about 44 µs. |

The NEON chunked-take alerts on #11090 are on a PR that changes
execution substantially, so those cases remain enabled. Other dictionary
simulation alerts need more attribution before excluding them.

</details>

CI passes formatting, Python lint and script tests, Rust lint,
default-feature and all-feature Rust builds, all 3 CPU walltime
benchmark jobs, and Python benchmarks (36 passed, 4 intentionally
skipped). The previously failed simulation shard also passes. Some
remaining CI jobs and CodSpeed performance analysis are still pending.

No local checks were run. The pre-commit and pre-push hooks were
bypassed as requested.

---------

Signed-off-by: Connor Tsui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants