build.rs caches stale PTX when nvcc availability changes (no rerun-if-env-changed=PATH) #8707

What's your question?

Summary

vortex-cuda/build.rs compiles CUDA kernels to PTX only when nvcc is found on PATH, but it does not register PATH as a rerun trigger. This causes cargo to cache a build.rs result from an environment without nvcc and silently skip PTX compilation when the developer later enters an environment with nvcc (e.g. nix develop). The resulting binary has an empty embedded_ptx table, every load_function_with_suffixes call fails, and execute_cuda falls back to CPU — returning Ok with no kernel launched.

Reproduction

1. From a shell without nvcc on PATH, build any CUDA benchmark:

   cargo bench -p vortex-cuda --bench alp_cuda --no-run

   build.rs runs, is_cuda_available() returns false, PTX compilation is skipped, and embedded_ptx.rs is generated as a no-op (embedded_ptx(_) -> None). Cargo caches this build.rs result.

2. Enter a shell with nvcc (e.g. nix develop) and run the benchmark:

   nix develop -c bash -c 'cargo bench -p vortex-cuda --bench alp_cuda'

   Cargo does not re-run build.rs (none of the declared rerun-if-changed / rerun-if-env-changed inputs changed), so the stale empty PTX table is linked in.

3. Every benchmark reports zero time:

   Benchmarking cuda/cuda/alp_f32/0%/100M: Analyzing
   Criterion.rs ERROR: At least one measurement of benchmark cuda/cuda/alp_f32/0%/100M
   took zero time per iteration. This should not be possible. If using iter_custom,
   please verify that your routine is correctly measured.
   

Root cause

build.rs declares:

println!("cargo:rerun-if-env-changed=PROFILE");

but not cargo:rerun-if-env-changed=PATH. The nvcc availability check (is_cuda_available()) depends on PATH, but that dependency is invisible to cargo's build script caching.

The downstream chain:

1. embedded_ptx.rs returns None for every module → load_function_with_suffixes fails
2. hybrid_dispatch::try_gpu_dispatch returns Err
3. CudaArrayExt::execute_cuda (executor.rs:473-492) catches the error and silently falls back to CPU, returning Ok
4. Benchmarks using iter_custom + TimedLaunchStrategy read timer.load() = 0 because on_complete was never called (no kernel was launched)
5. Criterion reports "took zero time per iteration"

Workaround

touch vortex-cuda/build.rs

This forces cargo to re-run build.rs, which will now find nvcc and compile PTX.

Proposed fix

Add cargo:rerun-if-env-changed=PATH to build.rs:

println!("cargo:rerun-if-env-changed=PROFILE");
println!("cargo:rerun-if-env-changed=PATH");  // new

This ensures cargo re-runs build.rs whenever PATH changes, so is_cuda_available() is re-evaluated.

Environment

• OS: NixOS / any system using nix develop for CUDA tooling
• GPU: present, driver loaded (nvidia-smi works, /dev/nvidia0 exists)
• nvcc: provided by nix develop via cudaPackages.cuda_nvcc, not on the base PATH

Additional note

The silent CPU fallback in execute_cuda (executor.rs:473-492) makes this class of bug very hard to diagnose — a missing PTX looks identical to a successful run from the caller's perspective (returns Ok, no panic). A debug! log is emitted but only visible if a tracing subscriber is initialized, which benchmarks don't do. Consider surfacing this fallback more prominently (e.g. a warning when no PTX is embedded and a GPU is present).

Context

No response