Repository navigation
nvcc: report subcommand exit codes as exit codes, not signals - #2932
Merged
Merged
Conversation
compile_result_to_output, error_to_output and aggregate_output passed exit codes to ExitStatus::from_raw, which takes a wait status on Unix. The server's fatal-error code -2 therefore surfaced as "Compiler killed by signal 126" (e.g. after `Missing "cubin" file output`), and an exit code of 1 as signal 1. Exit codes are now encoded as such, and aggregate_output keeps the first failing status instead of the maximum of mixed signal numbers and exit codes. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VszvDmRRPP1UcwozsUMY1L
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2932 +/- ##
==========================================
+ Coverage 78.20% 78.46% +0.25%
==========================================
Files 74 74
Lines 43708 43918 +210
==========================================
+ Hits 34184 34461 +277
+ Misses 9524 9457 -67 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sylvestre
reviewed
Oct 23, 2026
| // `ExitStatus::from_raw` takes a wait status on Unix, not an exit code: passed directly, the | ||
| // server's fatal-error code -2 reads as "killed by signal 126" and an exit code of 1 as signal 1. | ||
| #[cfg(unix)] | ||
| fn exit_code_status(code: i32) -> process::ExitStatus { |
Collaborator
There was a problem hiding this comment.
exit_status in src/dist/mod.rs has the same from_raw(code) problem on unix, no?
`From<ProcessOutput> for process::Output` had the same problem as the nvcc path: `ProcessOutput::code` is a real exit code (`status.code()` on the build server), but the client turned it back with `ExitStatus::from_raw`, which takes a wait status on Unix, so a remote compile failing with exit code 1 showed up as "killed by signal 1". The conversion now lives in one place, `util::exit_status_from_code`, used by both `dist` and `nvcc`, with tests for the helper and for the `ProcessOutput` round trip. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VszvDmRRPP1UcwozsUMY1L
A unit test with the fake dist client (runs in every CI job) and an end to end test in tests/dist.rs with a real scheduler and build server: a compile error found by the compiler on the build server must reach the client with exit code 1. Without the previous commit the end to end test fails with "sccache: Compiler killed by signal 1" and exit code 254, and the unit test sees no exit code at all. The source uses an undeclared identifier rather than `#error`, because preprocessing runs locally and an `#error` never reaches the server. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VszvDmRRPP1UcwozsUMY1L
Contributor
Author
|
@sylvestre yes, same bug in
It also affects every ordinary nvcc compile error, not only Tested: the full CI matrix on my fork, where all test jobs pass (Linux, macOS, Windows + CUDA 12.8, dist-tests; only the Codecov upload fails there for lack of a token), plus |
Collaborator
|
Thanks for your PR |
1 task done
bernardladenthin
added a commit
to bernardladenthin/java-llama.cpp
that referenced
this pull request
Oct 25, 2026
sccache 0.19.0 (2026-10-24) reports a failing compiler's exit code instead of "Compiler killed by signal <n>" -- mozilla/sccache#2932, contributed from this project. SCCACHE_DL_VERSION and the Windows install action move to it; the SHA-256 pins are the ones mozilla publishes next to the four assets used here, checked against the downloaded archives. nvcc goes through sccache by default again (SCCACHE_WRAP_NVCC=false opts out). It was made opt-in on 2026-10-09 after every CUDA 13.4 job failed with `Missing "cubin" file output`. That failure needs a PTX-only compute_90/compute_120 target -- ggml's default list has 90-virtual (mozilla/sccache#2862) -- and this project builds its own list since 2026-10-11, every arch -real and only 121-virtual PTX-only. Measured in nvidia/cuda:13.4.3 with llama.cpp b11815: sccache 0.18.0 builds six ggml-cuda TUs with that list, 108/108 hits warm (19 s vs. 243 s), while ggml's list fails exactly as CI did. build.sh's uncached-retry pattern now matches `fatbinary fatal` for the CUDA 13.3-style failure, which 0.19.0 no longer follows with a signal line; ordinary CUDA compile errors, which used to print that line too, now fail fast instead of costing an uncached rebuild. Verified: check-natives, check-run-scripts, check-shared-files, the workflow and action YAML, bash -n; the 0.19.0 musl binary runs. The CI number for the CUDA job is the second run after this (TODO.md). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VszvDmRRPP1UcwozsUMY1L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an nvcc subcommand fails inside sccache, the server answers it with exit code -2, and
nvcc.rsturns that into anExitStatuswithExitStatus::from_raw. On Unixfrom_rawtakes a wait status, not an exit code, so -2 reads as "killed by signal 126" and an exit code of 1 as signal 1. Users seesccache: Compiler killed by signal 126after the real error (e.g.Missing "cubin" file output, #2862), andCompiler killed by signal 1after a failingfatbinary(#2828), although no process was killed.compile_result_to_outputanderror_to_outputnow encode exit codes as exit codes on Unix (Windows'from_rawalready takes the code).aggregate_outputkeeps the first failing status instead of the maximum of mixed signal numbers and exit codes, so a real signal stays a signal;status_to_codeis gone.Unit tests in
nvcc.rscover the -2 case, exit codes,error_to_output, and that signals are kept; four of them fail without the change. Checked end to end with CUDA 13.4.3 and llama.cpp's ggml-cuda (a list with90-virtual, which triggers #2862): the build fails with the same exit code 254 and theMissing "cubin"error, without the signal line.test_run_log_no_permfails locally only because the tests ran as root.Context: we hit this in java-llama.cpp (Java bindings for llama.cpp), whose CI builds the CUDA (and ROCm) natives through sccache over a shared WebDAV cache. The signal line sent our own diagnosis the wrong way -- we recorded it as nvcc being killed and turned nvcc caching off -- so a correct exit status matters to us. The measurements above were made with that project's toolchain (CUDA 13.4.3, llama.cpp b11815, ggml-cuda).
🤖 Generated with Claude Code
https://claude.ai/code/session_01VszvDmRRPP1UcwozsUMY1L