Skip to content

Commit 4a922e7

Browse files
rust-timerrust-timer
authored
Unrolled build for rust-lang#117458
Rollup merge of rust-lang#117458 - kjetilkjeka:embedded-linker, r=petrochenkov LLVM Bitcode Linker: A self contained linker for nvptx and other targets This PR introduces a new linker named `llvm-bitcode-linker`. It is a `self-contained` linker that can be used to link programs in `llbc` before optimizing and compiling to native code. It will first be used internally in the Rust compiler to enable tests for the `nvptx64-nvidia-cuda` target as the original `rust-ptx-linker` is deprecated. It will then be provided to users of the `nvptx64-nvidia-cuda` target with the purpose of linking ptx. More targets than nvptx will also be supported eventually. The PR introduces a new unstable `LinkerFlavor` for the compiler. The compiler will also not be shipped with rustc but most likely instead be shipped in it's own unstable component (a follow up PR will be opened for this). This means that merging this PR should not add any stability guarantees. When more details of `self-contained` is implemented it will only be possible to use the linker when `-Clink-self-contained=+linker` is passed. <details> <summary>Original Description</summary> **When this PR was created it was focused a bit differently. The original text is preserved here in case there's some interests in it** I have experimenting with approaches to replace the ptx-linker and enable the nvptx target tests again. I think it's time to get some feedback on the approach. ### The problem The only useful linker for the nvptx target is [this crate](https://github.com/denzp/rust-ptx-linker). Since this linker performs linking on llvm bitcode it needs to track the llvm version of rustc and use the same format. It has not been maintained for 3+ years and must be considered abandoned. Over the years rust have upgraded LLVM while the linker has been left to bitrot. It is no longer in a usable state. Due to the difficulty of keeping the ptx-linker up to date outside of tree the nvptx tests was [disabled a long time ago](rust-lang@f8f9a28). It was [previously discussed](rust-lang#96842 (comment)) if adding the ptx-linker to the rust repo would be a possibility. My efforts in doing this stopped at getting an answered if the license would prohibit it from inclusion in the [Rust repo](rust-lang#96842 (comment)). I therefore concluded that a re-write would be necessary. ### The possible solution presented here The llvm tools know perfectly well how to link and optimize llvm bitcode. Each of them only perform a single task, and are therefore a bit cumbersome to call with the current linker approach rustc takes. This PR adds a simple tool (current name `embedded-linker`) which can link self contained (often embedded) programs in llvm bitcode before compiling to the target format. Optimization will also be performed if lto is enabled. The rust compiler will make a single invocation to this tool, while the tool will orchestrate the many calls to the llvm tools. ### The questions - Is having control over the nvptx linking and therefore also tests worth it to add such tool? or should the tool live outside the rust repo? - Is the approach of calling llvm tools acceptable? Or would we want to keep the ptx-linker approach of using the llvm library? The tools seems to provide more simplicity and stability, but more intermediate files are being written. Perhaps there also are some performance penalty for the calling tools approach. - What is the process for adding such tool? MCP? - Does adding `llvm-link` to the llvm-tool component require any process? - Does it require some sort of FCP to remove ptx-linker as the default linker for ptx? Or is it sufficient that using the upstream ptx-linker is broken in its current state. it is possible to use a somewhat patched version of ptx-linker. </details>
2 parents 65cd843 + 843dd28 commit 4a922e7

32 files changed

+529
-25
lines changed

‎Cargo.lock

+11
Original file line numberDiff line numberDiff line change
@@ -2265,6 +2265,17 @@ checksum = "f9d642685b028806386b2b6e75685faadd3eb65a85fff7df711ce18446a422da"
22652265
name = "lld-wrapper"
22662266
version = "0.1.0"
22672267

2268+
[[package]]
2269+
name = "llvm-bitcode-linker"
2270+
version = "0.0.1"
2271+
dependencies = [
2272+
"anyhow",
2273+
"clap",
2274+
"thiserror",
2275+
"tracing",
2276+
"tracing-subscriber",
2277+
]
2278+
22682279
[[package]]
22692280
name = "lock_api"
22702281
version = "0.4.11"

‎Cargo.toml

+1
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ members = [
3434
"src/tools/expand-yaml-anchors",
3535
"src/tools/jsondocck",
3636
"src/tools/jsondoclint",
37+
"src/tools/llvm-bitcode-linker",
3738
"src/tools/html-checker",
3839
"src/tools/bump-stage0",
3940
"src/tools/replace-version-placeholder",

‎compiler/rustc_codegen_ssa/src/back/link.rs

+19-4
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ use rustc_span::symbol::Symbol;
2424
use rustc_target::spec::crt_objects::CrtObjects;
2525
use rustc_target::spec::LinkSelfContainedComponents;
2626
use rustc_target::spec::LinkSelfContainedDefault;
27+
use rustc_target::spec::LinkerFlavorCli;
2728
use rustc_target::spec::{Cc, LinkOutputKind, LinkerFlavor, Lld, PanicStrategy};
2829
use rustc_target::spec::{RelocModel, RelroLevel, SanitizerSet, SplitDebuginfo};
2930

@@ -1350,6 +1351,7 @@ pub fn linker_and_flavor(sess: &Session) -> (PathBuf, LinkerFlavor) {
13501351
}
13511352
}
13521353
LinkerFlavor::Bpf => "bpf-linker",
1354+
LinkerFlavor::Llbc => "llvm-bitcode-linker",
13531355
LinkerFlavor::Ptx => "rust-ptx-linker",
13541356
}),
13551357
flavor,
@@ -1367,8 +1369,17 @@ pub fn linker_and_flavor(sess: &Session) -> (PathBuf, LinkerFlavor) {
13671369

13681370
// linker and linker flavor specified via command line have precedence over what the target
13691371
// specification specifies
1370-
let linker_flavor =
1371-
sess.opts.cg.linker_flavor.map(|flavor| sess.target.linker_flavor.with_cli_hints(flavor));
1372+
let linker_flavor = match sess.opts.cg.linker_flavor {
1373+
// The linker flavors that are non-target specific can be directly translated to LinkerFlavor
1374+
Some(LinkerFlavorCli::Llbc) => Some(LinkerFlavor::Llbc),
1375+
Some(LinkerFlavorCli::Ptx) => Some(LinkerFlavor::Ptx),
1376+
// The linker flavors that corresponds to targets needs logic that keeps the base LinkerFlavor
1377+
_ => sess
1378+
.opts
1379+
.cg
1380+
.linker_flavor
1381+
.map(|flavor| sess.target.linker_flavor.with_cli_hints(flavor)),
1382+
};
13721383
if let Some(ret) = infer_from(sess, sess.opts.cg.linker.clone(), linker_flavor) {
13731384
return ret;
13741385
}
@@ -2338,8 +2349,12 @@ fn add_order_independent_options(
23382349
});
23392350
}
23402351

2341-
if flavor == LinkerFlavor::Ptx {
2342-
// Provide the linker with fallback to internal `target-cpu`.
2352+
if flavor == LinkerFlavor::Llbc {
2353+
cmd.arg("--target");
2354+
cmd.arg(sess.target.llvm_target.as_ref());
2355+
cmd.arg("--target-cpu");
2356+
cmd.arg(&codegen_results.crate_info.target_cpu);
2357+
} else if flavor == LinkerFlavor::Ptx {
23432358
cmd.arg("--fallback-arch");
23442359
cmd.arg(&codegen_results.crate_info.target_cpu);
23452360
} else if flavor == LinkerFlavor::Bpf {

‎compiler/rustc_codegen_ssa/src/back/linker.rs

+100-1
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,7 @@ pub fn get_linker<'a>(
153153
LinkerFlavor::Msvc(..) => Box::new(MsvcLinker { cmd, sess }) as Box<dyn Linker>,
154154
LinkerFlavor::EmCc => Box::new(EmLinker { cmd, sess }) as Box<dyn Linker>,
155155
LinkerFlavor::Bpf => Box::new(BpfLinker { cmd, sess }) as Box<dyn Linker>,
156+
LinkerFlavor::Llbc => Box::new(LlbcLinker { cmd, sess }) as Box<dyn Linker>,
156157
LinkerFlavor::Ptx => Box::new(PtxLinker { cmd, sess }) as Box<dyn Linker>,
157158
}
158159
}
@@ -1824,7 +1825,7 @@ impl<'a> Linker for PtxLinker<'a> {
18241825
}
18251826

18261827
Lto::No => {}
1827-
};
1828+
}
18281829
}
18291830

18301831
fn output_filename(&mut self, path: &Path) {
@@ -1862,6 +1863,104 @@ impl<'a> Linker for PtxLinker<'a> {
18621863
fn linker_plugin_lto(&mut self) {}
18631864
}
18641865

1866+
/// The `self-contained` LLVM bitcode linker
1867+
pub struct LlbcLinker<'a> {
1868+
cmd: Command,
1869+
sess: &'a Session,
1870+
}
1871+
1872+
impl<'a> Linker for LlbcLinker<'a> {
1873+
fn cmd(&mut self) -> &mut Command {
1874+
&mut self.cmd
1875+
}
1876+
1877+
fn set_output_kind(&mut self, _output_kind: LinkOutputKind, _out_filename: &Path) {}
1878+
1879+
fn link_dylib_by_name(&mut self, _name: &str, _verbatim: bool, _as_needed: bool) {
1880+
panic!("external dylibs not supported")
1881+
}
1882+
1883+
fn link_staticlib_by_name(
1884+
&mut self,
1885+
_name: &str,
1886+
_verbatim: bool,
1887+
_whole_archive: bool,
1888+
_search_paths: &SearchPaths,
1889+
) {
1890+
panic!("staticlibs not supported")
1891+
}
1892+
1893+
fn link_staticlib_by_path(&mut self, path: &Path, _whole_archive: bool) {
1894+
self.cmd.arg(path);
1895+
}
1896+
1897+
fn include_path(&mut self, path: &Path) {
1898+
self.cmd.arg("-L").arg(path);
1899+
}
1900+
1901+
fn debuginfo(&mut self, _strip: Strip, _: &[PathBuf]) {
1902+
self.cmd.arg("--debug");
1903+
}
1904+
1905+
fn add_object(&mut self, path: &Path) {
1906+
self.cmd.arg(path);
1907+
}
1908+
1909+
fn optimize(&mut self) {
1910+
match self.sess.opts.optimize {
1911+
OptLevel::No => "-O0",
1912+
OptLevel::Less => "-O1",
1913+
OptLevel::Default => "-O2",
1914+
OptLevel::Aggressive => "-O3",
1915+
OptLevel::Size => "-Os",
1916+
OptLevel::SizeMin => "-Oz",
1917+
};
1918+
}
1919+
1920+
fn output_filename(&mut self, path: &Path) {
1921+
self.cmd.arg("-o").arg(path);
1922+
}
1923+
1924+
fn framework_path(&mut self, _path: &Path) {
1925+
panic!("frameworks not supported")
1926+
}
1927+
1928+
fn full_relro(&mut self) {}
1929+
1930+
fn partial_relro(&mut self) {}
1931+
1932+
fn no_relro(&mut self) {}
1933+
1934+
fn gc_sections(&mut self, _keep_metadata: bool) {}
1935+
1936+
fn no_gc_sections(&mut self) {}
1937+
1938+
fn pgo_gen(&mut self) {}
1939+
1940+
fn no_crt_objects(&mut self) {}
1941+
1942+
fn no_default_libraries(&mut self) {}
1943+
1944+
fn control_flow_guard(&mut self) {}
1945+
1946+
fn ehcont_guard(&mut self) {}
1947+
1948+
fn export_symbols(&mut self, _tmpdir: &Path, _crate_type: CrateType, symbols: &[String]) {
1949+
match _crate_type {
1950+
CrateType::Cdylib => {
1951+
for sym in symbols {
1952+
self.cmd.arg("--export-symbol").arg(sym);
1953+
}
1954+
}
1955+
_ => (),
1956+
}
1957+
}
1958+
1959+
fn subsystem(&mut self, _subsystem: &str) {}
1960+
1961+
fn linker_plugin_lto(&mut self) {}
1962+
}
1963+
18651964
pub struct BpfLinker<'a> {
18661965
cmd: Command,
18671966
sess: &'a Session,

‎compiler/rustc_target/src/spec/mod.rs

+19-3
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,8 @@ pub enum LinkerFlavor {
123123
Bpf,
124124
/// Linker tool for Nvidia PTX.
125125
Ptx,
126+
/// LLVM bitcode linker that can be used as a `self-contained` linker
127+
Llbc,
126128
}
127129

128130
/// Linker flavors available externally through command line (`-Clinker-flavor`)
@@ -141,6 +143,7 @@ pub enum LinkerFlavorCli {
141143
EmCc,
142144
Bpf,
143145
Ptx,
146+
Llbc,
144147

145148
// Legacy stable values
146149
Gcc,
@@ -160,6 +163,7 @@ impl LinkerFlavorCli {
160163
| LinkerFlavorCli::Msvc(Lld::Yes)
161164
| LinkerFlavorCli::EmCc
162165
| LinkerFlavorCli::Bpf
166+
| LinkerFlavorCli::Llbc
163167
| LinkerFlavorCli::Ptx => true,
164168
LinkerFlavorCli::Gcc
165169
| LinkerFlavorCli::Ld
@@ -219,6 +223,7 @@ impl LinkerFlavor {
219223
LinkerFlavorCli::Msvc(lld) => LinkerFlavor::Msvc(lld),
220224
LinkerFlavorCli::EmCc => LinkerFlavor::EmCc,
221225
LinkerFlavorCli::Bpf => LinkerFlavor::Bpf,
226+
LinkerFlavorCli::Llbc => LinkerFlavor::Llbc,
222227
LinkerFlavorCli::Ptx => LinkerFlavor::Ptx,
223228

224229
// Below: legacy stable values
@@ -258,6 +263,7 @@ impl LinkerFlavor {
258263
LinkerFlavor::Msvc(..) => LinkerFlavorCli::Msvc(Lld::No),
259264
LinkerFlavor::EmCc => LinkerFlavorCli::Em,
260265
LinkerFlavor::Bpf => LinkerFlavorCli::Bpf,
266+
LinkerFlavor::Llbc => LinkerFlavorCli::Llbc,
261267
LinkerFlavor::Ptx => LinkerFlavorCli::Ptx,
262268
}
263269
}
@@ -272,6 +278,7 @@ impl LinkerFlavor {
272278
LinkerFlavor::Msvc(lld) => LinkerFlavorCli::Msvc(lld),
273279
LinkerFlavor::EmCc => LinkerFlavorCli::EmCc,
274280
LinkerFlavor::Bpf => LinkerFlavorCli::Bpf,
281+
LinkerFlavor::Llbc => LinkerFlavorCli::Llbc,
275282
LinkerFlavor::Ptx => LinkerFlavorCli::Ptx,
276283
}
277284
}
@@ -286,6 +293,7 @@ impl LinkerFlavor {
286293
LinkerFlavorCli::Msvc(lld) => (Some(Cc::No), Some(lld)),
287294
LinkerFlavorCli::EmCc => (Some(Cc::Yes), Some(Lld::Yes)),
288295
LinkerFlavorCli::Bpf | LinkerFlavorCli::Ptx => (None, None),
296+
LinkerFlavorCli::Llbc => (None, None),
289297

290298
// Below: legacy stable values
291299
LinkerFlavorCli::Gcc => (Some(Cc::Yes), None),
@@ -340,7 +348,7 @@ impl LinkerFlavor {
340348
LinkerFlavor::WasmLld(cc) => LinkerFlavor::WasmLld(cc_hint.unwrap_or(cc)),
341349
LinkerFlavor::Unix(cc) => LinkerFlavor::Unix(cc_hint.unwrap_or(cc)),
342350
LinkerFlavor::Msvc(lld) => LinkerFlavor::Msvc(lld_hint.unwrap_or(lld)),
343-
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Ptx => self,
351+
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Llbc | LinkerFlavor::Ptx => self,
344352
}
345353
}
346354

@@ -355,20 +363,23 @@ impl LinkerFlavor {
355363
pub fn check_compatibility(self, cli: LinkerFlavorCli) -> Option<String> {
356364
let compatible = |cli| {
357365
// The CLI flavor should be compatible with the target if:
358-
// 1. they are counterparts: they have the same principal flavor.
359366
match (self, cli) {
367+
// 1. they are counterparts: they have the same principal flavor.
360368
(LinkerFlavor::Gnu(..), LinkerFlavorCli::Gnu(..))
361369
| (LinkerFlavor::Darwin(..), LinkerFlavorCli::Darwin(..))
362370
| (LinkerFlavor::WasmLld(..), LinkerFlavorCli::WasmLld(..))
363371
| (LinkerFlavor::Unix(..), LinkerFlavorCli::Unix(..))
364372
| (LinkerFlavor::Msvc(..), LinkerFlavorCli::Msvc(..))
365373
| (LinkerFlavor::EmCc, LinkerFlavorCli::EmCc)
366374
| (LinkerFlavor::Bpf, LinkerFlavorCli::Bpf)
375+
| (LinkerFlavor::Llbc, LinkerFlavorCli::Llbc)
367376
| (LinkerFlavor::Ptx, LinkerFlavorCli::Ptx) => return true,
377+
// 2. The linker flavor is independent of target and compatible
378+
(LinkerFlavor::Ptx, LinkerFlavorCli::Llbc) => return true,
368379
_ => {}
369380
}
370381

371-
// 2. or, the flavor is legacy and survives this roundtrip.
382+
// 3. or, the flavor is legacy and survives this roundtrip.
372383
cli == self.with_cli_hints(cli).to_cli()
373384
};
374385
(!compatible(cli)).then(|| {
@@ -387,6 +398,7 @@ impl LinkerFlavor {
387398
| LinkerFlavor::Unix(..)
388399
| LinkerFlavor::EmCc
389400
| LinkerFlavor::Bpf
401+
| LinkerFlavor::Llbc
390402
| LinkerFlavor::Ptx => LldFlavor::Ld,
391403
LinkerFlavor::Darwin(..) => LldFlavor::Ld64,
392404
LinkerFlavor::WasmLld(..) => LldFlavor::Wasm,
@@ -412,6 +424,7 @@ impl LinkerFlavor {
412424
| LinkerFlavor::Msvc(_)
413425
| LinkerFlavor::Unix(_)
414426
| LinkerFlavor::Bpf
427+
| LinkerFlavor::Llbc
415428
| LinkerFlavor::Ptx => false,
416429
}
417430
}
@@ -431,6 +444,7 @@ impl LinkerFlavor {
431444
| LinkerFlavor::Msvc(_)
432445
| LinkerFlavor::Unix(_)
433446
| LinkerFlavor::Bpf
447+
| LinkerFlavor::Llbc
434448
| LinkerFlavor::Ptx => false,
435449
}
436450
}
@@ -480,6 +494,7 @@ linker_flavor_cli_impls! {
480494
(LinkerFlavorCli::Msvc(Lld::No)) "msvc"
481495
(LinkerFlavorCli::EmCc) "em-cc"
482496
(LinkerFlavorCli::Bpf) "bpf"
497+
(LinkerFlavorCli::Llbc) "llbc"
483498
(LinkerFlavorCli::Ptx) "ptx"
484499

485500
// Legacy stable flavors
@@ -2220,6 +2235,7 @@ fn add_link_args_iter(
22202235
| LinkerFlavor::Unix(..)
22212236
| LinkerFlavor::EmCc
22222237
| LinkerFlavor::Bpf
2238+
| LinkerFlavor::Llbc
22232239
| LinkerFlavor::Ptx => {}
22242240
}
22252241
}

‎compiler/rustc_target/src/spec/targets/nvptx64_nvidia_cuda.rs

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
use crate::spec::LinkSelfContainedDefault;
12
use crate::spec::{LinkerFlavor, MergeFunctions, PanicStrategy, Target, TargetOptions};
23

34
pub fn target() -> Target {
@@ -52,6 +53,9 @@ pub fn target() -> Target {
5253
// The LLVM backend does not support stack canaries for this target
5354
supports_stack_protector: false,
5455

56+
// Support using `self-contained` linkers like the llvm-bitcode-linker
57+
link_self_contained: LinkSelfContainedDefault::True,
58+
5559
..Default::default()
5660
},
5761
}

‎compiler/rustc_target/src/spec/tests/tests_impl.rs

+4-1
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,10 @@ impl Target {
5656
LinkerFlavor::Msvc(..) => {
5757
assert_matches!(flavor, LinkerFlavor::Msvc(..))
5858
}
59-
LinkerFlavor::EmCc | LinkerFlavor::Bpf | LinkerFlavor::Ptx => {
59+
LinkerFlavor::EmCc
60+
| LinkerFlavor::Bpf
61+
| LinkerFlavor::Ptx
62+
| LinkerFlavor::Llbc => {
6063
assert_eq!(flavor, self.linker_flavor)
6164
}
6265
}

‎config.example.toml

+4
Original file line numberDiff line numberDiff line change
@@ -679,6 +679,10 @@
679679
# sysroot.
680680
#llvm-tools = true
681681

682+
# Indicates whether the `self-contained` llvm-bitcode-linker, will be made available
683+
# in the sysroot
684+
#llvm-bitcode-linker = false
685+
682686
# Whether to deny warnings in crates
683687
#deny-warnings = true
684688

‎src/bootstrap/configure.py

+2
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ def v(*args):
5454
o("profiler", "build.profiler", "build the profiler runtime")
5555
o("full-tools", None, "enable all tools")
5656
o("lld", "rust.lld", "build lld")
57+
o("llvm-bitcode-linker", "rust.llvm-bitcode-linker", "build llvm bitcode linker")
5758
o("clang", "llvm.clang", "build clang")
5859
o("use-libcxx", "llvm.use-libcxx", "build LLVM with libc++")
5960
o("control-flow-guard", "rust.control-flow-guard", "Enable Control Flow Guard")
@@ -366,6 +367,7 @@ def apply_args(known_args, option_checking, config):
366367
set('rust.codegen-backends', ['llvm'], config)
367368
set('rust.lld', True, config)
368369
set('rust.llvm-tools', True, config)
370+
set('rust.llvm-bitcode-linker', True, config)
369371
set('build.extended', True, config)
370372
elif option.name in ['option-checking', 'verbose-configure']:
371373
# this was handled above

‎src/bootstrap/defaults/config.compiler.toml

+2
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ lto = "off"
1717
# Forces frame pointers to be used with `-Cforce-frame-pointers`.
1818
# This can be helpful for profiling at a small performance cost.
1919
frame-pointers = true
20+
# Build the llvm-bitcode-linker as it is required for running nvptx tests
21+
llvm-bitcode-linker = true
2022

2123
[llvm]
2224
# Having this set to true disrupts compiler development workflows for people who use `llvm.download-ci-llvm = true`

0 commit comments

Comments
 (0)