Skip to content

Commit 8aafa50

Browse files
baderbader
authored
[clang][sycl][nvlink] Share static library linking in Frontend/Offloading (#201253)
Move clang-nvlink-wrapper's archive member selection engine into a new shared library in llvm/lib/Frontend/Offloading (ArchiveLinker.h/.cpp) and use it from both clang-nvlink-wrapper and clang-sycl-linker, adding static library (.a) and -l support to the SYCL linker. The shared llvm::offloading::resolveArchiveMembers() API: - Searches -L paths for -l library names (lib<name>.a or :<name>) - Expands archives, honouring --whole-archive/--no-whole-archive - Runs a symbol-driven fixed-point loop to extract only the archive members that resolve undefined symbols - Returns the resolved MemoryBuffers and symbol table; the symbol table is consumed by clang-nvlink-wrapper's LTO resolution pass clang-sycl-linker gains -l, --whole-archive/--no-whole-archive, and -u options (added to SYCLLinkOpts.td). The previous --bc-library option has been removed in favor of the standard -l mechanism. Bug fixes included: * Fix dangling StringRef UB: Args.getAllArgValues() returns a temporary vector; retain it in ForcedUndefStorage so the StringRefs remain valid through the resolveArchiveMembers call (both tools). * Fix assert crash in clang-sycl-linker when all positional inputs are non-existent: return a proper error instead of propagating an empty buffer vector to linkInputs. * Fixed forced undefined symbol handling: corrected -u option processing to properly handle symbols. Co-Authored-By: Claude
1 parent 1dae5de commit 8aafa50

12 files changed

Lines changed: 806 additions & 408 deletions

File tree

‎clang/docs/ClangSYCLLinker.rst‎

Lines changed: 77 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,10 @@ be passed down to downstream AOT compilation tools like 'ocloc' and 'opencl-aot'
5050
-help-hidden Display all available options
5151
-help Display available options (--help-hidden for more)
5252
-L <dir> Add <dir> to the library search path
53-
--bc-library <name> Add LLVM bitcode library <name> (with extension) to the link. A relative <name> is resolved against -L paths; an absolute path is taken as-is.
53+
-l <libname> Search for library <libname>
54+
--whole-archive Include all archive members in the link
55+
--no-whole-archive Only include archive members that resolve undefined symbols (default)
56+
-u <symbol> Force undefined symbol during linking
5457
--module-split-mode=<mode> Module split mode: 'source' (default), 'kernel', or 'none'
5558
--ocloc-options=<value> Options passed to ocloc for Intel GPU AOT compilation
5659
--opencl-aot-options=<value> Options passed to opencl-aot for Intel CPU AOT compilation
@@ -61,8 +64,58 @@ be passed down to downstream AOT compilation tools like 'ocloc' and 'opencl-aot'
6164
-v Print verbose information
6265
-spirv-dump-device-code=<dir> Directory to dump SPIR-V IR code into
6366
64-
Example
65-
=======
67+
Library Linking
68+
===============
69+
70+
Device bitcode libraries can be packaged into archive libraries (``.a`` files)
71+
using ``llvm-ar`` and linked using the ``-l`` option:
72+
73+
.. code-block:: console
74+
75+
llvm-ar rc libdevice.a func1.bc func2.bc func3.bc
76+
clang-sycl-linker input.bc -l device -L /path/to/libs
77+
78+
The linker supports standard archive library search semantics:
79+
80+
* ``-l <name>`` searches for ``lib<name>.a`` in the directories specified by ``-L``
81+
* ``-l :<exact-name>`` searches for the exact filename in the ``-L`` paths
82+
* Absolute paths can be passed as positional arguments: ``clang-sycl-linker input.bc /path/to/libdevice.a``
83+
84+
By default, archive linking is **lazy** - only archive members (individual ``.bc`` files)
85+
that resolve undefined symbols are extracted and linked. This happens at file
86+
granularity: if any symbol in a ``.bc`` file is needed, all symbols in that file
87+
are included. The linker uses a symbol-driven fixed-point algorithm: it
88+
repeatedly scans archives to extract members that resolve currently undefined
89+
symbols until no more extractions occur.
90+
91+
To force extraction of all archive members regardless of symbol resolution, use
92+
``--whole-archive``:
93+
94+
.. code-block:: console
95+
96+
clang-sycl-linker input.bc --whole-archive -l device --no-whole-archive -l other
97+
98+
The ``-u <symbol>`` option can be used to force a symbol to be undefined, which
99+
can trigger extraction of archive members that define that symbol:
100+
101+
.. code-block:: console
102+
103+
clang-sycl-linker input.bc -u my_init_function -l device
104+
105+
Implementation
106+
--------------
107+
108+
Archive linking in ``clang-sycl-linker`` is implemented using the shared
109+
``llvm::offloading::resolveArchiveMembers()`` API from
110+
``llvm/lib/Frontend/Offloading/ArchiveLinker.cpp``. This same infrastructure is
111+
also used by ``clang-nvlink-wrapper``, ensuring consistent archive linking
112+
semantics across offloading tools.
113+
114+
Examples
115+
========
116+
117+
Basic Usage
118+
-----------
66119

67120
This tool is intended to be invoked when targeting any of the target offloading
68121
toolchains. When the --sycl-link option is passed to the clang driver, the
@@ -74,3 +127,24 @@ generate the final executable.
74127
.. code-block:: console
75128
76129
clang-sycl-linker --triple spirv64 --arch bmg_g21 input.bc
130+
131+
Linking with Device Libraries
132+
------------------------------
133+
134+
To link device bitcode libraries, first package them into archive files:
135+
136+
.. code-block:: console
137+
138+
# Create device library archives
139+
llvm-ar rc libmath.a sin.bc cos.bc tan.bc
140+
llvm-ar rc libutils.a helper1.bc helper2.bc
141+
142+
# Link with lazy loading (only needed members extracted)
143+
clang-sycl-linker --triple spirv64 kernel.bc -l math -l utils -L /path/to/libs -o kernel.spv
144+
145+
# Force all members to be included from libmath.a
146+
clang-sycl-linker --triple spirv64 kernel.bc --whole-archive -l math --no-whole-archive -l utils -L /path/to/libs -o kernel.spv
147+
148+
# Use exact archive filename or absolute path
149+
clang-sycl-linker --triple spirv64 kernel.bc -l :libmath.a -L /path/to/libs -o kernel.spv
150+
clang-sycl-linker --triple spirv64 kernel.bc /absolute/path/libmath.a -o kernel.spv

‎clang/test/OffloadTools/clang-sycl-linker/basic.ll‎

Lines changed: 72 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -22,76 +22,100 @@
2222
;
2323
; Test non-existent input file
2424
; RUN: not clang-sycl-linker %t-missing.bc -o %t.out 2>&1 | FileCheck %s --check-prefix=MISSING
25-
; MISSING: Input file '{{.*}}-missing.bc' does not exist
25+
; MISSING: input file not found: '{{.*}}-missing.bc'
2626
;
2727
; Test the dry run of a simple case to link two input files.
2828
; Test that IMG_SPIRV image kind is set for non-AOT compilation.
2929
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc -o %t/spirv.out 2>&1 \
3030
; RUN: | FileCheck %s --check-prefix=SIMPLE-FO
31-
; SIMPLE-FO: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
31+
; SIMPLE-FO: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
3232
; SIMPLE-FO-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: {{.*}}_0.spv
3333
; SIMPLE-FO-NEXT: sycl-bundle: image kind: spv, triple: spirv64, arch: {{$}}
3434
; SIMPLE-FO-NOT: {{.+}}
3535
;
36-
; Test the dry run of a simple case with device library files specified.
36+
; Test the dry run of a simple case with device library archive specified using --whole-archive.
3737
; RUN: mkdir -p %t/libs
38-
; RUN: touch %t/libs/lib1.bc
39-
; RUN: touch %t/libs/lib2.bc
40-
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc --library-path=%t/libs --bc-library lib1.bc --bc-library lib2.bc -o a.spv 2>&1 \
38+
; RUN: llvm-as %t/lib1.ll -o %t/libs/lib1.bc
39+
; RUN: llvm-as %t/lib2.ll -o %t/libs/lib2.bc
40+
; RUN: rm -f %t/libs/libdevice.a
41+
; RUN: llvm-ar rc %t/libs/libdevice.a %t/libs/lib1.bc %t/libs/lib2.bc
42+
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc %t/input2.bc --library-path=%t/libs --whole-archive -l device -o a.spv 2>&1 \
4143
; RUN: | FileCheck %s --check-prefix=DEVLIBS
42-
; DEVLIBS: link: inputs: {{.*}}.bc libfiles: {{.*}}lib1.bc, {{.*}}lib2.bc output: [[LLVMLINKOUT:.*]].bc
44+
; DEVLIBS: link: inputs: {{.*}}.bc, {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: [[LLVMLINKOUT:.*]].bc
4345
; DEVLIBS-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: a_0.spv
4446
; DEVLIBS-NEXT: sycl-bundle: image kind: spv, triple: spirv64, arch: {{$}}
4547
; DEVLIBS-NOT: {{.+}}
4648
;
47-
; Test -L short form (joined) and --bc-library= joined form.
48-
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L%t/libs --bc-library=lib1.bc -o a.spv 2>&1 \
49+
; Test -L short form (joined) and -l with archive using --whole-archive.
50+
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L%t/libs --whole-archive -l device -o a.spv 2>&1 \
4951
; RUN: | FileCheck %s --check-prefix=DEVLIBS-SHORT
50-
; DEVLIBS-SHORT: link: inputs: {{.*}}.bc libfiles: {{.*}}libs{{[/\\]}}lib1.bc output: {{.*}}.bc
52+
; DEVLIBS-SHORT: link: inputs: {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: {{.*}}.bc
5153
;
52-
; Test that search continues past the first -L when the library is not found there. lib1.bc exists only in %t/libs (the second -L).
54+
; Test that search continues past the first -L when the library is not found there. libdevice.a exists only in %t/libs (the second -L).
5355
; RUN: mkdir -p %t/empty
54-
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/empty -L %t/libs --bc-library lib1.bc -o a.spv 2>&1 \
56+
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/empty -L %t/libs --whole-archive -l device -o a.spv 2>&1 \
5557
; RUN: | FileCheck %s --check-prefix=DEVLIBS-FALLTHROUGH
56-
; DEVLIBS-FALLTHROUGH: link: inputs: {{.*}}.bc libfiles: {{.*}}libs{{[/\\]}}lib1.bc output: {{.*}}.bc
57-
;
58-
; Test that -L paths are searched in order: when the same name exists in multiple -L dirs, the first one wins.
59-
; RUN: mkdir -p %t/libs2
60-
; RUN: touch %t/libs/shadow.bc %t/libs2/shadow.bc
61-
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none %t/input1.bc -L %t/libs2 -L %t/libs --bc-library shadow.bc -o a.spv 2>&1 \
62-
; RUN: | FileCheck %s --check-prefix=DEVLIBS-ORDER
63-
; DEVLIBS-ORDER: link: inputs: {{.*}}.bc libfiles: {{.*}}libs2{{[/\\]}}shadow.bc output: {{.*}}.bc
58+
; DEVLIBS-FALLTHROUGH: link: inputs: {{.*}}.bc, {{.*}}libdevice.a(lib1.bc), {{.*}}libdevice.a(lib2.bc) output: {{.*}}.bc
6459
;
6560
; Test a simple case with a random file (not bitcode) as input.
6661
; RUN: touch %t/dummy.o
6762
; RUN: not clang-sycl-linker %t/dummy.o -o a.spv 2>&1 \
6863
; RUN: | FileCheck %s --check-prefix=FILETYPEERROR
69-
; FILETYPEERROR: Unsupported file type
64+
; FILETYPEERROR: Unsupported file type: '{{.*}}dummy.o'
65+
;
66+
; Test that unsupported file type error includes buffer identifier when found inside an archive.
67+
; Create an archive containing an unsupported file (text file instead of bitcode).
68+
; RUN: echo "not bitcode" > %t/invalid.txt
69+
; RUN: rm -f %t/libinvalid.a
70+
; RUN: llvm-ar rc %t/libinvalid.a %t/invalid.txt
71+
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --whole-archive -l invalid -o a.spv 2>&1 \
72+
; RUN: | FileCheck %s --check-prefix=ARCHIVE-INVALID-MEMBER
73+
; ARCHIVE-INVALID-MEMBER: Unsupported file type: '{{.*}}libinvalid.a(invalid.txt)'
74+
;
75+
; Test mixed archive: valid bitcode member + invalid member.
76+
; The error should clearly identify which member is invalid.
77+
; RUN: rm -f %t/libmixed.a
78+
; RUN: llvm-ar rc %t/libmixed.a %t/libs/lib1.bc %t/invalid.txt
79+
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --whole-archive -l mixed -o a.spv 2>&1 \
80+
; RUN: | FileCheck %s --check-prefix=ARCHIVE-MIXED-INVALID
81+
; ARCHIVE-MIXED-INVALID: Unsupported file type: '{{.*}}libmixed.a(invalid.txt)'
7082
;
7183
; Test to see if device library related errors are emitted.
72-
; RUN: not clang-sycl-linker --dry-run %t/input1.bc %t/input2.bc --library-path=%t/libs --bc-library lib1.bc --bc-library lib2.bc --bc-library lib3.bc -o a.spv 2>&1 \
84+
; RUN: not clang-sycl-linker --dry-run %t/input1.bc %t/input2.bc --library-path=%t/libs -l device -l nonexistent -o a.spv 2>&1 \
7385
; RUN: | FileCheck %s --check-prefix=DEVLIBSERR
74-
; DEVLIBSERR: '{{.*}}lib3.bc' library file not found
86+
; DEVLIBSERR: unable to find library -lnonexistent
7587
;
76-
; Test that there is no implicit CWD search: a bare bitcode name without any -L
88+
; Test that there is no implicit CWD search: a bare library name without any -L
7789
; must fail to resolve, even if a same-named file exists in the CWD.
78-
; RUN: cd %t && not clang-sycl-linker --dry-run input1.bc --bc-library input1.bc -o a.spv 2>&1 \
90+
; RUN: cd %t && not clang-sycl-linker --dry-run input1.bc -l mixed -o a.spv 2>&1 \
7991
; RUN: | FileCheck %s --check-prefix=NO-CWD-SEARCH
80-
; NO-CWD-SEARCH: 'input1.bc' library file not found
92+
; NO-CWD-SEARCH: unable to find library -lmixed
8193
;
8294
; Test that a directory matching the requested name is not accepted as a library:
83-
; %t/libs is a directory created above; resolving --bc-library libs against -L %t
84-
; would otherwise pick it up and fail later with a confusing bitcode-reader error.
85-
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t --bc-library libs -o a.spv 2>&1 \
95+
; %t/libs is a directory created above; resolving -l:libs against -L %t
96+
; would detect it's a directory and error with the filename in the message.
97+
; RUN: not clang-sycl-linker --dry-run %t/input1.bc -L %t -l :libs -o a.spv 2>&1 \
8698
; RUN: | FileCheck %s --check-prefix=NO-DIR-AS-LIB
87-
; NO-DIR-AS-LIB: 'libs' library file not found
99+
; NO-DIR-AS-LIB: '{{.*}}libs': Is a directory
100+
;
101+
; Test that providing only an empty archive results in "No input files could be resolved" error
102+
; RUN: rm -f %t/empty.a
103+
; RUN: llvm-ar rc %t/empty.a
104+
; RUN: not clang-sycl-linker --dry-run --whole-archive %t/empty.a -o a.spv 2>&1 \
105+
; RUN: | FileCheck %s --check-prefix=NO-RESOLVED-INPUT
106+
; NO-RESOLVED-INPUT: No input files could be resolved
107+
;
108+
; Test that providing only a lazy archive with no extracted members results in "No input files could be resolved" error
109+
; RUN: not clang-sycl-linker --dry-run %t/libs/libdevice.a -o a.spv 2>&1 \
110+
; RUN: | FileCheck %s --check-prefix=NO-RESOLVED-LAZY
111+
; NO-RESOLVED-LAZY: No input files could be resolved
88112
;
89113
; Test AOT compilation for an Intel GPU.
90114
; Test that IMG_Object image kind is set for AOT compilation (Intel GPU).
91115
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none -arch=bmg_g21 %t/input1.bc %t/input2.bc -o %t/aot-gpu.out 2>&1 \
92116
; RUN: --ocloc-options="-a -b" \
93117
; RUN: | FileCheck %s --check-prefix=AOT-INTEL-GPU
94-
; AOT-INTEL-GPU: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
118+
; AOT-INTEL-GPU: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
95119
; AOT-INTEL-GPU-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: [[SPIRVTRANSLATIONOUT:.*]]_0.spv
96120
; AOT-INTEL-GPU-NEXT: "{{.*}}ocloc{{.*}}" {{.*}}-device bmg_g21 -a -b {{.*}}-output [[SPIRVTRANSLATIONOUT]]_0.out -file [[SPIRVTRANSLATIONOUT]]_0.spv
97121
; AOT-INTEL-GPU-NEXT: sycl-bundle: image kind: o, triple: spirv64, arch: bmg_g21
@@ -102,7 +126,7 @@
102126
; RUN: clang-sycl-linker --dry-run -v --module-split-mode=none -arch=graniterapids %t/input1.bc %t/input2.bc -o %t/aot-cpu.out 2>&1 \
103127
; RUN: --opencl-aot-options="-a -b" \
104128
; RUN: | FileCheck %s --check-prefix=AOT-INTEL-CPU
105-
; AOT-INTEL-CPU: link: inputs: {{.*}}.bc, {{.*}}.bc libfiles: output: [[LLVMLINKOUT:.*]].bc
129+
; AOT-INTEL-CPU: link: inputs: {{.*}}.bc, {{.*}}.bc output: [[LLVMLINKOUT:.*]].bc
106130
; AOT-INTEL-CPU-NEXT: LLVM backend: input: [[LLVMLINKOUT]].bc, output: [[SPIRVTRANSLATIONOUT:.*]]_0.spv
107131
; AOT-INTEL-CPU-NEXT: "{{.*}}opencl-aot{{.*}}" {{.*}}--device=cpu -a -b {{.*}}-o [[SPIRVTRANSLATIONOUT]]_0.out [[SPIRVTRANSLATIONOUT]]_0.spv
108132
; AOT-INTEL-CPU-NEXT: sycl-bundle: image kind: o, triple: spirv64, arch: graniterapids
@@ -164,3 +188,19 @@ target triple = "spirv64"
164188
define spir_func i32 @helper() {
165189
ret i32 0
166190
}
191+
192+
;--- lib1.ll
193+
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
194+
target triple = "spirv64"
195+
196+
define spir_func i32 @lib1_func() {
197+
ret i32 1
198+
}
199+
200+
;--- lib2.ll
201+
target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n8:16:32:64-G1"
202+
target triple = "spirv64"
203+
204+
define spir_func i32 @lib2_func() {
205+
ret i32 2
206+
}

0 commit comments

Comments
 (0)