Skip to content

Commit 6410be8

Browse files
navytuxkripken
navytux
authored and
kripken
committed
ldso: Teach dynamic linking to handle library -> library dependencies
Currently Emscripten allows to create shared libraries (DSO) and link them to main module. However a shared library itself cannot be linked to another shared library. The lack of support for DSO -> DSO linking becomes problematic in cases when there are several shared libraries that all need to use another should-be shared functionality, while linking that should-be shared functionality to main module is not an option for size reasons. My particular use-case is SciPy support for Pyodide: pyodide/pyodide#211, pyodide/pyodide#240 where several of `*.so` scipy modules need to link to LAPACK. If we link to LAPACK statically from all those `*.so` - it just blows up compiled size pyodide/pyodide#211 (comment) and if we link in LAPACK statically to main module, the main module size is also increased ~2x, which is not an option, since LAPACK functionality is not needed by every Pyodide user. This way we are here to add support for DSO -> DSO linking: 1. similarly to how it is already working for main module -> side module linking, when building a side module it is now possible to specify -s RUNTIME_LINKED_LIBS=[...] with list of shared libraries that side module needs to link to. 2. to store that information, for asm.js, similarly to how it is currently handled for main module (which always has js part), we transform RUNTIME_LINKED_LIBS to libModule.dynamicLibraries = [...] (see src/preamble_sharedlib.js) 3. for wasm module, in order to store the information about to which libraries a module links, we could in theory use "module" attribute in wasm imports. However currently emscripten almost always uses just "env" for that "module" attribute, e.g. (import "env" "abortStackOverflow" (func $fimport$0 (param i32))) (import "env" "_ffunc1" (func $fimport$1)) ... and this way we have to embed the information about required libraries for the dynamic linker somewhere else. What I came up with is to extend "dylink" section with information about which shared libraries a shared library needs. This is similar to DT_NEEDED entries in ELF. (see tools/shared.py) 4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information: - for asm.js, after loading a libModule, we check libModule.dynamicLibraries and post-load them recursively. (it would be better to load needed modules before the libModule, but for js we currently have to first eval whole libModule's code to be able to read .dynamicLibraries) - for wasm the needed libraries are loaded before the wasm module in question is instantiated. (see changes to loadWebAssemblyModule for details)
1 parent 8ec4b4a commit 6410be8

File tree

9 files changed

+416
-143
lines changed

9 files changed

+416
-143
lines changed

‎emcc.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2602,7 +2602,7 @@ def do_binaryen(target, asm_target, options, memfile, wasm_binary_target,
26022602
shared.Building.eval_ctors(final, wasm_binary_target, binaryen_bin, debug_info=debug_info)
26032603
# after generating the wasm, do some final operations
26042604
if shared.Settings.SIDE_MODULE:
2605-
wso = shared.WebAssembly.make_shared_library(final, wasm_binary_target)
2605+
wso = shared.WebAssembly.make_shared_library(final, wasm_binary_target, shared.Settings.RUNTIME_LINKED_LIBS)
26062606
# replace the wasm binary output with the dynamic library. TODO: use a specific suffix for such files?
26072607
shutil.move(wso, wasm_binary_target)
26082608
if not shared.Settings.WASM_BACKEND and not DEBUG:

‎src/library_browser.js‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,7 @@ var LibraryBrowser = {
244244
// promises to run in series.
245245
this['asyncWasmLoadPromise'] = this['asyncWasmLoadPromise'].then(
246246
function() {
247-
return loadWebAssemblyModule(byteArray, true);
247+
return loadWebAssemblyModule(byteArray, {loadAsync: true, nodelete: true});
248248
}).then(
249249
function(module) {
250250
Module['preloadedWasm'][name] = module;

‎src/preamble_sharedlib.js‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,19 @@
44
// Runtime essentials
55
//========================================
66

7+
{{{
8+
(function() {
9+
// add in RUNTIME_LINKED_LIBS, if provided
10+
//
11+
// for side module we only set Module.dynamicLibraries - and loading them
12+
// will be handled by dynamic linker runtime in the main module.
13+
if (RUNTIME_LINKED_LIBS.length > 0) {
14+
return "if (!Module['dynamicLibraries']) Module['dynamicLibraries'] = [];\n" +
15+
"Module['dynamicLibraries'] = " + JSON.stringify(RUNTIME_LINKED_LIBS) + ".concat(Module['dynamicLibraries']);\n";
16+
}
17+
return '';
18+
})()
19+
}}}
20+
721
// === Body ===
822

‎src/settings.js‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -683,9 +683,8 @@ var MAIN_MODULE = 0;
683683
// Corresponds to MAIN_MODULE (also supports modes 1 and 2)
684684
var SIDE_MODULE = 0;
685685

686-
// If this is a main module (MAIN_MODULE == 1), then
687-
// we will link these at runtime. They must have been built with
688-
// SIDE_MODULE == 1.
686+
// If this is a shared object (MAIN_MODULE == 1 || SIDE_MODULE == 1), then we
687+
// will link these at runtime. They must have been built with SIDE_MODULE == 1.
689688
var RUNTIME_LINKED_LIBS = [];
690689

691690
// If set to 1, this is a worker library, a special kind of library that is run

0 commit comments

Comments
 (0)