Merged
Conversation
Member
Author
|
Currently (locally for me) this errors with DetailsThis might be related to the dynamic linking with the |
mdboom
reviewed
Oct 5, 2018
.circleci/config.yml
Outdated
| pip install pytest pytest-xdist pytest-instafail selenium PyYAML flake8 | ||
|
|
||
| # Download BLAS/LAPACK | ||
| git clone https://github.com/adrianbg/CLAPACK-WA.git packages/scipy/CLAPACK-WA |
Collaborator
There was a problem hiding this comment.
Eventually, this should probably happen as part of the Scipy package build, or maybe its own package that Scipy is dependent on.
mdboom
reviewed
Oct 5, 2018
emsdk/patches/num_params.patch
Outdated
| // can't just detect this automatically in the module we see.) | ||
| -static const int NUM_PARAMS = 15; | ||
| +static const int NUM_PARAMS = 32; | ||
| +static const int NUM_PARAMS = 37; |
Collaborator
There was a problem hiding this comment.
For CI, we'll need to bump the emsdk cache value in .circleci/config.yaml
Collaborator
I've worked around this in other packages where this happens by forcing the use of malloc somewhere that will definitely get called, like in the |
Member
Author
|
Thanks a lot for the suggestions @mdboom -- great that you have already encountered that malloc issue ! |
Member
Author
|
The malloc fix did help. Now I am running into errors of the form, (see more detailed log below) DetailsHave you encountered this before @mdboom ? #include <string>If not I'll ask about it on emscripten channels as my attempts to fix it so far were not successful. |
Collaborator
|
Sorry, I haven't run into that. It seems like something is incomplete in the C++ stdlib, but I'm not sure... |
Member
Author
|
Thanks for the confirmation @mdboom ! |
rth
force-pushed
the
scipy
branch
2 times, most recently
from
762c49d to
505a99c
Compare
Member
Author
|
So the status on this, is that by aggressively patching to skip all compilations that fail, we currently have a scipy build that somewhat passes, out of a total of 69 DetailsDetailsFor future reference the above results were obtained with the script below, Detailsimport scipy as sp
import os
base_dir = os.path.dirname(sp.__file__)
for (dirpath, dirnames, filenames) in os.walk(base_dir):
for path in filenames:
if path.endswith('.so'):
rel_path = os.path.relpath(dirpath, base_dir)
print(os.path.join(rel_path, path))The resulting The major directions that need more work are,
|
rth
force-pushed
the
scipy
branch
3 times, most recently
from
2a088d8 to
e47d762
Compare
rth
commented
Oct 15, 2018
| - ./emsdk/emsdk | ||
| - ~/.ccache | ||
| key: v1-emsdk-{{ checksum "emsdk/Makefile" }}-v8-{{ .BuildNum }} | ||
| key: v1-emsdk-{{ checksum "emsdk/Makefile" }}-v11-{{ .BuildNum }} |
Member
Author
There was a problem hiding this comment.
This shift is because it took a few iterations of rebuilding emsdk to find a sufficiently large NUM_PARAMS: now it's 61 (as opposed to 32 before) as it's required by scipy's fitpack (~47) and odrpack (61). I'm not sure if it can have adverse effect (e.g. on performance).
rth
force-pushed
the
scipy
branch
3 times, most recently
from
f358d4a to
6c8dcee
Compare
Member
Author
|
The linking of Fortran files was fixed, and global cpp namespace issue somehow disappeared when moving to dockerized builds (happy enough with that and I have not investigated further -- probably something different in my environment). So now we have around ~46 Regarding last major point remaining -- linking to BLAS/LAPACK I was hoping @mdboom (or possibly @jakirkham) that you would have some suggestions. As suggested earlier, I'm using the CLAPACK setup with the Makesfiles patched to work for Webassembly https://github.com/rth/CLAPACK-WA. Because it uses a Regarding linking itself, I have read https://github.com/kripken/emscripten/wiki/Linking and have tried, Linking dynamicallyThis setup is contained in https://github.com/rth/pyodide/tree/scipy-dynamic-link-blas which includes 1 commit in addition to this PR. The CLAPACK needs some adaptation to produce .so. For host this works fine, for the target I'm still not sure how this is expected to work: I get however then the linking during scipy compilation fails, with, I'm not sure if this error is due to missing symbols, or is something else. I would have expected missing symbols (e.g. because blas/lapack wasn't properly linked) would trigger an error at runtime. The and the relative path is correct. FWIW, I did try to generate While renaming Even assuming that this step succeeds, I'm not sure how to include the resulting Linking staticallyNot convinced this makes much sense for BLAS/LAPACK, but since I had trouble with the previous approach I though it was worth a try. In and indeed that symbol is redundantly defined in |
Collaborator
Why do you need a version for the host environment? Is that just to support You are right that when statically linking things together into a shared object, emscripten can't remove much stuff (because it doesn't know what the application it's eventually linked into will use). However, I don't know how the linking from a Scipy extension module to a dynamically linked blas/lapack should work. I assume in the native case, it's using An alternative not explored above might be to statically link blas and lapack into the main executable (this is how libpng and libfreetype are currently handled, even though they are used only by the optional matplotlib package). You could look at how emscripten handles the libraries it "ships" and maybe implement that? For these errors: I've never seen this before, but maybe someone in the emscripten project can at least point in the right direction? |
Member
Author
Yes, that was the reason.
I was also wondering if we really needed it. Thanks for the suggestion, I think I can also also just strip the
Hah, I have not though about that. Well I guess it did raise an error at run time that some symbols are missing, it could have been be possible to dlopen some specific
Interesting idea. Thanks for the suggestion, and for the confirmation that static linking may be the way to go. I'll try to ask some questions on the emscripten mailing list as well. |
Merged
rth
force-pushed
the
scipy
branch
2 times, most recently
from
500d145 to
2a6e042
Compare
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 15, 2018
Currently Emscripten allows to create shared libraries (DSO) and link them to main module. However a shared library itself cannot be linked to another shared library. The lack of support for DSO -> DSO linking becomes problematic in cases when there are several shared libraries that all need to use another should-be shared functionality, while linking that should-be shared functionality to main module is not an option for size reasons. My particular use-case is SciPy support for Pyodide: pyodide/pyodide#211, pyodide/pyodide#240 where several of `*.so` scipy modules need to link to LAPACK. If we link to LAPACK statically from all those `*.so` - it just blows up compiled size pyodide/pyodide#211 (comment) and if we link in LAPACK statically to main module, the main module size is also increased ~2x, which is not an option, since LAPACK functionality is not needed by every Pyodide user. This way we are here to add support for DSO -> DSO linking: 1. similarly to how it is already working for main module -> side module linking, when building a side module it is now possible to specify -s RUNTIME_LINKED_LIBS=[...] with list of shared libraries that side module needs to link to. 2. to store that information, for asm.js, similarly to how it is currently handled for main module (which always has js part), we transform RUNTIME_LINKED_LIBS to libModule.dynamicLibraries = [...] (see src/preamble_sharedlib.js) 3. for wasm module, in order to store the information about to which libraries a module links, we could in theory use "module" attribute in wasm imports. However currently emscripten almost always uses just "env" for that "module" attribute, e.g. (import "env" "abortStackOverflow" (func $fimport$0 (param i32))) (import "env" "_ffunc1" (func $fimport$1)) ... and this way we have to embed the information about required libraries for the dynamic linker somewhere else. What I came up with is to extend "dylink" section with information about which shared libraries a shared library needs. This is similar to DT_NEEDED entries in ELF. (see tools/shared.py) 4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information: - for asm.js, after loading a libModule, we check libModule.dynamicLibraries and post-load them recursively. (it would be better to load needed modules before the libModule, but for js we currently have to first eval whole libModule's code to be able to read .dynamicLibraries) - for wasm the needed libraries are loaded before the wasm module in question is instantiated. (see changes to loadWebAssemblyModule for details) 5. since we also have to teach dlopen() to handle needed libraries, and since dlopen was already duplicating loadDynamicLibrary() code in many ways, instead of adding more duplication, dlopen is now reworked to use loadDynamicLibrary itself. This moves functionality to keep track of loaded DSO, their handles, refcounts, etc into the dynamic linker itself, with loadDynamicLibrary now accepting various flags (global/nodelete) to handle e.g. RTLD_LOCAL/RTLD_GLOBAL and RTLD_NODELETE dlopen cases (RTLD_NODELETE semantic is needed for initially-linked-in libraries). Also, since dlopen was using FS to read libraries, and loadDynamicLibrary was previously using Module['read'] and friends, loadDynamicLibrary now also accepts fs interface, which if provided, is used as FS-like interface to load library data, and if not - native loading capabilities of the environment are still used. Another aspect related to deduplication is that loadDynamicLibrary now also uses preloaded/precompiled wasm modules, that were previously only used by dlopen (see a5866a5 "Add preload plugin to compile wasm side modules async (emscripten-core#6663)"). (see changes to dlopen and loadDynamicLibrary) 6. The functionality to asynchronously load dynamic libraries is also integrated into loadDynamicLibrary. Libraries were asynchronously preloaded for the case when Module['readBinary'] is absent (browser, see 3446d2a "preload wasm dynamic libraries when we can't load them synchronously"). Since this codepath was also needed to be taught of DSO -> DSO dependency, the most straightforward thing to do was to teach loadDynamicLibrary to do its work asynchronously (under flag) and to switch the preloading to use loadDynamicLibrary(..., {loadAsync: true}) (see changes to src/preamble.js and loadDynamicLibrary) 7. A test is added for verifying linking/dlopening a DSO with other needed library. browser.test_dynamic_link is also amended to verify linking to DSO with dependencies. With the patch I've made sure that all core tests (including test_dylink_* and test_dlfcn_*) are passing for asm{0,1,2} and binaryen{0,1,2}. However since I cannot get full browser tests to pass even on pristine incoming (1.38.19-2-g77246e0c1 as of today; there are many failures for both Firefox 63.0.1 and Chromium 70.0.3538.67), I did not tried to verify full browser tests with my patch. Bit I've made sure that browser.test_preload_module browser.test_dynamic_link are passing. "other" kind of tests also do not pass on pristine incoming for me. This way I did not tried to verify "other" with my patch. Thanks beforehand, Kirill P.S. This is my first time I do anything with WebAssembly/Emscripten, and only a second time with JavaScript, so please forgive me if I missed something. P.P.S. I can split the patch into smaller steps, if it will help review. /cc @kripken, @juj, @sbc100, @max99x, @junjihashimoto, @mdboom, @rth
Member
Author
|
I rebased to fix a merge conflcit. Now that iodide-project/iodide#1122 is merged, @mdboom please let me know if there is anything else I need to to get this accepted. |
Member
Author
|
BTW, after rebase, I get a |
Collaborator
|
This is now blocked by iodide-project/iodide#1178, but I don't think there's anything remaining in this PR. I agree, the chrome failure seems unrelated. Probably need to tweak |
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 16, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 16, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 18, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 20, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 21, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 27, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 29, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Nov 29, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
Member
Author
|
Gentle ping @mdboom :) Looks like the deployment PR was merged, and I think it would help to have this in master for basing possible future improvements on top of it. |
Member
Author
|
Thanks a lot @mdboom ! |
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 3, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 5, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 5, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 5, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 6, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
navytux
added a commit
to navytux/emscripten
that referenced
this pull request
Dec 8, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
kripken
pushed a commit
to emscripten-core/emscripten
that referenced
this pull request
Dec 9, 2018
Currently Emscripten allows to create shared libraries (DSO) and link
them to main module. However a shared library itself cannot be linked to
another shared library.
The lack of support for DSO -> DSO linking becomes problematic in cases when
there are several shared libraries that all need to use another should-be
shared functionality, while linking that should-be shared functionality to main
module is not an option for size reasons. My particular use-case is SciPy
support for Pyodide:
pyodide/pyodide#211,
pyodide/pyodide#240
where several of `*.so` scipy modules need to link to LAPACK. If we link to
LAPACK statically from all those `*.so` - it just blows up compiled size
pyodide/pyodide#211 (comment)
and if we link in LAPACK statically to main module, the main module size is
also increased ~2x, which is not an option, since LAPACK functionality is not
needed by every Pyodide user.
This way we are here to add support for DSO -> DSO linking:
1. similarly to how it is already working for main module -> side module
linking, when building a side module it is now possible to specify
-s RUNTIME_LINKED_LIBS=[...]
with list of shared libraries that side module needs to link to.
2. to store that information, for asm.js, similarly to how it is currently
handled for main module (which always has js part), we transform
RUNTIME_LINKED_LIBS to
libModule.dynamicLibraries = [...]
(see src/preamble_sharedlib.js)
3. for wasm module, in order to store the information about to which libraries
a module links, we could in theory use "module" attribute in wasm imports.
However currently emscripten almost always uses just "env" for that "module"
attribute, e.g.
(import "env" "abortStackOverflow" (func $fimport$0 (param i32)))
(import "env" "_ffunc1" (func $fimport$1))
...
and this way we have to embed the information about required libraries for
the dynamic linker somewhere else.
What I came up with is to extend "dylink" section with information about
which shared libraries a shared library needs. This is similar to DT_NEEDED
entries in ELF.
(see tools/shared.py)
4. then, the dynamic linker (loadDynamicLibrary) is reworked to handle that information:
- for asm.js, after loading a libModule, we check libModule.dynamicLibraries
and post-load them recursively. (it would be better to load needed modules
before the libModule, but for js we currently have to first eval whole
libModule's code to be able to read .dynamicLibraries)
- for wasm the needed libraries are loaded before the wasm module in
question is instantiated.
(see changes to loadWebAssemblyModule for details)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a continuation of #75 that eventually aims to fix partially #72
This still needs a lot of works, so this PR mostly aims to facilitate discussion of intermediary results.
As outlined in #184,
f2cdoes not work for f90 that is used in scipy 1.1. Here I have reverted to scipy 0.17.1 (from 2 years ago) which is a last known version to be f77 only AFAIK. Then the issue is that.tar.gzfrom PyPi includes Cythonized files with an old Cython version that don't work on python 3.7, so here we download the sources from github and cythonize them manually.I can confirm that CLAPACK-WA builds, here with,
and produces
lapack_WA.bcandblas_WA.bcfiles, however I'm not yet sure how to makenumpy.distutilsdetect it properly during scipy installation. For now I removed allsetup.pythat explicitly require LAPACK to see what parts of scipy we can build without it.In terms of packaging, some of what I have done here is a not very clean -- it is only meant as a temporary solution until we have something that actually builds.
Comments or suggestions along the way would be very welcome.