The problem seems to be that it expects LAPACK to have a function called sgeqrfp but our LAPACK doesn't (we have sgeqrf without the p...).
I wonder if it's possible to just delete sgeqrfp from the various wrapper generation files.

Okay the problem here is related to incompatibilities between LAPACK versions. It is a bit hard to tell but it seems like sgeqrfp was added in lapack v3.3.0 http://www.netlib.org/lapack/lapack-3.3.0.html, whereas we are using CLAPACK 3.2.1
It seems like it might become an issue for us that CLAPACK is not keeping up with LAPACK releases. LAPACK is up to version 3.10 released a year ago, but CLAPACK is on 3.2 which was released in 2008.

To find exports from clapack_all.so

wasm-objdump clapack_all.so -d | sed -n 's/.*func.*<\(.*\)>:/\1/p' | sort -u > clapack_exports.txt

To find what scipy tries to import from clapack:

~/wabt-1.0.24/bin/wasm-objdump _flapack.so -d | sed -E -n 's/.*global.get [0-9]* <([a-z]*_?)>/\1/p' | sort -u > symbols.txt

there are two unrelated symbols in this second list. Anyways, I take the difference of these two sets and find the missing LAPACK functions:

cgemqrt cgeqrfp cgeqrt cuncsd csyconv csyconvf csyconvf_rook ctpmqrt ctpqrt 
dgemqrt dgeqrfp dgeqrt dorcsd dsyconv dsyconvf dsyconvf_rook dtpmqrt dtpqrt 
sgemqrt sgeqrfp sgeqrt sorcsd ssyconv ssyconvf ssyconvf_rook stpmqrt stpqrt
zgemqrt zgeqrfp zgeqrt zuncsd zsyconv zsyconvf zsyconvf_rook ztpmqrt ztpqrt

These can be copied from the LAPACK tree into our source tree and then I append them to the end of scipy/_build_utils/src/wrap_dummy_g77_abi.f. However, cuncsd, dorcsd, sorcsd and zuncsd are not fortran 77 compliant, so these need to be stubbed out with empty definitions. I was unable to figure out even how to stub them in fortran so I stub them in the generated C.

Build succeeded and tests pass:

scipy/test_scipy.py::scipy_linalg[firefox] PASSED
scipy/test_scipy.py::brentq[firefox] PASSED
scipy/test_scipy.py::scipy_linalg[chrome] PASSED
scipy/test_scipy.py::brentq[chrome] PASSED
scipy/test_scipy.py::scipy_linalg[numpy] PASSED
scipy/test_scipy.py::brentq[numpy] PASSED

but

FAILED test_packages_common.py::import[firefox-scipy]
FAILED test_packages_common.py::import[chrome-scipy]
FAILED test_packages_common.py::import[node-scipy]

Okay, all tests pass now!

Fix dynamic linking errors with _flapack.so

Okay, all tests pass now!

Perfect, thanks again for working on this! +1 for merging once CI pass. It's strange that it was only failing in Chrome.

It's strange that it was only failing in Chrome.

Yeah it's very strange. And only in chrome in CI. By rerunning with ssh, I was able to replicate it and read the selenium logs which gave clear diagnostic information. I was able to replicate the bug in Firefox on the CI server via ssh with just the right setup, but it was generally very sensitive to tiny changes.

Also, it would be great if we could run the full test suite for scipy once in a while.

There's a pretty significant number of fatal errors and test failures in the test suite. Currently it makes it about 20% through the test suite before hitting a fatal error. I'm running it with the following code:

pyodide._module.packages["scipy-tests"] = {"name":"scipy-tests","version":"0.17.1","depends":["scipy"],"imports":[]}
await pyodide.loadPackage(["scipy-tests", "nose","pytest"]);
pyodide.runPython(`
from scipy import test
test(extra_argv=["--continue-on-collection-errors", "-k", "not multithreading and not TestNQuad", '-vv'])
`);

Currently I am running into trouble that scipy.linalg has a large number of functions that expect x, &offx and are called like x+offx. There's also a noticeable fraction of functions that don't fail but return completely wrong answers.

Perhaps it would be a good idea to merge this and fix more of these problems in followup PRs.

Currently it makes it about 20% through the test suite before hitting a fatal error.

That's already better than on main, where I think there is a fatal error after 3% of the test suite. So what I had to do back in 2018 to work around it was to run submodules one by one, e.g.

import pytest
pytest.main(["--pyargs", "scipy.linalg", "--continue-on-collection-errors", "-k", "not multithreading and not TestNQuad", '-vv'])

though it's very time-consuming to exclude the few tests that produce a fatal error. An ideal situation would have been to have something like pytest-xdist that runs tests in a webworker pytest-dev/pytest-xdist#336 so that if there is a fatal error in a worker, it can mark the test as crashed, spawn a new worker and continue on. But well, that's another side project )

Something like this would be necessary to reliably run the test suite in CI until the end as well. #69

Currently it makes it about 20% through the test suite before hitting a fatal error.

What was the test success rate roughly for those 20% ?

There's also a noticeable fraction of functions that don't fail but return completely wrong answers.

That's worrisome. Do you have the pytest logs -- in which submodules does this happen?

Perhaps it would be a good idea to merge this and fix more of these problems in followup PRs.

I agree there are already many things in this PR and it would be better to work on the test suite in a follow-up PR. The question is should we include it in the 0.19 release or not. If we do, at least the release notes should mention that scipy still has a number of known issues.

What was the test success rate roughly for those 20%?

So far I fixed the fatal errors in integrate and interpolate. The current status of the test suite is:

The question is should we include it in the 0.19 release or not. If we do, at least the release notes should mention that scipy still has a number of known issues.

I think it's good to include in the 0.19 release, especially if the old test suite had a similar number of failures. We could release fixes in 0.19.1 for instance.

Merging, as I would like to see if we can update scikit-image with this #2005. For the changelog, we would need to review it in any case before the release. Thanks.