Switch to using a fork of f2c and remove most of _f2c_fixes.py by hoodmane · Pull Request #4920 · pyodide/pyodide

hoodmane · 2024-07-06T22:14:36Z

I made https://github.com/hoodmane/f2c/ which has some improvements that make it work much better for us including:

support for the recursive keyword
support for variable length arrays that are not arguments in recursive functions
/COMMON/ blocks are labeled with __attribute__((weak))
automatically omits ftn_len arguments from calls to clapack

This allows us to delete most of _f2c_fixes.py.

Thanks to @ilayn for writing scipy/scipy#20558, id_dist was going to be a huge pain to deal with.

cc @rgommers @agriyakhetarpal

Add a CHANGELOG entry
Add / update tests
Add new / update outdated documentation

for more information, see https://pre-commit.ci

hoodmane · 2024-07-06T22:17:01Z

Also, I still am amazed every time I build scipy in 5 minutes. Meson is so great.

ryanking13

Cool! I've skimmed through your f2c modifications, and TBH I don't understand it very much. Still, I think it is a great improvement if we can remove _f2c_fixes.py which is one of our pain point in pyodide-build.

Please ping me again when the PR is ready, I would like to review again.

packages/scipy/patches/0008-MAINT-linalg-Remove-id_dist-Fortran-files.patch

pyodide-build/pyodide_build/pywasmcross.py

ryanking13 · 2024-07-07T11:29:23Z

pyodide-build/pyodide_build/pypabuild.py

I believe passing F2C_PATH in the cmd line would work without setting it here, then we won't need to modify pyodide-build.

I tried removing this and it didn't work so I think it's needed. Is there something specific you're suggesting?

Hmm, that's strange. Let me check locally.

I opened a PR in your fork: hoodmane#9. The build passes while tests fail.

Looks like you didn't make the tests fail any worse than they already do though. Thanks!

packages/scipy/meta.yaml

ilayn · 2024-07-07T12:03:15Z

Thanks to @ilayn for writing scipy/scipy#20558, id_dist was going to be a huge pain to deal with.

My pleasure. Please let me know if you encounter any bugs because these translations make it extremely easy to make mistakes and I am very good at making those mistakes.

hoodmane · 2024-07-09T12:37:53Z

@lesteve could you run the scipy tests against this?

lesteve · 2024-07-09T15:11:14Z

It seems like there are Pyodide fatal errors on this PR mostly around scipy.sparse, for example when running the equivalent of pytest --pyargs scipy.sparse.tests.test_array_api, there is a Pyodide fatal error (the last output I get is about test_eigs[bsr] so maybe this is the culprit?)

The cause of the fatal error was:
RuntimeError: null function or function signature mismatch
    at wasm://wasm/000facae:wasm-function[187]:0x8cdf
    at wasm://wasm/0003e6d6:wasm-function[107]:0xbfa0
    at _PyEM_TrampolineCall_JS (/home/lesteve/dev/scipy-tests-pyodide/node_modules/pyodide/pyodide.asm.js:1735:83)
    at pyodide.asm.wasm._PyObject_MakeTpCall (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[1136]:0x1c2631)
    at pyodide.asm.wasm.PyObject_Vectorcall (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[1145]:0x1c2e37)
    at pyodide.asm.wasm._PyEval_EvalFrameDefault (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[3411]:0x29f694)
    at pyodide.asm.wasm._PyEval_Vector (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[3412]:0x2a409a)
    at pyodide.asm.wasm._PyFunction_Vectorcall (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[1150]:0x1c31e0)
    at pyodide.asm.wasm._PyObject_FastCallDictTstate (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[1137]:0x1c2748)
    at pyodide.asm.wasm._PyObject_Call_Prepend (wasm://wasm/pyodide.asm.wasm-029e027e:wasm-function[1154]:0x1c3405) {
  pyodide_fatal_error: true
}

The output from running all the scipy tests using https://github.com/lesteve/scipy-tests-pyodide:

--------------------------------------------------------------------------------
Grouped by category
--------------------------------------------------------------------------------
category fatal error or timeout (4 modules)
    scipy.sparse.linalg._eigen.arpack.tests
    scipy.sparse.linalg._eigen.lobpcg.tests
    scipy.sparse.linalg._eigen.tests
    scipy.sparse.tests
category passed (28 modules)
    scipy.cluster.tests
    scipy.constants.tests
    scipy.fftpack.tests
    scipy.fft._pocketfft.tests
    scipy.fft.tests
    scipy.integrate._ivp.tests
    scipy.integrate.tests
    scipy.interpolate.tests
    scipy.io.arff.tests
    scipy.io._harwell_boeing.tests
    scipy.io.matlab.tests
    scipy.io.tests
    scipy._lib.tests
    scipy.linalg.tests
    scipy.misc.tests
    scipy.ndimage.tests
    scipy.odr.tests
    scipy.optimize.tests
    scipy.optimize._trustregion_constr.tests
    scipy.signal.tests
    scipy.sparse.csgraph.tests
    scipy.sparse.linalg._dsolve.tests
    scipy.sparse.linalg._isolve.tests
    scipy.sparse.linalg.tests
    scipy.spatial.tests
    scipy.spatial.transform.tests
    scipy.special.tests
    scipy.stats.tests

--------------------------------------------------------------------------------
Unexpected test results
--------------------------------------------------------------------------------
scipy.sparse.linalg._eigen.arpack.tests result expected in ['passed'], got 'fatal error or timeout' instead
scipy.sparse.linalg._eigen.lobpcg.tests result expected in ['passed'], got 'fatal error or timeout' instead
scipy.sparse.linalg._eigen.tests result expected in ['passed'], got 'fatal error or timeout' instead
scipy.sparse.tests result expected in ['passed'], got 'fatal error or timeout' instead

hoodmane · 2024-07-09T15:32:05Z

Thanks @lesteve. I fixed one problem with scipy.sparse, how does it look now?

lesteve · 2024-07-09T15:47:59Z

Yep the scipy.sparse.tests works now, but there are still some Pyodide fatal errors in the equivalent of pytest --pyargs scipy.sparse.linalg

lesteve · 2024-07-09T19:04:07Z

The remaining issue seems related to cneupd_, call_indirect is calling it with 25 i32 arguments but its signature has 26 i32 arguments.

hoodmane · 2024-07-10T08:18:35Z

Great, thanks @lesteve!

ryanking13

Codewise looks good to me. Thanks for improving this!

agriyakhetarpal · 2024-07-10T14:28:50Z

I'll rebase #4719 as soon as this is merged, it looks like there are a lot of simplifications, thanks!

for more information, see https://pre-commit.ci

ilayn · 2024-07-16T09:26:37Z

In the meantime, we merged MINPACK in scipy/scipy#21131; FYI if that is causing any issues and QUADPACK is next in line, then ARPACK hopefully all in v1.15 timeline.

hoodmane · 2024-07-16T11:40:18Z

We have special workarounds for ARPACK, PROPACK, SUPERLU, fitpack, blas and lapack. Also, there are a bunch of test failures in different parts of scipy but I don't really know were or why.

hoodmane · 2024-07-16T11:41:10Z

id_dist was by far the worst though.

@ilayn

…de#4920) I made https://github.com/hoodmane/f2c/ which has some improvements that make it work much better for us including: 1. support for the `recursive` keyword 2. support for variable length arrays that are not arguments in recursive functions 3. `/COMMON/` blocks are labeled with `__attribute__((weak))` 4. automatically omits `ftn_len` arguments from calls to clapack This allows us to delete most of `_f2c_fixes.py`. Thanks to @ilayn for writing scipy/scipy#20558, `id_dist` was going to be a huge pain to deal with.

hoodmane and others added 2 commits July 7, 2024 00:08

Switch to using a fork of f2c and remove most of _f2c_fixes.py

ea9acee

[pre-commit.ci] auto fixes from pre-commit.com hooks

2a1c399

for more information, see https://pre-commit.ci

Avoid building gram.c

86eb4d7

ryanking13 reviewed Jul 7, 2024

View reviewed changes

hoodmane and others added 6 commits July 7, 2024 19:30

Clean up more

2efa8fd

Fix f2c build rule again

a639ea8

Remove f2c from executable requirements

302aff0

More cleanup in f2c_fixes

dc478fc

Trigger ci

c7987ae

Remove F2C_PATH from pyodide-build

c07e0f1

Fix *eupd

ce9ff7b

ryanking13 approved these changes Jul 10, 2024

View reviewed changes

agriyakhetarpal mentioned this pull request Jul 10, 2024

Update to NumPy v2 and bump versions for several dependent packages #4925

Merged

2 tasks

lesteve mentioned this pull request Jul 15, 2024

Run scipy tests as part of the Github Action CI #4935

Merged

3 tasks

hoodmane and others added 2 commits July 16, 2024 11:08

More fixes

03666d4

[pre-commit.ci] auto fixes from pre-commit.com hooks

af145ff

for more information, see https://pre-commit.ci

hoodmane merged commit 5ff2f8e into pyodide:main Jul 16, 2024

hoodmane deleted the f2c-fork branch July 16, 2024 11:41

ryanking13 added a commit to ryanking13/pyodide-build that referenced this pull request Jul 17, 2024

Port pyodide/pyodide#4920

db8d47a

agriyakhetarpal mentioned this pull request Jul 30, 2024

SciPy v1.13.0 #4719

Merged

8 tasks

agriyakhetarpal mentioned this pull request Sep 17, 2024

ENH: optimize.minimize: Expose LINCOA, BOBYQA, NEWUOA, and UOBYQA scipy/scipy#21554

Closed

Uh oh!

Conversation

hoodmane commented Jul 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoodmane commented Jul 6, 2024

Uh oh!

ryanking13 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ryanking13 Jul 7, 2024

Choose a reason for hiding this comment

Uh oh!

hoodmane Jul 7, 2024

Choose a reason for hiding this comment

Uh oh!

ryanking13 Jul 8, 2024

Choose a reason for hiding this comment

Uh oh!

ryanking13 Jul 8, 2024

Choose a reason for hiding this comment

Uh oh!

hoodmane Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ilayn commented Jul 7, 2024

Uh oh!

hoodmane commented Jul 9, 2024

Uh oh!

lesteve commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoodmane commented Jul 9, 2024

Uh oh!

lesteve commented Jul 9, 2024

Uh oh!

lesteve commented Jul 9, 2024

Uh oh!

hoodmane commented Jul 10, 2024

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal commented Jul 10, 2024

Uh oh!

ilayn commented Jul 16, 2024

Uh oh!

hoodmane commented Jul 16, 2024

Uh oh!

hoodmane commented Jul 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hoodmane commented Jul 6, 2024 •

edited

Loading

ryanking13 left a comment •

edited

Loading

lesteve commented Jul 9, 2024 •

edited

Loading