Disable parallel cythonization for "spawn" start method. by mbargull · Pull Request #3263 · cython/cython

mbargull · 2019-12-06T17:42:58Z

I have not looked into Cython's test framework.
An integration test that runs a small script like the following would work:

#cython: language_level=3

from multiprocessing import set_start_method
set_start_method("spawn", force=True)

from Cython.Build import cythonize
cythonize([__file__] * 2, nthreads=1)

samuelcolvin · 2019-12-06T18:00:19Z

Surely better to have a warning or error to alert people to this?

mbargull · 2019-12-06T18:09:51Z

warning:
Yes, I agree. I only used print because that seemed like the prevalent choice in the source file.
If the maintainers are okay with using warnings, I'll gladly switch to that.
error:
see: Handle cythonize(..., nthreads=1) for "spawn" start method #3262

re. 2.: Cython already only warns and continues with serial processing if nthreads is non-zero but multiprocessing isn't available. So raising a error seems inconsequential.

Thanks for taking a look @samuelcolvin!

scoder · 2020-04-30T10:18:42Z

I think it's ok as it is, just needs to work on Py2.7 as well, which lacks mp.get_start_method().

mLewisLogic · 2020-10-04T04:50:28Z

Just hit this error trying to enable multiple nthreads. Would be nice to mainline.

scoder · 2020-10-04T06:03:46Z

See my comment just above.

mLewisLogic · 2020-10-04T06:49:35Z

Ya, recognize that's a blocker. Just sharing user experience (from somebody running 3.8).
A search for the exact error led me to this issue, otherwise debugging it would have been a rabbit hole.

da-woods · 2020-10-04T08:38:45Z

Presumably it'd do to check for sys.platform == "win32" instead on Python 2.7

scoder · 2020-10-04T09:31:04Z

Presumably …

Well, yes, maybe. There is no regression test for this, so I can't tell.

jakeschurch · 2023-09-04T00:01:23Z

this seems to fix issues im seeing currently on my machine when added as patch. What's blocking getting this merged? Is this anything I can help with?

jakeschurch · 2023-09-04T00:04:21Z

I think it's ok as it is, just needs to work on Py2.7 as well, which lacks mp.get_start_method().

@scoder what is cython's bugfix policy for python2 bugs? For Python I know the sunset date for any more bugfixes on python 2 was Jan 1, 2020. Surely this isn't a blocker?

jakeschurch · 2023-09-04T00:40:40Z

we could also guard check with sys.version_info >= (3, 8)

scoder · 2025-08-30T15:08:56Z

Thanks. Sorry it took so long.

Resolves #3262

webknjaz · 2025-10-01T21:02:31Z

So while this took care of Cython/Build/Dependencies.py, the same problem exists in Cython/Build/Cythonize.py. It should be patched the same.

A similar problem was fixed in cython#3263 which is what this patch is porting to another module. That fixed the Python-land API while this change aims to also fix its command-line counterpart. The symptoms are Python 3.8+, macOS and the underlying compiler invocations failing. If these are met, `cythonize` would hang before this commit. Fixes cython#3973. Refs: * cython#3262 * cython#7146 * ansible/pylibssh#769

webknjaz · 2025-10-01T22:40:49Z

#7183 should complete the fix started in #3263.

A similar problem was fixed in cython#3263 which is what this patch is porting to another module. That fixed the Python-land API while this change aims to also fix its command-line counterpart. The symptoms are Python 3.8+, macOS and the underlying compiler invocations failing. If these are met, `cythonize` would hang before this commit. Fixes cython#3973. Refs: * cython#3262 * cython#7146 * ansible/pylibssh#769

jakeschurch · 2025-10-08T22:34:26Z

no worries - i really appreciate it!

A similar problem was fixed in cython#3263 which is what this patch is porting to another module. That fixed the Python-land API while this change aims to also fix its command-line counterpart. The symptoms are Python 3.8+, macOS and the underlying compiler invocations failing. If these are met, `cythonize` would hang before this commit. Fixes cython#3973. Refs: * cython#3262 * cython#7146 * ansible/pylibssh#769

A similar problem was worked around in cython#3263 by disabling parallelism altogether. This patch reverts that and updates the worker management to use concurrent.futures.ProcessPoolExecutor instead of multiprocessing.Pool. The symptoms are Python 3.8+ and the underlying compiler invocations failing. If these are met, cythonize would hang before this commit. Original PR by Sviatoslav Sydorenko in cython#7183 Fixes cython#3973 See cython#3262 See cython#7146

A similar problem was worked around in #3263 by disabling parallelism altogether. This patch reverts that and updates the worker management to use concurrent.futures.ProcessPoolExecutor instead of multiprocessing.Pool. The symptoms are Python 3.8+ and the underlying compiler invocations failing. If these are met, cythonize would hang before this commit. Original PR by Sviatoslav Sydorenko in #7183 Fixes #3973 See #3262 See #7146

A similar problem was worked around in #3263 by disabling parallelism altogether. This patch reverts that and updates the worker management to use concurrent.futures.ProcessPoolExecutor instead of multiprocessing.Pool. The symptoms are Python 3.8+ and the underlying compiler invocations failing. If these are met, cythonize would hang before this commit. Original PR by Sviatoslav Sydorenko in #7183 Fixes #3973 Closes #7183 See #3262 See #7146

Disable parallel cythonization for "spawn" start method.

15ae78b

mbargull mentioned this pull request Dec 6, 2019

Rebuild for python38 conda-forge/pydantic-feedstock#32

Merged

jakeschurch mentioned this pull request Sep 4, 2023

Cython builds on python3.8+ on MacOS appears to be broken NixOS/nixpkgs#253205

Open

scoder added this to the 3.2 milestone Aug 30, 2025

scoder added defect Build System labels Aug 30, 2025

Merge branch 'master' into fix-cythonize-multiprocessing-spawn

6a54de6

scoder merged commit 1a4f6a7 into cython:master Aug 30, 2025
79 checks passed

scoder modified the milestones: 3.2, 3.1.4 Sep 13, 2025

scoder pushed a commit that referenced this pull request Sep 13, 2025

Disable parallel cythonization for "spawn" start method. (GH-3263)

dd18f11

Resolves #3262

webknjaz mentioned this pull request Oct 1, 2025

Compilation failures cause installation hang ansible/pylibssh#769

Closed

This was referenced Oct 1, 2025

[BUG] python setup.py build_ext -j4 fails on macOS with Python 3.8+ #3973

Closed

[BUG] multiprocessing module hangs when starting workers #7146

Open

webknjaz mentioned this pull request Oct 1, 2025

Fix parallelism in cythonize cmd & callable via concurrent.futures #7183

Closed

webknjaz mentioned this pull request Oct 1, 2025

Handle cythonize(..., nthreads=1) for "spawn" start method #3262

Closed

Uh oh!

Conversation

mbargull commented Dec 6, 2019

Uh oh!

samuelcolvin commented Dec 6, 2019

Uh oh!

mbargull commented Dec 6, 2019

Uh oh!

scoder commented Apr 30, 2020

Uh oh!

mLewisLogic commented Oct 4, 2020

Uh oh!

scoder commented Oct 4, 2020

Uh oh!

mLewisLogic commented Oct 4, 2020

Uh oh!

da-woods commented Oct 4, 2020

Uh oh!

scoder commented Oct 4, 2020

Uh oh!

jakeschurch commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakeschurch commented Sep 4, 2023

Uh oh!

jakeschurch commented Sep 4, 2023

Uh oh!

Uh oh!

scoder commented Aug 30, 2025

Uh oh!

webknjaz commented Oct 1, 2025

Uh oh!

webknjaz commented Oct 1, 2025

Uh oh!

jakeschurch commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jakeschurch commented Sep 4, 2023 •

edited

Loading