Skip to content

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator#109808

Merged
vstinner merged 8 commits intopython:mainfrom
chgnrdv:_thread-start-new-thread-use-raw-malloc-for-bootstate
Sep 25, 2023
Merged

gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator#109808
vstinner merged 8 commits intopython:mainfrom
chgnrdv:_thread-start-new-thread-use-raw-malloc-for-bootstate

Conversation

@chgnrdv
Copy link
Copy Markdown
Contributor

@chgnrdv chgnrdv commented Sep 24, 2023 •
edited by bedevere-app Bot
Loading

Fixes #109795

When new thread is starting, but Python is finalizing, PyMem_Free can't be called on new thread bootstate, because this thread doesn't own the GIL. We should use raw memory allocator to allocate/free bootstate to avoid crashes/other unexpected behaviour.

@chgnrdv
_thread.start_new_thread: allocate/free thread bootstate using raw me…
4e1d6df 
…mory allocator

When new thread is starting, but Python is finalizing, PyMem_Free can't be called on new thread bootstate, because this created thread doesn't own the GIL. We should use raw memory allocator to allocate/free bootstate to avoid crashes/other unexpected behaviour.
@bedevere-app bedevere-app Bot mentioned this pull request Sep 24, 2023
Closed
@chgnrdv chgnrdv changed the title gh-109795: _thread.start_new_thread: allocate/free thread bootstate using raw memory allocator gh-109795: _thread.start_new_thread: allocate thread bootstate using raw memory allocator Sep 24, 2023
@chgnrdv
Copy link
Copy Markdown
Contributor Author

chgnrdv commented Sep 24, 2023

Windows (x86) failed due to #109739.

vstinner
vstinner reviewed Sep 25, 2023
View reviewed changes
Comment thread Modules/_threadmodule.c
@chgnrdv
added comment that refers the changes
e5c0f92
@chgnrdv
Merge branch '_thread-start-new-thread-use-raw-malloc-for-bootstate' …
b2f5e37 
…of github.com:chgnrdv/cpython into _thread-start-new-thread-use-raw-malloc-for-bootstate
vstinner
vstinner reviewed Sep 25, 2023
View reviewed changes
Comment thread Modules/_threadmodule.c Outdated
vstinner
vstinner reviewed Sep 25, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Core and Builtins/2023-09-24-22-26-56.gh-issue-109795.gI7KW6.rst Outdated
@@ -0,0 +1 @@
:func:`!_thread.start_new_thread` now uses raw memory allocator to allocate new thread bootstate. This helps to avoid crashes in case when new thread gets started at interpreter finalization.
Copy link
Copy Markdown
Member

@vstinner vstinner Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. Please remove this Changelog entry. I don't think that it's worth it. It doesn't impact directly users and it's a recent regression, no?

@vstinner vstinner merged commit 1b8f236 into python:main Sep 25, 2023
@vstinner vstinner added the needs backport to 3.11 only security fixes label Sep 25, 2023
@miss-islington
Copy link
Copy Markdown
Contributor

miss-islington commented Sep 25, 2023

Thanks @chgnrdv for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@miss-islington
Copy link
Copy Markdown
Contributor

miss-islington commented Sep 25, 2023

Sorry, @chgnrdv and @vstinner, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 1b8f2366b38c87b0450d9c15bdfdd4c4a2fc3a01 3.11

vstinner pushed a commit to vstinner/cpython that referenced this pull request Sep 25, 2023
@chgnrdv @vstinner
pythongh-109795: _thread.start_new_thread: allocate thread bootstat…
75e36a4 
…e using raw memory allocator (python#109808)

(cherry picked from commit 1b8f236)
@vstinner vstinner mentioned this pull request Sep 25, 2023
Merged
@bedevere-app
Copy link
Copy Markdown

bedevere-app Bot commented Sep 25, 2023

GH-109852 is a backport of this pull request to the 3.11 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.11 only security fixes label Sep 25, 2023
@vstinner
Copy link
Copy Markdown
Member

vstinner commented Sep 25, 2023

I didn't backport PR gh-109135 yet, since I'm waiting PR gh-109133 to be merged. I will merge this fix at the same time than backporting gh-109135.

@vstinner
Copy link
Copy Markdown
Member

vstinner commented Sep 25, 2023

Merged, thanks!

@bedevere-bot
Copy link
Copy Markdown

bedevere-bot commented Sep 25, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 LTO 3.x has failed when building commit 1b8f236.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/338/builds/5153) and take a look at the build logs.
  4. Check if the failure is related to this commit (1b8f236) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/338/builds/5153

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1004, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1100, in get_code
  File "<frozen importlib._bootstrap_external>", line 1199, in get_data
TypeError: descriptor 'close' for '_io.BufferedReader' objects doesn't apply to a '_io.FileIO' object


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/threading.py", line 996, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/test/test_interpreters.py", line 483, in task
    interp = interpreters.create()
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-aarch64.lto/build/Lib/test/support/interpreters.py", line 25, in create
    id = _interpreters.create(isolated=isolated)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: interpreter creation failed
k

vstinner added a commit that referenced this pull request Sep 25, 2023
@vstinner @chgnrdv
[3.11] gh-109795: _thread.start_new_thread: allocate thread bootsta…
0afae89 
…te usin… (#109852)

gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808)

(cherry picked from commit 1b8f236)

Co-authored-by: Radislav Chugunov <[email protected]>
csm10495 pushed a commit to csm10495/cpython that referenced this pull request Sep 28, 2023
@chgnrdv @csm10495
pythongh-109795: _thread.start_new_thread: allocate thread bootstat…
f3d62b3 
…e using raw memory allocator (python#109808)
vstinner pushed a commit to vstinner/cpython that referenced this pull request Oct 4, 2023
@chgnrdv @vstinner
pythongh-109795: _thread.start_new_thread: allocate thread bootstat…
b4da602 
…e using raw memory allocator (python#109808)

(cherry picked from commit 1b8f236)
vstinner added a commit that referenced this pull request Oct 4, 2023
@vstinner @chgnrdv
[3.12] gh-108987: Fix _thread.start_new_thread() race condition (#109135
4936fa9 
) (#110342)

* gh-108987: Fix _thread.start_new_thread() race condition (#109135)

Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.

thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.

Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().

(cherry picked from commit 517cd82)

* gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808)

(cherry picked from commit 1b8f236)

---------

Co-authored-by: Radislav Chugunov <[email protected]>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@chgnrdv @Glyphack
pythongh-109795: _thread.start_new_thread: allocate thread bootstat…
c709195 
…e using raw memory allocator (python#109808)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments

Assignees

@vstinner vstinner

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Starting new thread during finalization leads to call to PyMem_Free without holding the GIL

4 participants

@chgnrdv @miss-islington @vstinner @bedevere-bot