Skip to content

Commit 05f2f0a

Browse files
DinoVcolesburytiranvstinner
authored
gh-90815: Add mimalloc memory allocator (#109914)
* Add mimalloc v2.12 Modified src/alloc.c to remove include of alloc-override.c and not compile new handler. Did not include the following files: - include/mimalloc-new-delete.h - include/mimalloc-override.h - src/alloc-override-osx.c - src/alloc-override.c - src/static.c - src/region.c mimalloc is thread safe and shares a single heap across all runtimes, therefore finalization and getting global allocated blocks across all runtimes is different. * mimalloc: minimal changes for use in Python: - remove debug spam for freeing large allocations - use same bytes (0xDD) for freed allocations in CPython and mimalloc This is important for the test_capi debug memory tests * Don't export mimalloc symbol in libpython. * Enable mimalloc as Python allocator option. * Add mimalloc MIT license. * Log mimalloc in Lib/test/pythoninfo.py. * Document new mimalloc support. * Use macro defs for exports as done in: #31164 Co-authored-by: Sam Gross <[email protected]> Co-authored-by: Christian Heimes <[email protected]> Co-authored-by: Victor Stinner <[email protected]>
1 parent 4ebf2fa commit 05f2f0a

61 files changed

Lines changed: 16789 additions & 158 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Doc/c-api/init_config.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,11 +253,21 @@ PyPreConfig
253253
* ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` (``6``): :ref:`Python pymalloc
254254
memory allocator <pymalloc>` with :ref:`debug hooks
255255
<pymem-debug-hooks>`.
256+
* ``PYMEM_ALLOCATOR_MIMALLOC`` (``6``): use ``mimalloc``, a fast
257+
malloc replacement.
258+
* ``PYMEM_ALLOCATOR_MIMALLOC_DEBUG`` (``7``): use ``mimalloc``, a fast
259+
malloc replacement with :ref:`debug hooks <pymem-debug-hooks>`.
260+
256261
257262
``PYMEM_ALLOCATOR_PYMALLOC`` and ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` are
258263
not supported if Python is :option:`configured using --without-pymalloc
259264
<--without-pymalloc>`.
260265
266+
``PYMEM_ALLOCATOR_MIMALLOC`` and ``PYMEM_ALLOCATOR_MIMALLOC_DEBUG`` are
267+
not supported if Python is :option:`configured using --without-mimalloc
268+
<--without-mimalloc>` or if the underlying atomic support isn't
269+
available.
270+
261271
See :ref:`Memory Management <memory>`.
262272
263273
Default: ``PYMEM_ALLOCATOR_NOT_SET``.

‎Doc/c-api/memory.rst‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -391,6 +391,8 @@ Legend:
391391
* ``malloc``: system allocators from the standard C library, C functions:
392392
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`.
393393
* ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
394+
* ``mimalloc``: :ref:`mimalloc memory allocator <mimalloc>`. The pymalloc
395+
allocator will be used if mimalloc support isn't available.
394396
* "+ debug": with :ref:`debug hooks on the Python memory allocators
395397
<pymem-debug-hooks>`.
396398
* "Debug build": :ref:`Python build in debug mode <debug-build>`.
@@ -672,6 +674,16 @@ Customize pymalloc Arena Allocator
672674
673675
Set the arena allocator.
674676
677+
.. _mimalloc:
678+
679+
The mimalloc allocator
680+
======================
681+
682+
.. versionadded:: 3.13
683+
684+
Python supports the mimalloc allocator when the underlying platform support is available.
685+
mimalloc "is a general purpose allocator with excellent performance characteristics.
686+
Initially developed by Daan Leijen for the runtime systems of the Koka and Lean languages."
675687
676688
tracemalloc C API
677689
=================

‎Doc/license.rst‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1040,3 +1040,29 @@ https://www.w3.org/TR/xml-c14n2-testcases/ and is distributed under the
10401040
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
10411041
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
10421042
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
1043+
1044+
1045+
mimalloc
1046+
--------
1047+
1048+
MIT License
1049+
1050+
Copyright (c) 2018-2021 Microsoft Corporation, Daan Leijen
1051+
1052+
Permission is hereby granted, free of charge, to any person obtaining a copy
1053+
of this software and associated documentation files (the "Software"), to deal
1054+
in the Software without restriction, including without limitation the rights
1055+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
1056+
copies of the Software, and to permit persons to whom the Software is
1057+
furnished to do so, subject to the following conditions:
1058+
1059+
The above copyright notice and this permission notice shall be included in all
1060+
copies or substantial portions of the Software.
1061+
1062+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
1063+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
1064+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
1065+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
1066+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
1067+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
1068+
SOFTWARE.

‎Doc/using/cmdline.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -921,13 +921,17 @@ conflict.
921921
* ``pymalloc``: use the :ref:`pymalloc allocator <pymalloc>` for
922922
:c:macro:`PYMEM_DOMAIN_MEM` and :c:macro:`PYMEM_DOMAIN_OBJ` domains and use
923923
the :c:func:`malloc` function for the :c:macro:`PYMEM_DOMAIN_RAW` domain.
924+
* ``mimalloc``: use the :ref:`mimalloc allocator <mimalloc>` for
925+
:c:macro:`PYMEM_DOMAIN_MEM` and :c:macro:`PYMEM_DOMAIN_OBJ` domains and use
926+
the :c:func:`malloc` function for the :c:macro:`PYMEM_DOMAIN_RAW` domain.
924927

925928
Install :ref:`debug hooks <pymem-debug-hooks>`:
926929

927930
* ``debug``: install debug hooks on top of the :ref:`default memory
928931
allocators <default-memory-allocators>`.
929932
* ``malloc_debug``: same as ``malloc`` but also install debug hooks.
930933
* ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks.
934+
* ``mimalloc_debug``: same as ``mimalloc`` but also install debug hooks.
931935

932936
.. versionchanged:: 3.7
933937
Added the ``"default"`` allocator.

‎Doc/using/configure.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -579,6 +579,13 @@ also be used to improve performance.
579579
Enable computed gotos in evaluation loop (enabled by default on supported
580580
compilers).
581581

582+
.. option:: --without-mimalloc
583+
584+
Disable the fast mimalloc allocator :ref:`mimalloc <mimalloc>`
585+
(enabled by default).
586+
587+
See also :envvar:`PYTHONMALLOC` environment variable.
588+
582589
.. option:: --without-pymalloc
583590

584591
Disable the specialized Python memory allocator :ref:`pymalloc <pymalloc>`

‎Include/cpython/pymem.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ typedef enum {
2323
PYMEM_ALLOCATOR_PYMALLOC = 5,
2424
PYMEM_ALLOCATOR_PYMALLOC_DEBUG = 6,
2525
#endif
26+
#ifdef WITH_MIMALLOC
27+
PYMEM_ALLOCATOR_MIMALLOC = 7,
28+
PYMEM_ALLOCATOR_MIMALLOC_DEBUG = 8,
29+
#endif
2630
} PyMemAllocatorName;
2731

2832

0 commit comments

Comments
 (0)