Skip to content

Commit 5d39e04

Browse files
vstinnervstinner
authored
bpo-32030: Rework memory allocators (#4625)
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to the defaults, rather than only caling _PyMem_SetupDebugHooks(). * Add _PyMem_SetDefaultAllocator() helper to set the "default" allocator. * Add _PyMem_GetAllocatorsName(): get the name of the allocators * main() now uses debug hooks on memory allocators if Py_DEBUG is defined, rather than calling directly malloc() * Document default memory allocators in C API documentation * _Py_InitializeCore() now fails with a fatal user error if PYTHONMALLOC value is an unknown memory allocator, instead of failing with a fatal internal error. * Add new tests on the PYTHONMALLOC environment variable * Add support.with_pymalloc() * Add the _testcapi.WITH_PYMALLOC constant and expose it as support.with_pymalloc(). * sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so replace it with support.with_pymalloc(). * pythoninfo: add _testcapi collector for pymem
1 parent c15bb49 commit 5d39e04

File tree

14 files changed

+403
-169
lines changed

14 files changed

+403
-169
lines changed

‎Doc/c-api/memory.rst‎

Lines changed: 38 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -100,9 +100,10 @@ The following function sets are wrappers to the system allocator. These
100100
functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
101101
need to be held.
102102

103-
The default raw memory block allocator uses the following functions:
104-
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call
105-
``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes.
103+
The :ref:`default raw memory allocator <default-memory-allocators>` uses
104+
the following functions: :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc`
105+
and :c:func:`free`; call ``malloc(1)`` (or ``calloc(1, 1)``) when requesting
106+
zero bytes.
106107

107108
.. versionadded:: 3.4
108109

@@ -165,7 +166,8 @@ The following function sets, modeled after the ANSI C standard, but specifying
165166
behavior when requesting zero bytes, are available for allocating and releasing
166167
memory from the Python heap.
167168
168-
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
169+
The :ref:`default memory allocator <default-memory-allocators>` uses the
170+
:ref:`pymalloc memory allocator <pymalloc>`.
169171
170172
.. warning::
171173
@@ -270,7 +272,8 @@ The following function sets, modeled after the ANSI C standard, but specifying
270272
behavior when requesting zero bytes, are available for allocating and releasing
271273
memory from the Python heap.
272274
273-
By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
275+
The :ref:`default object allocator <default-memory-allocators>` uses the
276+
:ref:`pymalloc memory allocator <pymalloc>`.
274277
275278
.. warning::
276279
@@ -326,6 +329,31 @@ By default, these functions use :ref:`pymalloc memory allocator <pymalloc>`.
326329
If *p* is *NULL*, no operation is performed.
327330
328331
332+
.. _default-memory-allocators:
333+
334+
Default Memory Allocators
335+
=========================
336+
337+
Default memory allocators:
338+
339+
=============================== ==================== ================== ===================== ====================
340+
Configuration Name PyMem_RawMalloc PyMem_Malloc PyObject_Malloc
341+
=============================== ==================== ================== ===================== ====================
342+
Release build ``"pymalloc"`` ``malloc`` ``pymalloc`` ``pymalloc``
343+
Debug build ``"pymalloc_debug"`` ``malloc`` + debug ``pymalloc`` + debug ``pymalloc`` + debug
344+
Release build, without pymalloc ``"malloc"`` ``malloc`` ``malloc`` ``malloc``
345+
Release build, without pymalloc ``"malloc_debug"`` ``malloc`` + debug ``malloc`` + debug ``malloc`` + debug
346+
=============================== ==================== ================== ===================== ====================
347+
348+
Legend:
349+
350+
* Name: value for :envvar:`PYTHONMALLOC` environment variable
351+
* ``malloc``: system allocators from the standard C library, C functions:
352+
:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`
353+
* ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`
354+
* "+ debug": with debug hooks installed by :c:func:`PyMem_SetupDebugHooks`
355+
356+
329357
Customize Memory Allocators
330358
===========================
331359
@@ -431,7 +459,8 @@ Customize Memory Allocators
431459
displayed if :mod:`tracemalloc` is tracing Python memory allocations and the
432460
memory block was traced.
433461
434-
These hooks are installed by default if Python is compiled in debug
462+
These hooks are :ref:`installed by default <default-memory-allocators>` if
463+
Python is compiled in debug
435464
mode. The :envvar:`PYTHONMALLOC` environment variable can be used to install
436465
debug hooks on a Python compiled in release mode.
437466
@@ -453,9 +482,9 @@ to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
453482
with a fixed size of 256 KiB. It falls back to :c:func:`PyMem_RawMalloc` and
454483
:c:func:`PyMem_RawRealloc` for allocations larger than 512 bytes.
455484
456-
*pymalloc* is the default allocator of the :c:data:`PYMEM_DOMAIN_MEM` (ex:
457-
:c:func:`PyMem_Malloc`) and :c:data:`PYMEM_DOMAIN_OBJ` (ex:
458-
:c:func:`PyObject_Malloc`) domains.
485+
*pymalloc* is the :ref:`default allocator <default-memory-allocators>` of the
486+
:c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) and
487+
:c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) domains.
459488
460489
The arena allocator uses the following functions:
461490

‎Doc/using/cmdline.rst‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -687,6 +687,8 @@ conflict.
687687

688688
Set the family of memory allocators used by Python:
689689

690+
* ``default``: use the :ref:`default memory allocators
691+
<default-memory-allocators>`.
690692
* ``malloc``: use the :c:func:`malloc` function of the C library
691693
for all domains (:c:data:`PYMEM_DOMAIN_RAW`, :c:data:`PYMEM_DOMAIN_MEM`,
692694
:c:data:`PYMEM_DOMAIN_OBJ`).
@@ -696,20 +698,17 @@ conflict.
696698

697699
Install debug hooks:
698700

699-
* ``debug``: install debug hooks on top of the default memory allocator
701+
* ``debug``: install debug hooks on top of the :ref:`default memory
702+
allocators <default-memory-allocators>`.
700703
* ``malloc_debug``: same as ``malloc`` but also install debug hooks
701704
* ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks
702705

703-
When Python is compiled in release mode, the default is ``pymalloc``. When
704-
compiled in debug mode, the default is ``pymalloc_debug`` and the debug hooks
705-
are used automatically.
706+
See the :ref:`default memory allocators <default-memory-allocators>` and the
707+
:c:func:`PyMem_SetupDebugHooks` function (install debug hooks on Python
708+
memory allocators).
706709

707-
If Python is configured without ``pymalloc`` support, ``pymalloc`` and
708-
``pymalloc_debug`` are not available, the default is ``malloc`` in release
709-
mode and ``malloc_debug`` in debug mode.
710-
711-
See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python
712-
memory allocators.
710+
.. versionchanged:: 3.7
711+
Added the ``"default"`` allocator.
713712

714713
.. versionadded:: 3.6
715714

‎Include/pymem.h‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
2121
allocators. */
2222
PyAPI_FUNC(int) _PyMem_SetupAllocators(const char *opt);
2323

24+
/* Try to get the allocators name set by _PyMem_SetupAllocators(). */
25+
PyAPI_FUNC(const char*) _PyMem_GetAllocatorsName(void);
26+
2427
#ifdef WITH_PYMALLOC
2528
PyAPI_FUNC(int) _PyMem_PymallocEnabled(void);
2629
#endif
@@ -230,7 +233,12 @@ PyAPI_FUNC(void) PyMem_SetupDebugHooks(void);
230233
#endif
231234

232235
#ifdef Py_BUILD_CORE
233-
PyAPI_FUNC(void) _PyMem_GetDefaultRawAllocator(PyMemAllocatorEx *alloc);
236+
/* Set the memory allocator of the specified domain to the default.
237+
Save the old allocator into *old_alloc if it's non-NULL.
238+
Return on success, or return -1 if the domain is unknown. */
239+
PyAPI_FUNC(int) _PyMem_SetDefaultAllocator(
240+
PyMemAllocatorDomain domain,
241+
PyMemAllocatorEx *old_alloc);
234242
#endif
235243

236244
#ifdef __cplusplus

‎Lib/test/pythoninfo.py‎

Lines changed: 32 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,14 @@ def copy_attributes(info_add, obj, name_fmt, attributes, *, formatter=None):
5656
info_add(name, value)
5757

5858

59+
def copy_attr(info_add, name, mod, attr_name):
60+
try:
61+
value = getattr(mod, attr_name)
62+
except AttributeError:
63+
return
64+
info_add(name, value)
65+
66+
5967
def call_func(info_add, name, mod, func_name, *, formatter=None):
6068
try:
6169
func = getattr(mod, func_name)
@@ -168,11 +176,10 @@ def format_attr(attr, value):
168176
call_func(info_add, 'os.gid', os, 'getgid')
169177
call_func(info_add, 'os.uname', os, 'uname')
170178

171-
if hasattr(os, 'getgroups'):
172-
groups = os.getgroups()
173-
groups = map(str, groups)
174-
groups = ', '.join(groups)
175-
info_add("os.groups", groups)
179+
def format_groups(groups):
180+
return ', '.join(map(str, groups))
181+
182+
call_func(info_add, 'os.groups', os, 'getgroups', formatter=format_groups)
176183

177184
if hasattr(os, 'getlogin'):
178185
try:
@@ -184,11 +191,7 @@ def format_attr(attr, value):
184191
else:
185192
info_add("os.login", login)
186193

187-
if hasattr(os, 'cpu_count'):
188-
cpu_count = os.cpu_count()
189-
if cpu_count:
190-
info_add('os.cpu_count', cpu_count)
191-
194+
call_func(info_add, 'os.cpu_count', os, 'cpu_count')
192195
call_func(info_add, 'os.loadavg', os, 'getloadavg')
193196

194197
# Get environment variables: filter to list
@@ -219,7 +222,9 @@ def format_attr(attr, value):
219222
)
220223
for name, value in os.environ.items():
221224
uname = name.upper()
222-
if (uname in ENV_VARS or uname.startswith(("PYTHON", "LC_"))
225+
if (uname in ENV_VARS
226+
# Copy PYTHON* and LC_* variables
227+
or uname.startswith(("PYTHON", "LC_"))
223228
# Visual Studio: VS140COMNTOOLS
224229
or (uname.startswith("VS") and uname.endswith("COMNTOOLS"))):
225230
info_add('os.environ[%s]' % name, value)
@@ -313,12 +318,10 @@ def collect_time(info_add):
313318
)
314319
copy_attributes(info_add, time, 'time.%s', attributes)
315320

316-
if not hasattr(time, 'get_clock_info'):
317-
return
318-
319-
for clock in ('time', 'perf_counter'):
320-
tinfo = time.get_clock_info(clock)
321-
info_add('time.%s' % clock, tinfo)
321+
if hasattr(time, 'get_clock_info'):
322+
for clock in ('time', 'perf_counter'):
323+
tinfo = time.get_clock_info(clock)
324+
info_add('time.%s' % clock, tinfo)
322325

323326

324327
def collect_sysconfig(info_add):
@@ -331,14 +334,14 @@ def collect_sysconfig(info_add):
331334
'CCSHARED',
332335
'CFLAGS',
333336
'CFLAGSFORSHARED',
334-
'PY_LDFLAGS',
335337
'CONFIG_ARGS',
336338
'HOST_GNU_TYPE',
337339
'MACHDEP',
338340
'MULTIARCH',
339341
'OPT',
340342
'PY_CFLAGS',
341343
'PY_CFLAGS_NODIST',
344+
'PY_LDFLAGS',
342345
'Py_DEBUG',
343346
'Py_ENABLE_SHARED',
344347
'SHELL',
@@ -422,6 +425,16 @@ def collect_decimal(info_add):
422425
copy_attributes(info_add, _decimal, '_decimal.%s', attributes)
423426

424427

428+
def collect_testcapi(info_add):
429+
try:
430+
import _testcapi
431+
except ImportError:
432+
return
433+
434+
call_func(info_add, 'pymem.allocator', _testcapi, 'pymem_getallocatorsname')
435+
copy_attr(info_add, 'pymem.with_pymalloc', _testcapi, 'WITH_PYMALLOC')
436+
437+
425438
def collect_info(info):
426439
error = False
427440
info_add = info.add
@@ -444,6 +457,7 @@ def collect_info(info):
444457
collect_zlib,
445458
collect_expat,
446459
collect_decimal,
460+
collect_testcapi,
447461
):
448462
try:
449463
collect_func(info_add)

‎Lib/test/support/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2848,3 +2848,8 @@ def save(self):
28482848
def restore(self):
28492849
for signum, handler in self.handlers.items():
28502850
self.signal.signal(signum, handler)
2851+
2852+
2853+
def with_pymalloc():
2854+
import _testcapi
2855+
return _testcapi.WITH_PYMALLOC

‎Lib/test/test_capi.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -654,8 +654,7 @@ class PyMemMallocDebugTests(PyMemDebugTests):
654654
PYTHONMALLOC = 'malloc_debug'
655655

656656

657-
@unittest.skipUnless(sysconfig.get_config_var('WITH_PYMALLOC') == 1,
658-
'need pymalloc')
657+
@unittest.skipUnless(support.with_pymalloc(), 'need pymalloc')
659658
class PyMemPymallocDebugTests(PyMemDebugTests):
660659
PYTHONMALLOC = 'pymalloc_debug'
661660

‎Lib/test/test_cmd_line.py‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import os
66
import subprocess
77
import sys
8+
import sysconfig
89
import tempfile
910
import unittest
1011
from test import support
@@ -559,10 +560,14 @@ def test_xdev(self):
559560
except ImportError:
560561
pass
561562
else:
562-
code = "import _testcapi; _testcapi.pymem_api_misuse()"
563+
code = "import _testcapi; print(_testcapi.pymem_getallocatorsname())"
563564
with support.SuppressCrashReport():
564565
out = self.run_xdev("-c", code, check_exitcode=False)
565-
self.assertIn("Debug memory block at address p=", out)
566+
if support.with_pymalloc():
567+
alloc_name = "pymalloc_debug"
568+
else:
569+
alloc_name = "malloc_debug"
570+
self.assertEqual(out, alloc_name)
566571

567572
try:
568573
import faulthandler
@@ -573,6 +578,49 @@ def test_xdev(self):
573578
out = self.run_xdev("-c", code)
574579
self.assertEqual(out, "True")
575580

581+
def check_pythonmalloc(self, env_var, name):
582+
code = 'import _testcapi; print(_testcapi.pymem_getallocatorsname())'
583+
env = dict(os.environ)
584+
if env_var is not None:
585+
env['PYTHONMALLOC'] = env_var
586+
else:
587+
env.pop('PYTHONMALLOC', None)
588+
args = (sys.executable, '-c', code)
589+
proc = subprocess.run(args,
590+
stdout=subprocess.PIPE,
591+
stderr=subprocess.STDOUT,
592+
universal_newlines=True,
593+
env=env)
594+
self.assertEqual(proc.stdout.rstrip(), name)
595+
self.assertEqual(proc.returncode, 0)
596+
597+
def test_pythonmalloc(self):
598+
# Test the PYTHONMALLOC environment variable
599+
pydebug = hasattr(sys, "gettotalrefcount")
600+
pymalloc = support.with_pymalloc()
601+
if pymalloc:
602+
default_name = 'pymalloc_debug' if pydebug else 'pymalloc'
603+
default_name_debug = 'pymalloc_debug'
604+
else:
605+
default_name = 'malloc_debug' if pydebug else 'malloc'
606+
default_name_debug = 'malloc_debug'
607+
608+
tests = [
609+
(None, default_name),
610+
('debug', default_name_debug),
611+
('malloc', 'malloc'),
612+
('malloc_debug', 'malloc_debug'),
613+
]
614+
if pymalloc:
615+
tests.extend((
616+
('pymalloc', 'pymalloc'),
617+
('pymalloc_debug', 'pymalloc_debug'),
618+
))
619+
620+
for env_var, name in tests:
621+
with self.subTest(env_var=env_var, name=name):
622+
self.check_pythonmalloc(env_var, name)
623+
576624

577625
class IgnoreEnvironmentTest(unittest.TestCase):
578626

‎Lib/test/test_sys.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -753,8 +753,15 @@ def test_debugmallocstats(self):
753753
@unittest.skipUnless(hasattr(sys, "getallocatedblocks"),
754754
"sys.getallocatedblocks unavailable on this build")
755755
def test_getallocatedblocks(self):
756+
try:
757+
import _testcapi
758+
except ImportError:
759+
with_pymalloc = support.with_pymalloc()
760+
else:
761+
alloc_name = _testcapi.pymem_getallocatorsname()
762+
with_pymalloc = (alloc_name in ('pymalloc', 'pymalloc_debug'))
763+
756764
# Some sanity checks
757-
with_pymalloc = sysconfig.get_config_var('WITH_PYMALLOC')
758765
a = sys.getallocatedblocks()
759766
self.assertIs(type(a), int)
760767
if with_pymalloc:

0 commit comments

Comments
 (0)