Skip to content

Commit d43adfe

Browse files
authored
Drop -I from the ASan test command so PYTHONMALLOC takes effect (#1499)
<!-- Thank you for your contribution! --> ## What do these changes do? The ASan test command in `AGENTS.md` and in the CI `asan` job ran `python -Im pytest` with `PYTHONMALLOC=malloc`. `-I` implies `-E`, so Python ignored `PYTHONMALLOC` and kept serving small allocations from pymalloc arenas. Most `htkeys_t` tables are 512 bytes or less, so ASan could not detect use-after-free on them. This drops `-I` in both places and adds a short note explaining why it must stay out. The TSan command and job, and the `Makefile`, do not combine `-I`/`-E` with `PYTHONMALLOC` and are unchanged. ## Are there changes in behavior for the user? No. Contributor-facing only. ## Is it a substantial burden for the maintainers to support this? No. ## Related issue number N/A ## Checklist - [x] I think the code is well written - [ ] Unit tests for the changes exist: N/A, docs and CI invocation only - [x] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt`: N/A - [x] Add a new news fragment into the `CHANGES/` folder - [x] `make doc-spelling` passes and any new technical words are added to `docs/spelling_wordlist.txt` Drafted with Claude Opus 5 (Claude Code); reviewed by @asvetlov. <details> <summary>Agent run details (optional, for reviewers)</summary> Allocator check on CPython 3.13.2: `PYTHONMALLOC=malloc python -c ...pymem_getallocatorsname()` reports `malloc`; with `-I` it reports `pymalloc`. Earlier manual check: `tests/test_multidict.py::test_items_iter_key_str_reinits` (from #1496) hits a heap-use-after-free on the pre-fix code under `python -m pytest`, but passes under `python -Im pytest`. Corrected command on current master, ASan+UBSan build, isolated 3.13.2 venv: `LD_PRELOAD=$ASAN_SO ASAN_OPTIONS=detect_leaks=0:halt_on_error=1 PYTHONMALLOC=malloc python -m pytest tests -q --no-cov -k "not test_leak"` gave 2048 passed, 21 skipped, 11 deselected. So turning on real malloc in CI does not make master fail. `make doc-spelling`: passed. Pre-commit hooks: passed. </details>
1 parent 2a68472 commit d43adfe

3 files changed

Lines changed: 29 additions & 2 deletions

File tree

‎.github/workflows/ci-cd.yml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -586,6 +586,12 @@ jobs:
586586
# for its own aborts by default, independently of the kernel
587587
# core_pattern/ulimit setup above.
588588
ASAN_OPTIONS: detect_leaks=0:halt_on_error=1:disable_coredump=0
589+
# pymalloc serves small blocks (<= 512 bytes, most hash tables)
590+
# from its own arenas, so ASan never sees them freed and misses
591+
# use-after-free on them. Route every allocation through libc
592+
# malloc instead. Free-threaded builds reject this value (they
593+
# only accept the mimalloc allocators), so keep this job on a
594+
# GIL build.
589595
PYTHONMALLOC: malloc
590596
run: |
591597
ulimit -c unlimited
@@ -596,7 +602,9 @@ jobs:
596602
# abort calls _exit() before pytest gets to flush it, silently
597603
# dropping the report. Uncaptured output goes straight to the
598604
# job log instead.
599-
LD_PRELOAD="${ASAN_SO}" python -Im pytest tests -q --no-cov \
605+
# No -I: it implies -E, which ignores PYTHONMALLOC and leaves
606+
# small allocations in pymalloc arenas that ASan cannot check.
607+
LD_PRELOAD="${ASAN_SO}" python -m pytest tests -q --no-cov \
600608
--capture=no -k "not test_leak"
601609
shell: bash
602610
- name: Check for a core dump

‎AGENTS.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -379,9 +379,23 @@ ASAN_SO=$(cc -print-file-name=libasan.so)
379379
MULTIDICT_DEBUG_BUILD=1 MULTIDICT_ASAN_BUILD=1 \
380380
pip install -e . --force-reinstall --no-deps
381381
LD_PRELOAD="$ASAN_SO" ASAN_OPTIONS=detect_leaks=0 PYTHONMALLOC=malloc \
382-
python -Im pytest tests -q -k "not test_leak"
382+
python -m pytest tests -q -k "not test_leak"
383383
```
384384

385+
`PYTHONMALLOC=malloc` routes every allocation through libc `malloc`,
386+
which ASan intercepts. Without it, pymalloc serves small blocks
387+
(512 bytes or less, which covers most hash tables) from its own
388+
arenas; ASan sees the arena as one live allocation, so a freed block
389+
is never flagged and use-after-free goes unreported. Free-threaded
390+
builds do not support it: they abort at startup with
391+
`PYTHONMALLOC: unknown allocator`, since only the mimalloc allocators
392+
are available there. Run ASan on a GIL build.
393+
394+
Do not add `-I` (or `-E`) to that command. Both make Python ignore
395+
`PYTHON*` environment variables, so `PYTHONMALLOC=malloc` is silently
396+
dropped and small allocations (most hash tables) come from pymalloc
397+
arenas, where ASan cannot see use-after-free.
398+
385399
`detect_leaks=0` and excluding `test_leaks.py` are required: CPython
386400
itself retains allocations at shutdown (interned strings, caches)
387401
that LeakSanitizer reports as leaks, and `test_leaks.py` asserts on

‎CHANGES/1499.contrib.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Dropped ``-I`` from the AddressSanitizer test command in ``AGENTS.md``
2+
and in the CI job. It implies ``-E``, which made Python ignore
3+
``PYTHONMALLOC=malloc``, so small hash tables were still served from
4+
``pymalloc`` arenas where use-after-free went undetected
5+
-- by :user:`asvetlov`.

0 commit comments

Comments
 (0)