Benchmarks for CPython's cyclic garbage collector, for both the default (GIL) and the free-threaded build. Most need nothing beyond the interpreter under test and a C compiler (for the extension module used by some workloads); they work with uninstalled build trees. The real-application tier also needs pyperformance and a Sphinx installation.
Most of these are stress tests, not realistic workloads: GC takes 7-90% of their time, against a median of about 0.5% across pyperformance, and almost all their garbage is cyclic, while real programs free most garbage by reference counting. They are good at reproducing and explaining specific collector problems; they exaggerate the effect of a GC change on real programs. Use three tiers:
- Pathology tests (
suite.py --tier pathology): one known problem each; read their primary result (below). - Steady-state workloads (
suite.py --tier workloadsandthreads):web_servicehas realistic proportions (a large mixed heap, mostly acyclic garbage, an LRU cache with turnover, sessions living many requests); run them for 30-60 s, or they may never reach a full collection. - Real applications (
realapps.py): the GC-relevant pyperformance benchmarks and a Sphinx build of CPython's documentation, with the GC statistics hook.
| file | what |
|---|---|
suite.py |
runs the recommended set (pathology, workloads, threads), pinned to cores, one JSON line per run |
realapps.py |
real-application tier: pyperformance subset and the docs build with GC statistics |
hook/sitecustomize.py |
per-process GC statistics, used by realapps.py (active with GCBENCH_STATS_DIR) |
gcstats.py |
collection statistics shared by the benchmarks: young/full split, pause percentiles |
gen_frag.py |
page occupancy: sparse (scattered survivors), churn (a table with turnover) |
bigcollect.py |
collect (one large collection: time, RSS returned) and container (a large container built while young) |
revert_cases.py |
migrations, doctree, deque: the workloads behind the incremental-GC reverts |
workloads.py |
steady-state workloads run by a common multi-threaded harness |
gcbench.py |
synthetic micro-benchmarks: collect, traverse, pause, release |
scaling.py |
heap size, run time, stack depth and threads varied one at a time |
_gcbench_ext.c, buildext.py |
extension module with GC and non-GC types used by some workloads |
compare.py |
compares suite.py results (and realapps.py output) of two or more configurations: per benchmark its time or throughput, memory, longest collection and other primary results, with the change from the baseline |
perfgc.py |
analysis: perf script samples in the collector |
EXAMPLE.md, example/ |
a complete example run, default build against free-threaded build, with its output and how to read it |
| benchmark | problem it reproduces | primary result |
|---|---|---|
gen_frag.py sparse |
long-lived survivors scattered thinly over young pages | final RSS |
gen_frag.py churn [--acyclic F] |
medium-lived objects (a table with turnover) that die after promotion | time, full collections and their time |
bigcollect.py collect |
a 3M-object collection next to a live heap | final collection time; RSS before and after (is the memory returned?) |
bigcollect.py container |
a large container that stays young while it is filled | build time, young pauses |
revert_cases.py migrations |
Django migrate: classes living about one step (gh-148726) |
peak RSS, longest pause (its wall time is GC-bound, not a real migrate's) |
revert_cases.py doctree |
Sphinx/docutils: a heap growing with every document (gh-124717) | GC time per document against live nodes, per quarter; longest pause |
revert_cases.py deque |
Tim Peters' test: reclamation delay of garbage holding big payloads | backlog of uncollected garbage, peak RSS (timings say little) |
bigbuffers.py SIZE |
two-object cycles each holding a bytearray of SIZE bytes: memory from Python's allocator that the object count can't see (the suite runs 200 KB and 4 MB) | megabytes allocated per collection, peak RSS (the runs are too short to time) |
workloads.py ext_buffers_malloc, ext_buffers_pymem |
cycles holding large side buffers the count-based trigger can't see, taken from malloc() or from PyMem_Malloc() (ext_buffers mixes the two: two malloc() buffers and one PyMem_Malloc() buffer per step) |
side_peak_mb, peak RSS (ops/s depends on the allocator as much as on the collector: read it next to page_faults, the memory handed back to the OS and taken again) |
workloads.py hidden_refs --ops N |
a slow leak through a cycle the GC can't see | throughput, leaked holders, peak RSS |
workloads.py c_globals and py_globals |
objects owned by a C static global (not reachable from known roots) | c_globals against py_globals: only meaningful as a pair |
Every workload builds its long-lived data, then runs --threads worker
threads (default 4) for --duration seconds (default 30) or, with
--ops N, until N steps are done in total (the same work, and for leaking
workloads the same heap growth, for every configuration). A ticker thread
records every stall longer than 200 us unless --no-ticker is given.
| workload | real-world pattern | what makes it interesting for the GC |
|---|---|---|
web_service |
a web service: 2M-object mixed heap built gradually, LRU cache with turnover, sessions living 50 requests, request handling with mostly acyclic garbage and a few percent cyclic | realistic proportions: run for 60 s, and with 4-8 threads |
orm_session |
ORM unit of work: session with a WeakValueDictionary identity map, __slots__ models with parent/children and session back-references |
every request leaves cyclic garbage with weakrefs and weakref callbacks |
event_widgets |
GUI/event-emitter object tree: parent/children, listeners as bound methods, lambdas closing over self, functools.partial |
many small cycles through bound methods, closures and cells |
xml_dom |
XML processing with xml.dom.minidom (parse, query, modify, serialize) |
real library code; stdlib objects with parent/child links |
async_server |
asyncio request handling: gather, async generators closed early, exceptions kept in a local |
exception/traceback/frame cycles, objects with finalizers |
json_churn |
json round trips next to a large configuration |
acyclic garbage only: a control where automatic collections should not happen |
ext_buffers_malloc, ext_buffers_pymem, hidden_refs, c_globals, py_globals |
see the pathology tests |
orm_session, event_widgets, xml_dom and async_server are one
family: a static long-lived heap plus short-lived, almost entirely cyclic
garbage. They differ in object types, not in what they ask of the
collector; treat them as one piece of evidence, not four.
Options that make them more realistic:
--acyclic Fadds request-like acyclic garbage to every step so that a fraction F (for example 0.9-0.99) of the GC objects a step leaves behind is acyclic, as in real programs. The harness measures a step's cyclic garbage before the run (cyclic_objs_per_step).--medium Nkeeps a small cyclic object graph per step for N steps: a medium-lived component (sessions, requests in flight) whose objects survive several young collections and then die.
workloads.py reports, per workload, its primary results first, then
throughput, the share of time spent in collections, collections split into
young and full with the median, p99 and longest pause, stalls (with the
ticker), RSS after setup and at its peak (and once a second in
rss_series_mb), and workload-specific counters.
gcstats.py splits collections into young and full ones: in the default
build, generation 2 collections are full; in the free-threaded build every
collection is full unless generational collection is enabled
(PYTHON_GC_GENERATIONAL=1), which reports young collections as generation
0 and full ones as generation 2.
PY=path/to/build/python
$PY suite.py --python $PY --env PYTHON_GC_GENERATIONAL=1 --out gen.jsonl
$PY suite.py --python $PY --out nongen.jsonl
$PY suite.py --python $PY --quick # smoke test only
$PY compare.py nongen.jsonl gen.jsonl --names nongen,gen
$PY workloads.py run web_service --threads 8 --duration 60
$PY realapps.py sphinx --venv DOCS_VENV --doc path/to/cpython/Doc --out out/sphinx
$PY realapps.py pyperf --python $PY --pyperformance "path/to/pyperformance" --out out/pyperfsuite.py pins single-threaded runs to two cores (--cpus, default 2,3)
and multi-threaded runs to one core per thread (--thread-cpus, default
0-7); pick performance cores on hybrid CPUs, and run one suite at a time on
an idle machine. For the docs build, create a virtual environment of the
interpreter under test with Doc/requirements.txt installed; on the
free-threaded build pass --env PYTHON_GIL=0, or MarkupSafe's speedups
module re-enables the GIL.
- Give GC-time changes next to throughput changes, and the share of time
spent in GC: a 10% faster collector saves 10% of the GC share, which is
2-8% of these stress tests' time but typically 0.05-1.5% of a real
program's (
realapps.pyreports the share for real programs). - Lead with each benchmark's primary result; don't read the wall time of GC-bound pathology tests as representative.
- Compare several runs: most results vary by 2% or less, the longest pause much more (one outlier sets it), and multi-threaded runs more than single-threaded ones. In the default build, multi-threaded workloads are dominated by GIL contention, and stalls include GIL waits.