Skip to content

Commit 5b34e33

Browse files
alexeyshockovclaude
andcommitted
build(bench): micro-benchmarks run on the newest Python, in a nox session
The shared tooling at v0.73.0 runs `just bench::micro` in the noxfile's sessions tagged `bench` instead of the floor `.venv`; this project's noxfile gains the `bench` session (one per supported Python, the tag on the newest, 3.15 today). Results are saved per interpreter, and `bench::compare` compares runs of one interpreter. On 3.15 the JSON comparison now gives pyparsing 2.15x and parsy 1.73x (medians, `just bench::micro`), in line with the depth sweep on 3.15 (2.22x, 1.69x); on 3.12 the two harnesses disagreed by up to 40%, with the starting call-stack depth (python/cpython#142183). benchmarks/README.md says so, and how to run another interpreter. `.nox/` is ignored: the uv environments in it hide themselves, a repository without them should not depend on that. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent c07178d commit 5b34e33

5 files changed

Lines changed: 49 additions & 10 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,4 @@ venv/
1818
.copier-answers.yml
1919
.hypothesis/
2020
benchmarks/results/
21+
.nox/

‎CONVENTIONS.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -293,9 +293,13 @@ its package directories in `[tool.conventions] packages`. A rule that is one pro
293293
- Micro-benchmarks are pytest-benchmark cases, `bench_*.py` files in a `benchmarks/` directory
294294
beside a distribution's `tests/`, and run in-process with no server or network. A case whose
295295
timing includes threads or sockets is compared with others of the same run only.
296-
- `just bench::micro [DIST] [ARGS]` runs them in the shared `.venv` under `benchmarks/pytest.ini`,
297-
which no test suite reads, and saves each run under `benchmarks/results/`, with the machine and
298-
the commit; `just bench::compare [RUN ...]` compares saved runs, the last two by default.
296+
- `just bench::micro [DIST] [ARGS]` runs them in the noxfile's sessions tagged `bench` under
297+
`benchmarks/pytest.ini`, which no test suite reads, and saves each run under
298+
`benchmarks/results/`, with the machine, the interpreter and the commit. The `bench` session
299+
runs on one interpreter each; the tag is on the newest by default, and a project whose benchmark
300+
dependencies have no wheel for it yet moves the tag in its noxfile. A figure is quoted with the
301+
interpreter it ran on. `just bench::compare [RUN ...]` compares saved runs, by default the two
302+
newest of one machine and interpreter.
299303
- Macro-benchmarks run against a server or a driver, or drive a load generator, and are noisy,
300304
local measurement tools, not CI gates: do not add them to routine verification or run them
301305
unless the task is performance work.

‎benchmarks/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,12 @@
33
Micro-benchmarks of tokenizing and parsing: `just bench::micro` runs them, `just bench::compare` compares saved runs,
44
and `CONVENTIONS.md`'s Benchmarks section has the rules.
55

6+
They run in the noxfile's `bench` session tagged `bench`, on the newest supported Python; another interpreter's runs by
7+
name, `just tools::nox -s "bench(python='3.14')" -- benchmarks`. Quote a figure with the interpreter it ran on. On
8+
Python 3.12 a recursive parser's time depends on the call-stack depth it starts at (up to 57% here, CPython's slow path
9+
for calls across data-stack chunks, [python/cpython#142183](https://github.com/python/cpython/issues/142183)); on 3.14
10+
and 3.15 it varies by about 1 ms.
11+
612
- `bench_grammars.py`: whole grammars over a generated document, the JSON and DOT grammars of `tests/` and the
713
calculator of `examples/`.
814
- `bench_combinators.py`: one combinator each, over a long input, so a regression points at its combinator.

‎benchmarks/justfile‎

Lines changed: 19 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
#
44
# The micro-benchmarks are the `bench_*.py` files of each distribution's `benchmarks/` directory,
55
# beside its `tests/`; a single-distribution repository keeps them in this directory. They run in
6-
# the shared `.venv`, which `just venv` syncs with every group, under `pytest.ini` beside this file.
6+
# the noxfile's `bench` sessions, one interpreter each, under `pytest.ini` beside this file; the
7+
# session tagged `bench` is the default, and the project's noxfile says which interpreter that is.
78

89
set working-directory := '..'
910

@@ -27,25 +28,36 @@ types:
2728
[ ${#dirs[@]} -gt 0 ] || { echo "ty: no benchmarks/ directory to check"; exit 1; }
2829
exec {{ tools }} toolbox ty -- "${dirs[@]}"
2930
30-
# `results/` keeps every run, with the machine and the commit it ran on.
31-
[doc("Run the micro-benchmarks, every distribution's or DIST's; the other arguments go to pytest")]
31+
# `results/` keeps every run, with the machine, the interpreter and the commit it ran on. Another
32+
# interpreter's session runs by name: `just tools::nox -s "bench(python='3.14')" -- benchmarks`.
33+
[doc("Run the micro-benchmarks in the noxfile's sessions tagged `bench`, every distribution's or DIST's; the other arguments go to pytest")]
3234
[positional-arguments]
3335
micro *args:
3436
#!/usr/bin/env bash
3537
set -euo pipefail
3638
dirs=({{ bench_dirs }})
3739
if [ $# -gt 0 ] && [ -d "$1/benchmarks" ]; then dirs=("$1/benchmarks"); shift; fi
3840
[ ${#dirs[@]} -gt 0 ] || { echo "no benchmarks/ directory to run"; exit 1; }
39-
exec uv run --no-sync pytest -c benchmarks/pytest.ini "${dirs[@]}" "$@"
41+
{{ tools }} nox --tags bench --list --json | {{ tools }} python -c '
42+
import json, sys
43+
if not json.load(sys.stdin):
44+
sys.exit(
45+
"bench::micro: no noxfile session is tagged `bench`; add the `bench` session of the"
46+
" template noxfile to this project'"'"'s noxfile.py"
47+
)
48+
'
49+
exec {{ tools }} nox --tags bench -- -c benchmarks/pytest.ini "${dirs[@]}" "$@"
4050
41-
[doc("Compare saved runs, the last two by default; the table also goes to benchmarks/results/compare.csv")]
51+
[doc("Compare saved runs, by default the two newest of one machine and interpreter; the table also goes to benchmarks/results/compare.csv")]
4252
[positional-arguments]
4353
compare *runs:
4454
#!/usr/bin/env bash
4555
set -euo pipefail
4656
if [ $# -eq 0 ]; then
47-
set -- $(ls -t benchmarks/results/*/*.json 2>/dev/null | head -n 2)
48-
[ $# -gt 0 ] || { echo "no saved runs under benchmarks/results/"; exit 1; }
57+
# The two newest runs of one machine and interpreter: the directory of the newest run.
58+
newest=$(ls -t benchmarks/results/*/*.json 2>/dev/null | head -n 1)
59+
[ -n "$newest" ] || { echo "no saved runs under benchmarks/results/"; exit 1; }
60+
set -- $(ls -t "$(dirname "$newest")"/*.json | head -n 2)
4961
fi
5062
exec uv run --no-sync pytest-benchmark --storage file://benchmarks/results compare \
5163
--csv=benchmarks/results/compare "$@"

‎noxfile.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,19 @@ def tests(session: nox.Session, member: str) -> None:
5757
sync(session, *(["--package", MEMBERS[member]] if WORKSPACE else []))
5858
session.chdir(member)
5959
session.run("pytest", *session.posargs)
60+
61+
62+
# The micro-benchmarks (`just bench::micro`), where the project has them: one session per
63+
# interpreter. The one tagged `bench` is what `just bench::micro` runs: the newest, unless a
64+
# dependency the benchmarks build has no wheel for it yet, in which case the tag moves to another
65+
# version in BENCH_PYTHONS.
66+
BENCH_PYTHONS = [nox.param(v, tags=["bench"] if v == NEWEST else []) for v in VERSIONS]
67+
68+
if any(Path(member, "benchmarks").is_dir() for member in MEMBERS):
69+
70+
@nox.session
71+
@nox.parametrize("python", BENCH_PYTHONS)
72+
def bench(session: nox.Session) -> None:
73+
"""The micro-benchmarks on one interpreter; the arguments go to pytest."""
74+
sync(session, "--group", "bench", *(["--all-packages"] if WORKSPACE else []))
75+
session.run("pytest", *session.posargs)

0 commit comments

Comments
 (0)