Skip to content

Commit f14321e

Browse files
committed
ci: Compare benchmarks with the base branch on every PR
maint/benchmark/ci_benchmark.py measures CPU time per unit for the parsers (typical and worst-case headers, cookies, header params, multipart), the websocket mask, xhtml_escape, and the server scenarios from dos_benchmark.py (refactored so they can be called from other scripts). compare_benchmarks.py runs it against two source trees. Each round starts a worker process per tree and alternates measurements between them, so both see the same machine conditions, and keeps the fastest result for each. Anything more than 20% slower is measured again with twice as many processes and fails only if that agrees, since one side occasionally gets an unusually fast or slow process. The new CI job runs the comparison against the PR's base branch, alongside the full tests (it takes a little over a minute), and puts the table on the run's summary page. It also fails if an adversarial input exceeds an absolute budget in microseconds per byte. In repeated comparisons of a tree with itself nothing failed (one benchmark needed a recheck); a 40% slowdown in header parsing failed on the header benchmarks only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01TuTKdWgh56FRF7yJqaX9ug (cherry picked from commit f9f6f35)
1 parent f088359 commit f14321e

5 files changed

Lines changed: 484 additions & 13 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,36 @@ jobs:
205205
- name: Run test suite
206206
run: python -m tornado.test
207207

208+
benchmark:
209+
# Compares CPU cost per request, per byte, etc. with the PR's base
210+
# branch (see maint/benchmark/compare_benchmarks.py) and fails on a
211+
# regression of more than 20%. The table is shown on the run's summary
212+
# page. This runs alongside the full tests, so it adds no wall time.
213+
name: Compare benchmarks with base branch
214+
needs: test_quick
215+
runs-on: ubuntu-24.04
216+
timeout-minutes: 10
217+
steps:
218+
- uses: actions/checkout@v6
219+
with:
220+
persist-credentials: false
221+
- uses: actions/checkout@v6
222+
name: Check out base branch
223+
with:
224+
persist-credentials: false
225+
ref: ${{ github.event.pull_request.base.sha }}
226+
path: base
227+
- uses: actions/setup-python@v6
228+
name: Install Python
229+
with:
230+
# [[[cog cog.outl(f"python-version: '3.{default_python_minor}'")]]]
231+
python-version: '3.13'
232+
# [[[end]]]
233+
- name: Run benchmarks
234+
run: >-
235+
python maint/benchmark/compare_benchmarks.py --base base --head .
236+
--markdown "$GITHUB_STEP_SUMMARY"
237+
208238
zizmor:
209239
name: Analyze action configs with zizmor
210240
runs-on: ubuntu-24.04

‎CONTRIBUTING.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,12 @@
4949
escaping, highly compressible data) and keep sizes small enough that the test takes a
5050
fraction of a second. A fix for a denial-of-service bug should come with such a test, and
5151
the test should fail on the code before the fix.
52+
* The "Compare benchmarks" CI job runs `maint/benchmark/ci_benchmark.py` against the PR and
53+
its base branch and fails if anything is more than 20% slower, or if an adversarial input
54+
exceeds its absolute budget (`BUDGETS` in that file). To run the comparison locally, check
55+
out the base in another directory and run
56+
`python maint/benchmark/compare_benchmarks.py --base ../base --head .`. When you add a
57+
parser or a resource-budget test for one, add a benchmark of its worst-case input too.
5258
* Tests can also be run with the standard library's `unittest` package CLI. This is useful
5359
for integration with some editors. However, this mode skips a few checks, notably the
5460
requirement that tests do not generate any log output without corresponding ExpectLog

‎maint/benchmark/ci_benchmark.py‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
#!/usr/bin/env python
2+
#
3+
# A set of short benchmarks whose results can be compared between two
4+
# versions of Tornado. Each benchmark reports one number, the CPU time
5+
# per unit of work. compare_benchmarks.py runs this script against two
6+
# checkouts and reports the differences; the CI benchmark job uses that to
7+
# catch performance regressions in PRs.
8+
#
9+
# Usage (from the root of the repo):
10+
# PYTHONPATH=. python maint/benchmark/ci_benchmark.py
11+
# PYTHONPATH=. python maint/benchmark/ci_benchmark.py --only headers_many
12+
#
13+
# This prints one measurement of each benchmark as JSON. With --worker it
14+
# instead reads benchmark names from stdin and answers each with one
15+
# measurement on a line of JSON, which lets compare_benchmarks.py
16+
# interleave measurements of two versions.
17+
#
18+
# A parser measurement repeats the call until it has run for a minimum time
19+
# and takes the fastest of several such repetitions; server benchmarks
20+
# reuse the scenarios in dos_benchmark.py, measuring the CPU time of the
21+
# server thread only.
22+
#
23+
# BUDGETS sets absolute ceilings for the adversarial inputs, in
24+
# microseconds per input byte. They are set several times above what a
25+
# 2026 laptop measures, so that any CI runner passes them comfortably. They
26+
# are a backstop for the per-PR comparison: a regression that creeps in a
27+
# little at a time can't exceed them.
28+
29+
import argparse
30+
import asyncio
31+
import json
32+
import sys
33+
import time
34+
import typing
35+
from collections.abc import Callable
36+
37+
import dos_benchmark
38+
39+
from tornado import escape, httputil, util
40+
41+
42+
class Benchmark(typing.NamedTuple):
43+
# Returns CPU seconds for the given number of units of work.
44+
run: Callable[[], float]
45+
units: int
46+
unit: str
47+
48+
49+
# Minimum CPU time for one parser repetition, in seconds. Short
50+
# measurements are dominated by timer resolution and scheduling noise.
51+
_MIN_TIME = 0.02
52+
53+
54+
def parser(func: Callable[[], object], units: int, unit: str) -> Benchmark:
55+
def run() -> float:
56+
func() # warm up caches
57+
best = float("inf")
58+
for _ in range(5):
59+
count = 0
60+
start = time.thread_time()
61+
while True:
62+
func()
63+
count += 1
64+
elapsed = time.thread_time() - start
65+
if elapsed >= _MIN_TIME:
66+
break
67+
best = min(best, elapsed / count)
68+
return best
69+
70+
return Benchmark(run, units, unit)
71+
72+
73+
def server(name: str, n: int, per_byte: bool = True) -> Benchmark:
74+
sc = dos_benchmark.SCENARIOS[name]
75+
_, data = sc.build(n, sc.size)
76+
warmed_up = False
77+
78+
def run() -> float:
79+
nonlocal warmed_up
80+
if not warmed_up:
81+
# The first run in a process pays for imports and warmup.
82+
asyncio.run(dos_benchmark.measure(sc, max(1, n // 10), sc.size))
83+
warmed_up = True
84+
return asyncio.run(dos_benchmark.measure(sc, n, sc.size)).cpu
85+
86+
if per_byte:
87+
return Benchmark(run, len(data), "byte")
88+
return Benchmark(run, n, sc.unit)
89+
90+
91+
TYPICAL_HEADERS = (
92+
"Host: example.com\r\n"
93+
"User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 "
94+
"Firefox/140.0\r\n"
95+
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n"
96+
"Accept-Language: en-US,en;q=0.5\r\n"
97+
"Accept-Encoding: gzip, deflate, br, zstd\r\n"
98+
"Connection: keep-alive\r\n"
99+
"Cookie: session=abcdef0123456789; theme=dark; _ga=GA1.1.123456789.1700000000\r\n"
100+
"Upgrade-Insecure-Requests: 1\r\n"
101+
)
102+
103+
# Default limits bound the adversarial inputs: max_header_size is 64KB and
104+
# the default max_parts for multipart bodies is 100.
105+
MANY_HEADERS = "".join(f"X{i}: b\r\n" for i in range(8_000))
106+
REPEATED_HEADERS = "X: a\r\n" * 10_000
107+
MANY_COOKIES = "".join(f"{i}=;" for i in range(10_000))
108+
MANY_PARAMS = "form-data" + "; a=b" * 10_000
109+
MULTIPART_PART = (
110+
b"--1234\r\n"
111+
b'Content-Disposition: form-data; name="a"; filename="b"\r\n'
112+
b"Content-Type: text/plain\r\n\r\n\r\n"
113+
)
114+
MULTIPART_PARTS = MULTIPART_PART * 99 + b"--1234--\r\n"
115+
MULTIPART_LARGE = (
116+
b"--1234\r\n"
117+
b'Content-Disposition: form-data; name="a"; filename="b"\r\n\r\n'
118+
+ b"x" * 1_000_000
119+
+ b"\r\n--1234--\r\n"
120+
)
121+
MASK_DATA = b"x" * 65536
122+
ESCAPE_TEXT = '<p>Tom & Jerry\'s "show"</p> ' * 2_000
123+
124+
125+
def parse_multipart(body: bytes) -> None:
126+
args: dict[str, list[bytes]] = {}
127+
files: dict[str, list[httputil.HTTPFile]] = {}
128+
httputil.parse_multipart_form_data(b"1234", body, args, files)
129+
130+
131+
BENCHMARKS: dict[str, Benchmark] = {
132+
"headers_typical": parser(
133+
lambda: httputil.HTTPHeaders.parse(TYPICAL_HEADERS),
134+
len(TYPICAL_HEADERS),
135+
"byte",
136+
),
137+
"headers_many": parser(
138+
lambda: httputil.HTTPHeaders.parse(MANY_HEADERS), len(MANY_HEADERS), "byte"
139+
),
140+
"headers_repeated": parser(
141+
lambda: httputil.HTTPHeaders.parse(REPEATED_HEADERS),
142+
len(REPEATED_HEADERS),
143+
"byte",
144+
),
145+
"start_line": parser(
146+
lambda: httputil.parse_request_start_line("GET /foo/bar?a=b HTTP/1.1"),
147+
1,
148+
"line",
149+
),
150+
"cookies_many": parser(
151+
lambda: httputil.parse_cookie(MANY_COOKIES), len(MANY_COOKIES), "byte"
152+
),
153+
"header_params": parser(
154+
lambda: httputil._parse_header(MANY_PARAMS), len(MANY_PARAMS), "byte"
155+
),
156+
"multipart_parts": parser(
157+
lambda: parse_multipart(MULTIPART_PARTS), len(MULTIPART_PARTS), "byte"
158+
),
159+
"multipart_large": parser(
160+
lambda: parse_multipart(MULTIPART_LARGE), len(MULTIPART_LARGE), "byte"
161+
),
162+
"websocket_mask": parser(
163+
lambda: util._websocket_mask(b"abcd", MASK_DATA), len(MASK_DATA), "byte"
164+
),
165+
"xhtml_escape": parser(
166+
lambda: escape.xhtml_escape(ESCAPE_TEXT), len(ESCAPE_TEXT), "byte"
167+
),
168+
"http_requests": server("http_pipelined", 2_000, per_byte=False),
169+
"http_chunked": server("http_chunked", 50_000),
170+
"http_fixed": server("http_fixed", 1_000),
171+
"ws_messages": server("ws_messages", 50_000),
172+
"ws_fragments": server("ws_fragments", 50_000),
173+
"ws_deflate": server("ws_deflate", 30_000),
174+
"ws_large": server("ws_large", 50),
175+
}
176+
177+
# Ceilings in microseconds per unit (see the comment at the top).
178+
BUDGETS: dict[str, float] = {
179+
"headers_many": 2.0,
180+
"headers_repeated": 2.0,
181+
"cookies_many": 0.5,
182+
"header_params": 2.0,
183+
"multipart_parts": 1.0,
184+
"http_chunked": 8.0,
185+
"ws_messages": 8.0,
186+
"ws_fragments": 8.0,
187+
"ws_deflate": 8.0,
188+
}
189+
190+
191+
def measure(name: str) -> dict[str, typing.Any]:
192+
bm = BENCHMARKS[name]
193+
result: dict[str, typing.Any] = {"unit": bm.unit, "budget": BUDGETS.get(name)}
194+
try:
195+
result["us_per_unit"] = bm.run() / bm.units * 1e6
196+
except Exception as e:
197+
# Probably a benchmark of something the other version under
198+
# comparison lacks.
199+
result["error"] = repr(e)
200+
return result
201+
202+
203+
def main() -> None:
204+
parser = argparse.ArgumentParser()
205+
parser.add_argument("--only", action="append", help="benchmarks to run")
206+
parser.add_argument(
207+
"--worker", action="store_true", help="read benchmark names from stdin"
208+
)
209+
parser.add_argument("--list", action="store_true", help="print the benchmark names")
210+
args = parser.parse_args()
211+
if args.list:
212+
print("\n".join(BENCHMARKS))
213+
elif args.worker:
214+
for line in sys.stdin:
215+
print(json.dumps(measure(line.strip())), flush=True)
216+
else:
217+
results = {name: measure(name) for name in args.only or BENCHMARKS}
218+
json.dump(results, sys.stdout, indent=1)
219+
print()
220+
221+
222+
if __name__ == "__main__":
223+
main()

0 commit comments

Comments
 (0)