Skip to content

Commit 1427d8e

Browse files
claudebdarnell
authored andcommitted
test: Fix flaky cookie unquoting linear-performance test on PyPy
test_unquote_linear_performance failed intermittently on PyPy with a ratio just over its limit of 20. It was not timer noise: each sample is already at least 0.1s and the minimum of several. Three things were wrong: * assert_linear_scaling disabled the GC on every implementation. On PyPy that only stops major collections, so objects promoted out of the nursery pile up for the whole measurement (about 1MB per call here), and the larger input pays for touching fresh memory. This inflated the ratio of linear code to 20-90 depending on PYPY_GC_NURSERY. The GC is now only disabled on CPython. * Even without that, this test allocates heavily per input byte and its PyPy ratio depends on the nursery and cache sizes (11-24 measured), so its limit is raised to 40. The quadratic 6.4.1 implementation measures about 100 on CPython and 60-100 on PyPy. * The quick measurement timed n1 first, before PyPy's JIT had warmed up, which inflated it enough that the quadratic 6.4.1 implementation passed this test on PyPy. A warmup now precedes it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0168zU4yXrAsAoY3KAWuiE8n
1 parent a4dd782 commit 1427d8e

2 files changed

Lines changed: 50 additions & 18 deletions

File tree

‎tornado/test/httputil_test.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -817,13 +817,20 @@ def test_unquote_linear_performance(self):
817817
# escape sequences in a cookie value was quadratic. (Only the r"\\"
818818
# case reproduces it on 6.4.1; r"\134" is here for symmetry with
819819
# test_unquote_large.)
820+
#
821+
# 6.4.1 measures a ratio of about 100 on CPython and 60-100 on PyPy.
822+
# The fixed version measures about 10 on CPython, but this test
823+
# allocates heavily for each byte of input, and PyPy's ratio depends
824+
# on how that compares to its nursery and CPU caches: as of 2026-10
825+
# it measures 11-24 depending on PYPY_GC_NURSERY, so the usual limit
826+
# of 20 is too tight.
820827
for encoded in r"\\", r"\134":
821828
with self.subTest(encoded):
822829
assert_linear_scaling(
823830
lambda n: parse_cookie('a="b=' + encoded * n + '"'),
824831
1_000,
825832
10_000,
826-
max_ratio=20,
833+
max_ratio=40,
827834
msg="Cookie unquoting is not linear",
828835
)
829836

‎tornado/test/util.py‎

Lines changed: 42 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,38 @@ def _calibrate(
228228
iterations *= 2
229229

230230

231+
@contextlib.contextmanager
232+
def _gc_disabled() -> Generator[None, None, None]:
233+
"""Disable the garbage collector for the measurements in
234+
`assert_linear_scaling`, on CPython only.
235+
236+
CPython's collections are triggered by allocation counts, and their cost
237+
depends on the number of live objects, so leaving the collector enabled
238+
charges an arbitrary and unevenly distributed cost to whichever operation
239+
happens to trigger one. Reference counting still frees almost everything
240+
while it is disabled.
241+
242+
PyPy has no reference counting, and `gc.disable` there only stops major
243+
collections: minor collections keep promoting objects that survive the
244+
nursery into an old generation that is never collected. Memory then
245+
grows throughout the measurement (by about 1MB per call of the cookie
246+
unquoting test at ``n=10_000``), and the larger input, which promotes
247+
more, pays disproportionately for touching fresh memory. This has been
248+
seen to inflate a linear ratio of about 10 to 20 or more, and in some
249+
configurations to over 80, so on PyPy the collector is left alone.
250+
"""
251+
if platform.python_implementation() != "CPython":
252+
yield
253+
return
254+
gc_was_enabled = gc.isenabled()
255+
gc.disable()
256+
try:
257+
yield
258+
finally:
259+
if gc_was_enabled:
260+
gc.enable()
261+
262+
231263
def assert_linear_scaling(
232264
func: Callable[[int], object],
233265
n1: int,
@@ -255,10 +287,8 @@ def assert_linear_scaling(
255287
* Only CPU time used by this process is measured (see `_time_calls`).
256288
* Each operation is repeated enough times to be measured reliably (see
257289
`_MIN_MEASUREMENT_TIME`).
258-
* The garbage collector is disabled during the measurements. Collections
259-
are triggered by allocation counts, and their cost depends on the
260-
number of live objects, so leaving it enabled charges an arbitrary and
261-
unevenly distributed cost to whichever operation happens to trigger one.
290+
* On CPython, the garbage collector is disabled during the measurements
291+
(see `_gc_disabled`).
262292
* Each size is measured ``rounds`` times and the lowest result is used.
263293
Interference can only make an operation look slower, never faster, so
264294
the minimum is a much more stable estimate of its cost than the mean.
@@ -275,33 +305,28 @@ def assert_linear_scaling(
275305
factor larger than the margin between ``max_ratio`` and the ratio of a
276306
real change in complexity class. Linear code often measures a little
277307
over ``n2 / n1``, so a threshold of half of ``max_ratio`` would send
278-
most passing tests to the slow measurement.)
308+
most passing tests to the slow measurement.) The quick measurement is
309+
itself preceded by a warmup of the same length, because the first calls
310+
are systematically slower, especially on PyPy before its JIT has
311+
compiled the code. That inflates the time for ``n1``, which is measured
312+
first, and can hide even a quadratic regression.
279313
"""
280-
gc_was_enabled = gc.isenabled()
281-
gc.disable()
282-
try:
314+
with _gc_disabled():
315+
_calibrate(func, n1, _QUICK_MEASUREMENT_TIME)
283316
iterations1, elapsed1 = _calibrate(func, n1, _QUICK_MEASUREMENT_TIME)
284317
iterations2, elapsed2 = _calibrate(func, n2, _QUICK_MEASUREMENT_TIME)
285-
finally:
286-
if gc_was_enabled:
287-
gc.enable()
288318
if (elapsed2 / iterations2) / (elapsed1 / iterations1) <= max_ratio * 0.75:
289319
return
290320
t1 = t2 = 0.0
291321
for _ in range(attempts):
292-
gc_was_enabled = gc.isenabled()
293-
gc.disable()
294-
try:
322+
with _gc_disabled():
295323
iterations1, elapsed1 = _calibrate(func, n1)
296324
iterations2, elapsed2 = _calibrate(func, n2)
297325
t1 = elapsed1 / iterations1
298326
t2 = elapsed2 / iterations2
299327
for _ in range(rounds - 1):
300328
t1 = min(t1, _time_calls(func, n1, iterations1) / iterations1)
301329
t2 = min(t2, _time_calls(func, n2, iterations2) / iterations2)
302-
finally:
303-
if gc_was_enabled:
304-
gc.enable()
305330
if t2 / t1 <= max_ratio:
306331
return
307332
raise AssertionError(

0 commit comments

Comments
 (0)