Skip to content

Commit c2e24d2

Browse files
authored
Merge pull request #3823 from bdarnell/claude/project-thread-uzh2s8
Backport new tests to branch6.6
2 parents 27c6f91 + c2d8884 commit c2e24d2

27 files changed

Lines changed: 4068 additions & 37 deletions

‎.coveragerc‎

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,38 @@
11
# Test coverage configuration.
22
# Usage:
3+
# tox -e coverage
4+
# or, without tox:
35
# pip install coverage
46
# coverage erase # clears previous data if any
57
# coverage run -m tornado.test.runtests
8+
# coverage combine
69
# coverage report # prints to stdout
710
# coverage html # creates ./htmlcov/*.html including annotated source
811
[run]
912
branch = true
10-
source = tornado
11-
omit =
12-
tornado/platform/*
13-
tornado/test/*
14-
*/_auto2to3*
13+
source_pkgs = tornado
14+
omit =
15+
*/tornado/test/*
16+
# Deprecated; still tested in the -full configurations, but we don't
17+
# track its coverage.
18+
*/tornado/platform/caresresolver.py
19+
# Each process writes its own data file; run `coverage combine` before
20+
# reporting.
21+
parallel = true
22+
# Several tests (autoreload, process, import) start new python processes,
23+
# and autoreload re-execs itself. Measure those too.
24+
patch =
25+
subprocess
26+
execv
27+
_exit
28+
fork
29+
30+
[paths]
31+
# tox runs the tests against an installed copy of tornado; map those paths
32+
# back to the source tree so reports point at the files in this repo.
33+
source =
34+
tornado/
35+
*/site-packages/tornado/
1536

1637
[report]
1738
# Ignore missing source files, i.e. fake template-generated "files"

‎.github/workflows/test.yml‎

Lines changed: 92 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -124,12 +124,49 @@ jobs:
124124
- name: Run test suite
125125
run: python -m tox -e ${{ matrix.tox_env }}
126126

127+
test_coverage:
128+
# Measure test coverage (including the alternate http client and
129+
# resolver configurations) and fail if it drops below the floor set in
130+
# tox.ini. The report is shown on the run's summary page and the HTML
131+
# version is uploaded as an artifact.
132+
name: Measure test coverage
133+
needs: test_quick
134+
runs-on: ubuntu-24.04
135+
timeout-minutes: 10
136+
steps:
137+
- uses: actions/checkout@v6
138+
with:
139+
persist-credentials: false
140+
- uses: actions/setup-python@v6
141+
name: Install Python
142+
with:
143+
# [[[cog cog.outl(f"python-version: '3.{default_python_minor}'")]]]
144+
python-version: '3.13'
145+
# [[[end]]]
146+
- name: Install apt packages
147+
run: sudo apt-get update && sudo apt-get install libcurl4-openssl-dev
148+
- name: Install tox
149+
run: python -m pip install tox -c requirements.txt
150+
151+
- name: Run test suite with coverage
152+
run: python -m tox -e coverage
153+
154+
- name: Write coverage summary
155+
if: ${{ !cancelled() }}
156+
run: >-
157+
.tox/coverage/bin/python -m coverage report --format=markdown
158+
--data-file=.tox/coverage-data/.coverage >> "$GITHUB_STEP_SUMMARY"
159+
- name: Upload HTML coverage report
160+
if: ${{ !cancelled() }}
161+
uses: actions/upload-artifact@v7
162+
with:
163+
name: coverage-html
164+
path: .tox/coverage-data/html
165+
127166
test_win:
128167
# Windows tests are fairly slow, so only run one configuration here.
129-
# We test on windows but not mac because even though mac is a more
130-
# fully-supported platform, it's similar enough to linux that we
131-
# don't generally need to test it separately. Windows is different
132-
# enough that we'll break it if we don't test it in CI.
168+
# Windows is different enough from linux and mac that we'll break it
169+
# if we don't test it in CI.
133170
name: Run windows tests
134171
needs: test_quick
135172
runs-on: windows-2025
@@ -147,6 +184,57 @@ jobs:
147184
- name: Run test suite
148185
run: py -m tornado.test
149186

187+
test_mac:
188+
# Mac is similar to linux, but uses kqueue instead of epoll and has
189+
# other platform differences that we only exercise here. One
190+
# configuration is enough.
191+
name: Run mac tests
192+
needs: test_quick
193+
runs-on: macos-15
194+
timeout-minutes: 5
195+
steps:
196+
- uses: actions/checkout@v6
197+
with:
198+
persist-credentials: false
199+
- uses: actions/setup-python@v6
200+
name: Install Python
201+
with:
202+
# [[[cog cog.outl(f"python-version: '3.{default_python_minor}'")]]]
203+
python-version: '3.13'
204+
# [[[end]]]
205+
- name: Run test suite
206+
run: python -m tornado.test
207+
208+
benchmark:
209+
# Compares CPU cost per request, per byte, etc. with the PR's base
210+
# branch (see maint/benchmark/compare_benchmarks.py) and fails on a
211+
# regression of more than 20%. The table is shown on the run's summary
212+
# page. This runs alongside the full tests, so it adds no wall time.
213+
name: Compare benchmarks with base branch
214+
needs: test_quick
215+
runs-on: ubuntu-24.04
216+
timeout-minutes: 10
217+
steps:
218+
- uses: actions/checkout@v6
219+
with:
220+
persist-credentials: false
221+
- uses: actions/checkout@v6
222+
name: Check out base branch
223+
with:
224+
persist-credentials: false
225+
ref: ${{ github.event.pull_request.base.sha }}
226+
path: base
227+
- uses: actions/setup-python@v6
228+
name: Install Python
229+
with:
230+
# [[[cog cog.outl(f"python-version: '3.{default_python_minor}'")]]]
231+
python-version: '3.13'
232+
# [[[end]]]
233+
- name: Run benchmarks
234+
run: >-
235+
python maint/benchmark/compare_benchmarks.py --base base --head .
236+
--markdown "$GITHUB_STEP_SUMMARY"
237+
150238
zizmor:
151239
name: Analyze action configs with zizmor
152240
runs-on: ubuntu-24.04

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,4 @@ MANIFEST
1515
# Used in demo apps
1616
secrets.cfg
1717
.mypy_cache/
18+
.hypothesis/

‎CONTRIBUTING.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,36 @@
2828
* The fastest way to run the tests is to bypass `tox` and run `python3 -m tornado.test`.
2929
To run a subset of tests, add a module, class, or method name to the command line:
3030
`python3 -m tornado.test.httputil_test`.
31+
* `tox -e coverage` measures test coverage across the `-full` configurations and writes an
32+
HTML report to `.tox/coverage-data/html`. CI fails if total coverage drops below the floor
33+
set in `tox.ini`; raise the floor when you raise coverage.
34+
* `tornado/test/property_test.py` holds property-based tests using `hypothesis`, which is
35+
installed in the `-full` configurations (the tests are skipped without it). They are a good
36+
place for invariants of parsers and encoders: round-trips, agreement with a simple model, and
37+
"malformed input raises `HTTPInputError` and nothing else". When one fails, also add the
38+
failing input as an example-based test so it stays covered.
39+
* `tornado/test/api_test.py` compares the public API (names and signatures) to the snapshot in
40+
`tornado/test/api.txt`. When you change the API on purpose, run
41+
`python -m tornado.test.api_test --update` from the repository root (with the `-full`
42+
dependencies installed) and commit the updated snapshot. Imports are left out of the snapshot
43+
unless they have a `# re-exported` comment. `tox -e docs` checks that everything in the docs
44+
is in the snapshot.
45+
* Code that parses untrusted input should have resource-budget tests. `assert_linear_scaling`
46+
in `tornado/test/util.py` checks that time grows linearly with the input, and
47+
`assert_memory_budget` checks peak memory per byte of input (or per byte of a configured
48+
limit such as `max_body_size`). Look for the worst-case input (many tiny fields, deep
49+
escaping, highly compressible data) and keep sizes small enough that the test takes a
50+
fraction of a second. A fix for a denial-of-service bug should come with such a test, and
51+
the test should fail on the code before the fix.
52+
* The "Compare benchmarks" CI job runs `maint/benchmark/ci_benchmark.py` against the PR and
53+
its base branch and fails if anything is more than 20% slower, or if an adversarial input
54+
exceeds its absolute budget (`BUDGETS` in that file). To run the comparison locally, check
55+
out the base in another directory and run
56+
`python maint/benchmark/compare_benchmarks.py --base ../base --head .`. When you add a
57+
parser or a resource-budget test for one, add a benchmark of its worst-case input too.
58+
* Don't remove a public name, argument or behavior without deprecating it first: it should
59+
emit a `DeprecationWarning` and have a `.. deprecated::` note (saying what to use instead)
60+
for at least one minor release before it goes. See `maint/RELEASING.md`.
3161
* Tests can also be run with the standard library's `unittest` package CLI. This is useful
3262
for integration with some editors. However, this mode skips a few checks, notably the
3363
requirement that tests do not generate any log output without corresponding ExpectLog

‎MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ prune docs/build
33
include tornado/py.typed
44
include tornado/speedups.c
55
include tornado/test/README
6+
include tornado/test/api.txt
67
include tornado/test/csv_translations/fr_FR.csv
78
include tornado/test/gettext_translations/fr_FR/LC_MESSAGES/tornado_test.mo
89
include tornado/test/gettext_translations/fr_FR/LC_MESSAGES/tornado_test.po

‎SECURITY.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,12 @@ A report of a denial-of-service vulnerability should demonstrate either an
2727
input that costs substantially more CPU per byte than this baseline, or one
2828
that blocks the event loop (stalling all other connections) for substantially
2929
longer than this.
30+
31+
## Tests for security fixes
32+
33+
A security fix should have the unit tests appropriate to the code it
34+
changes, and also a test in `tornado/test/security_corpus_test.py`. That
35+
file sends the trigger input of each past vulnerability to a real server
36+
(or explains where the tests are for fixes that can't be reached that
37+
way), and checks only what a client can see, so that its tests keep
38+
working when the code they cover is rewritten.

‎maint/RELEASING.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Releasing Tornado
2+
3+
Notes for maintainers. Pushing a `v*` tag runs the build workflow, which
4+
builds the sdist and wheels and uploads them to PyPI, so everything up to
5+
the tag has to be right before it is pushed.
6+
7+
## Tags and branches
8+
9+
- Each release has a tag of the form `vX.Y.Z` (such as `v6.5.0`). The tag
10+
always has all three numbers, although the trailing `.0` is usually left
11+
out elsewhere. Pre-releases have a suffix such as `a1`, `b1` or `rc1`
12+
(`v6.6.0b1`).
13+
- Final release tags are annotated: create them with `git tag -a` and paste
14+
in the release notes. Pre-release tags don't need to be.
15+
- `branchX.Y` tracks the latest release in each series. Pushes to it run
16+
the build workflow and the downstream testbed
17+
(`.github/workflows/downstream.yml`), which runs the test suites of the
18+
most popular dependents against the built package. These branches also
19+
host each version's documentation on readthedocs.org.
20+
- `stable` tracks the latest release (the same commit as the newest
21+
`branchX.Y`) and is the default version of the docs, so that doc URLs
22+
stay the same across versions.
23+
24+
## Release steps
25+
26+
1. For a patch release, work on the appropriate `branchX.Y` and
27+
cherry-pick from master as needed.
28+
2. Write the release notes in `docs/releases/vX.Y.Z.rst` and link them
29+
from `docs/releases.rst`.
30+
3. Update `version` and `version_info` in `tornado/__init__.py` (the
31+
comments there explain `version_info`).
32+
4. Run `python maint/scripts/pre_tag_checks.py vX.Y.Z` on the commit to be
33+
tagged. It checks the version number and release notes against the tag,
34+
runs the full test suite, lint, docs and coverage, and compares
35+
benchmarks with the previous release.
36+
5. Tag the release (`git tag -a vX.Y.Z` for a final release, pasting in the
37+
release notes), then push the tag (`git push origin vX.Y.Z`) and all
38+
updated branches, including `stable` for a final release. CI builds the
39+
sdist and wheels and uploads them to PyPI.
40+
6. If the release creates a new `branchX.Y`, enable builds for that branch
41+
on readthedocs.org.
42+
7. On master, set the version to the next development version: between
43+
releases it has a `.dev1` suffix (`6.6.0` is followed by `6.7.dev1`), or
44+
occasionally `.dev2` and so on to mark milestones.
45+
8. Announce the release on the
46+
[python-tornado](https://groups.google.com/g/python-tornado) and
47+
[python-tornado-announce](https://groups.google.com/g/python-tornado-announce)
48+
mailing lists, and in a GitHub release.
49+
50+
## Minor releases (X.Y.0)
51+
52+
Ship at least one pre-release (`vX.Y.0b1`, optionally an alpha first and a
53+
release candidate after) and announce it the same way. Leave about two
54+
weeks between the first beta and the final release, so downstream projects
55+
and users have time to report problems.
56+
57+
### Deprecations
58+
59+
A public name, argument or behavior is removed only after at least one
60+
minor release in which it emits a `DeprecationWarning` and is documented
61+
with a `.. deprecated::` note saying what to use instead. The API snapshot
62+
test (`tornado/test/api_test.py`) makes any removal show up in review.
63+
64+
## Patch releases (X.Y.Z)
65+
66+
Patch releases have no beta, so the pre-tag checks are the main
67+
protection against regressions.
68+
69+
## Security releases
70+
71+
Security releases skip the public pre-release, and that is where recent
72+
regressions have come from (6.5.3 was followed by 6.5.4, and 6.5.9 by
73+
6.5.10). GitHub Actions doesn't run in a security advisory's private fork,
74+
so the fix is developed and tested locally, and the public PR, CI run,
75+
build and release happen as fast as possible once the fix is disclosed.
76+
The aim is for that public CI run to be a formality, not the first real
77+
test:
78+
79+
- Before anything is public, prepare the whole release on the fix branch:
80+
the version bump, the release notes, and a branch for each release line
81+
that gets the fix.
82+
- Run `pre_tag_checks.py` on each of those branches. It covers the full
83+
suite with the optional dependencies (including the property tests and
84+
the regression tests from earlier security fixes), lint, docs, coverage
85+
and benchmarks. What it can't cover is Windows, macOS and the wheel
86+
builds, which first run in public CI.
87+
- Keep each fix to the smallest behavior change that closes the hole. When
88+
a fix tightens behavior that applications may rely on, ship a way to opt
89+
out in the same release. Behavior changes are what downstream projects
90+
(tornado-testbed) catch, and for a security release they only see them
91+
after the release.
92+
- Be ready to make a follow-up release within a day, and say so in the
93+
advisory.

0 commit comments

Comments
 (0)