Skip to content

Commit 7848790

Browse files
committed
maint: Add pre-tag checks and release instructions
maint/scripts/pre_tag_checks.py checks that the version number and release notes match the tag about to be pushed, then runs the full tests, lint, docs and coverage, and compares benchmarks with the previous release. It passes on the commits of past tags (v6.5.0, v6.5.10, v6.6.0b1, v6.6.0rc1) and reports each mismatch when given the wrong tag. maint/RELEASING.md replaces the Release-instructions wiki page: tags and branches, the release steps (now including the pre-tag checks), pre-releases before every minor release, the deprecation rule (also in CONTRIBUTING.md), and how security releases are prepared and checked before the fix is public. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01TuTKdWgh56FRF7yJqaX9ug (cherry picked from commit 21f4440)
1 parent f14321e commit 7848790

3 files changed

Lines changed: 267 additions & 0 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@
5555
out the base in another directory and run
5656
`python maint/benchmark/compare_benchmarks.py --base ../base --head .`. When you add a
5757
parser or a resource-budget test for one, add a benchmark of its worst-case input too.
58+
* Don't remove a public name, argument or behavior without deprecating it first: it should
59+
emit a `DeprecationWarning` and have a `.. deprecated::` note (saying what to use instead)
60+
for at least one minor release before it goes. See `maint/RELEASING.md`.
5861
* Tests can also be run with the standard library's `unittest` package CLI. This is useful
5962
for integration with some editors. However, this mode skips a few checks, notably the
6063
requirement that tests do not generate any log output without corresponding ExpectLog

‎maint/RELEASING.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Releasing Tornado
2+
3+
Notes for maintainers. Pushing a `v*` tag runs the build workflow, which
4+
builds the sdist and wheels and uploads them to PyPI, so everything up to
5+
the tag has to be right before it is pushed.
6+
7+
## Tags and branches
8+
9+
- Each release has a tag of the form `vX.Y.Z` (such as `v6.5.0`). The tag
10+
always has all three numbers, although the trailing `.0` is usually left
11+
out elsewhere. Pre-releases have a suffix such as `a1`, `b1` or `rc1`
12+
(`v6.6.0b1`).
13+
- Final release tags are annotated: create them with `git tag -a` and paste
14+
in the release notes. Pre-release tags don't need to be.
15+
- `branchX.Y` tracks the latest release in each series. Pushes to it run
16+
the build workflow and the downstream testbed
17+
(`.github/workflows/downstream.yml`), which runs the test suites of the
18+
most popular dependents against the built package. These branches also
19+
host each version's documentation on readthedocs.org.
20+
- `stable` tracks the latest release (the same commit as the newest
21+
`branchX.Y`) and is the default version of the docs, so that doc URLs
22+
stay the same across versions.
23+
24+
## Release steps
25+
26+
1. For a patch release, work on the appropriate `branchX.Y` and
27+
cherry-pick from master as needed.
28+
2. Write the release notes in `docs/releases/vX.Y.Z.rst` and link them
29+
from `docs/releases.rst`.
30+
3. Update `version` and `version_info` in `tornado/__init__.py` (the
31+
comments there explain `version_info`).
32+
4. Run `python maint/scripts/pre_tag_checks.py vX.Y.Z` on the commit to be
33+
tagged. It checks the version number and release notes against the tag,
34+
runs the full test suite, lint, docs and coverage, and compares
35+
benchmarks with the previous release.
36+
5. Tag the release (`git tag -a vX.Y.Z` for a final release, pasting in the
37+
release notes), then push the tag (`git push origin vX.Y.Z`) and all
38+
updated branches, including `stable` for a final release. CI builds the
39+
sdist and wheels and uploads them to PyPI.
40+
6. If the release creates a new `branchX.Y`, enable builds for that branch
41+
on readthedocs.org.
42+
7. On master, set the version to the next development version: between
43+
releases it has a `.dev1` suffix (`6.6.0` is followed by `6.7.dev1`), or
44+
occasionally `.dev2` and so on to mark milestones.
45+
8. Announce the release on the
46+
[python-tornado](https://groups.google.com/g/python-tornado) and
47+
[python-tornado-announce](https://groups.google.com/g/python-tornado-announce)
48+
mailing lists, and in a GitHub release.
49+
50+
## Minor releases (X.Y.0)
51+
52+
Ship at least one pre-release (`vX.Y.0b1`, optionally an alpha first and a
53+
release candidate after) and announce it the same way. Leave about two
54+
weeks between the first beta and the final release, so downstream projects
55+
and users have time to report problems.
56+
57+
### Deprecations
58+
59+
A public name, argument or behavior is removed only after at least one
60+
minor release in which it emits a `DeprecationWarning` and is documented
61+
with a `.. deprecated::` note saying what to use instead. The API snapshot
62+
test (`tornado/test/api_test.py`) makes any removal show up in review.
63+
64+
## Patch releases (X.Y.Z)
65+
66+
Patch releases have no beta, so the pre-tag checks are the main
67+
protection against regressions.
68+
69+
## Security releases
70+
71+
Security releases skip the public pre-release, and that is where recent
72+
regressions have come from (6.5.3 was followed by 6.5.4, and 6.5.9 by
73+
6.5.10). GitHub Actions doesn't run in a security advisory's private fork,
74+
so the fix is developed and tested locally, and the public PR, CI run,
75+
build and release happen as fast as possible once the fix is disclosed.
76+
The aim is for that public CI run to be a formality, not the first real
77+
test:
78+
79+
- Before anything is public, prepare the whole release on the fix branch:
80+
the version bump, the release notes, and a branch for each release line
81+
that gets the fix.
82+
- Run `pre_tag_checks.py` on each of those branches. It covers the full
83+
suite with the optional dependencies (including the property tests and
84+
the regression tests from earlier security fixes), lint, docs, coverage
85+
and benchmarks. What it can't cover is Windows, macOS and the wheel
86+
builds, which first run in public CI.
87+
- Keep each fix to the smallest behavior change that closes the hole. When
88+
a fix tightens behavior that applications may rely on, ship a way to opt
89+
out in the same release. Behavior changes are what downstream projects
90+
(tornado-testbed) catch, and for a security release they only see them
91+
after the release.
92+
- Be ready to make a follow-up release within a day, and say so in the
93+
advisory.

‎maint/scripts/pre_tag_checks.py‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
#!/usr/bin/env python
2+
"""Checks to run on a release branch before pushing a release tag.
3+
4+
Usage (from the root of the repo, on the commit to be tagged):
5+
6+
python maint/scripts/pre_tag_checks.py v6.7.0b1
7+
python maint/scripts/pre_tag_checks.py v6.6.1 --quick # metadata only
8+
9+
This matters most for security releases, which skip the betas that would
10+
otherwise catch regressions. See maint/RELEASING.md for the parts of the
11+
process that can't be automated.
12+
13+
The checks:
14+
15+
* The working tree is clean.
16+
* ``tornado.version`` and ``tornado.version_info`` match the tag.
17+
* The release notes exist, are in docs/releases.rst, and (for a final
18+
release) have a date.
19+
* Unless ``--quick``: the full test suite with the optional dependencies
20+
(property tests and the curl client included), lint, docs and coverage,
21+
via tox.
22+
* Unless ``--quick``: maint/benchmark/compare_benchmarks.py against the
23+
previous release tag (``--previous`` overrides the guess).
24+
"""
25+
26+
import argparse
27+
import os
28+
import re
29+
import subprocess
30+
import sys
31+
import tempfile
32+
from typing import NamedTuple
33+
34+
_TAG_RE = re.compile(r"v(\d+)\.(\d+)\.(\d+)(?:(a|b|rc)(\d+))?")
35+
_PRERELEASE_INFO = {"a": -99, "b": -98, "rc": -97}
36+
37+
38+
class Version(NamedTuple):
39+
version: str
40+
version_info: tuple[int, int, int, int]
41+
notes: str
42+
43+
44+
def expected_version(tag: str) -> Version:
45+
"""Returns what tornado/__init__.py and the notes should say for ``tag``.
46+
47+
>>> expected_version("v6.5.0")
48+
Version(version='6.5', version_info=(6, 5, 0, 0), notes='v6.5.0')
49+
>>> expected_version("v6.5.10")
50+
Version(version='6.5.10', version_info=(6, 5, 10, 0), notes='v6.5.10')
51+
>>> expected_version("v6.6.0b1")
52+
Version(version='6.6b1', version_info=(6, 6, 0, -98), notes='v6.6.0')
53+
"""
54+
m = _TAG_RE.fullmatch(tag)
55+
if m is None:
56+
raise ValueError(f"not a release tag: {tag!r}")
57+
major, minor, patch = int(m[1]), int(m[2]), int(m[3])
58+
base = f"{major}.{minor}" if patch == 0 else f"{major}.{minor}.{patch}"
59+
if m[4]:
60+
# Pre-releases have used the same fourth element for every
61+
# number so far (6.6a1 and 6.5b1 alike), so only the kind is
62+
# checked.
63+
return Version(
64+
base + m[4] + m[5],
65+
(major, minor, patch, _PRERELEASE_INFO[m[4]]),
66+
f"v{major}.{minor}.{patch}",
67+
)
68+
return Version(base, (major, minor, patch, 0), tag)
69+
70+
71+
def git(*args: str) -> str:
72+
return subprocess.run(
73+
["git", *args], check=True, capture_output=True, text=True
74+
).stdout.strip()
75+
76+
77+
def check_metadata(tag: str) -> list[str]:
78+
errors = []
79+
if git("status", "--porcelain"):
80+
errors.append("the working tree has uncommitted changes")
81+
82+
expected = expected_version(tag)
83+
with open("tornado/__init__.py") as f:
84+
init = f.read()
85+
version = re.search(r'^version = "(.*)"$', init, re.M)
86+
info = re.search(r"^version_info = (\(.*\))$", init, re.M)
87+
if version is None or version[1] != expected.version:
88+
errors.append(
89+
f"tornado.version is {version and version[1]!r}, expected "
90+
f"{expected.version!r}"
91+
)
92+
if info is None or info[1] != repr(expected.version_info):
93+
errors.append(
94+
f"tornado.version_info is {info and info[1]}, expected "
95+
f"{expected.version_info}"
96+
)
97+
98+
notes = f"docs/releases/{expected.notes}.rst"
99+
if not os.path.exists(notes):
100+
errors.append(f"{notes} does not exist")
101+
else:
102+
with open("docs/releases.rst") as f:
103+
if f"releases/{expected.notes}\n" not in f.read():
104+
errors.append(f"{notes} is not listed in docs/releases.rst")
105+
with open(notes) as f:
106+
if expected.version_info[3] == 0 and re.search(r"\bXX\b", f.read()):
107+
errors.append(f"{notes} still has a placeholder (XX) date")
108+
return errors
109+
110+
111+
def previous_tag(tag: str) -> str:
112+
# The highest final release below tag. Release branches aren't merged
113+
# back, so this can't be found from the commit history.
114+
target = expected_version(tag).version_info[:3]
115+
finals = []
116+
for t in git("tag", "--list", "v*").split():
117+
m = _TAG_RE.fullmatch(t)
118+
if m and not m[4]:
119+
v = (int(m[1]), int(m[2]), int(m[3]))
120+
if v < target:
121+
finals.append((v, t))
122+
if not finals:
123+
raise ValueError("no previous release tag found")
124+
return max(finals)[1]
125+
126+
127+
def run(*cmd: str) -> bool:
128+
print("+", " ".join(cmd), flush=True)
129+
return subprocess.run(cmd).returncode == 0
130+
131+
132+
def main() -> int:
133+
parser = argparse.ArgumentParser()
134+
parser.add_argument("tag", help="the tag to be pushed, such as v6.7.0b1")
135+
parser.add_argument("--previous", help="release to compare benchmarks with")
136+
parser.add_argument(
137+
"--quick", action="store_true", help="skip the tests and benchmarks"
138+
)
139+
args = parser.parse_args()
140+
141+
errors = check_metadata(args.tag)
142+
for e in errors:
143+
print("error:", e)
144+
if not args.quick:
145+
if not run(sys.executable, "-m", "tox", "-e", "py3-full,lint,docs,coverage"):
146+
errors.append("tox failed")
147+
previous = args.previous or previous_tag(args.tag)
148+
with tempfile.TemporaryDirectory() as tmp:
149+
base = os.path.join(tmp, previous)
150+
git("worktree", "add", "--detach", base, previous)
151+
try:
152+
if not run(
153+
sys.executable,
154+
"maint/benchmark/compare_benchmarks.py",
155+
"--base",
156+
base,
157+
"--head",
158+
".",
159+
):
160+
errors.append(f"benchmarks regressed since {previous}")
161+
finally:
162+
git("worktree", "remove", "--force", base)
163+
if errors:
164+
print(f"\n{len(errors)} problem(s); do not tag {args.tag} yet.")
165+
return 1
166+
print(f"\nAll checks passed for {args.tag}. See maint/RELEASING.md for the rest.")
167+
return 0
168+
169+
170+
if __name__ == "__main__":
171+
sys.exit(main())

0 commit comments

Comments
 (0)