docs(release): document admin-created desktop release tags - #1694

…2607)

## Problem

The PR flake check repeated only the tests in files a pull request
edited. If a product change made an unedited test racy, the check
selected nothing. #2000 repeated one emoji journey and skipped the seven
other specs that load `emoji.html`. Five of its flaky journeys were in
those specs.

## What changes

`scripts/flake-check.mjs select` now selects in two tiers:

- **Tier 1, edited tests.** This is exactly the old rule: same
repetitions, both engines. The budget never trims it, even when tier 1
alone goes over.
- **Tier 2, related tests**, run as whole files:
- Browser specs that import a changed helper, or that open a
`tests/fixtures/*.html` page whose module graph holds a changed source
file, stylesheet or patched package. One rolldown build of every spec
and fixture page supplies the graph and writes nothing (1,027 modules in
about 1 s).
- Vitest files related to the changed modules through Vitest's module
graph. This is the graph the pre-push `vitest related` uses.

Picks are ranked by specificity: the smallest route that holds a changed
file comes first, and the cheaper test wins a tie. A test named after a
changed module ranks ahead of that order (`ComposerAttachments.tsx` →
`composer-attachments.spec.mjs`). The step summary marks those rows as a
naming heuristic, not graph evidence. Tier 2 adds picks in rank order
while each lane's predicted wall time (`flake_browser` and
`flake_vitest`) stays within **9 minutes** (`LANE_BUDGET_MINUTES`;
`--budget=<minutes>` overrides it). The 40-journey and 20-Vitest-file
caps still apply, and the pick that crosses a cap still runs.

**Cost table.** `scripts/flake-check-costs.json` is checked in and
written by the new `flake-check costs [run-id…]` subcommand. By default
it reads the latest three green `main` push runs. It reads the required
lanes' Playwright and Vitest reports and stores median passing seconds
per journey title as [Chromium, WebKit], plus a median per Vitest file.
A test missing from the table falls back to its file's median, then to
the engine-wide median or the median of all Vitest files. I checked in
the table instead of fetching timings in CI for three reasons: `select`
stays reproducible from the commit, `flake_select` needs no token or
network, and the artifacts are about 100 MB per run and expire after 7
days. The current table comes from runs 41990511962, 41980968364 and
41979442279 (Nov 5): 1,049 journeys and 964 Vitest files in 181 KB.

**Lane model.** I fit the model to hosted PR runs from Nov 4–5. The
median predicted/actual ratio is 0.95 for the browser lane and 1.02 for
Vitest.

- Browser shard: 105 s of overhead + 1.11 × the shard's summed test time
× 10 repetitions / 2 workers, then × 1.22 for imbalance when there is
more than one shard.
- Vitest batch: 26 s + 5 × (5.7 s + 0.72 × max(sum / 2, slowest file)).

The step summary gains a Tier column. It also shows each lane's
predicted time against the budget, with the edited-only time beside it,
and the related picks dropped for time and for the cap.

`docs/contributing.md` describes the tiers and the budget. It also
corrects "at most eight" shards to sixteen (`MAX_SHARDS`).

## For reviewers: the budget takes the longest fitting prefix

Tier 2 does **not** stop at the first pick that would go over the
budget. It takes the longest rank-order prefix whose predicted lane
fits. Browser shards grow with the selection (one shard per five
journeys), so a longer prefix can predict a *shorter* lane than a
shorter prefix. In the 248-PR replay, stopping at the first overrun
would have trimmed browser picks in 45 PRs. In 44 of those, the cap-only
selection already fit, and 40 predicted a slower lane after trimming.
#2000 would have kept 2 related journeys at 8.6 minutes instead of 46 at
6.2, and lost both of its flaky specs. Ranks are never skipped. The
Vitest lane only grows with each pick, so both rules give the same
result there. A unit test covers the non-monotonic case, and that test
fails under a first-overrun loop.

## Validation

Replays of `select <merge>^1 <merge>` in detached checkouts of each
merge commit. Selection before this PR → after:

| PR | Edited Vitest / journeys | Related journeys | Related Vitest |
Browser lane (min) | Vitest lane (min) | Flaky targets kept |
| --- | --- | --- | --- | --- | --- | --- |
| #2000 0944c1d | 1 / 1 | 0 → 46 (12 specs, incl. reactions-polish,
message-overlays) | 0 → 4 | 6.2 | 1.5 | yes |
| #1361 5992e69 | 2 / 0 | 0 → 40 (incl. composer-attachments, by name)
| 0 → 0 (budget) | 8.0 | 9.4 (edited alone 9.4) | yes |
| #2291 83b72a3 | 2 / 153 | 0 → 0 (budget) | 0 → 18 | 15.3 (edited
alone 15.3) | 6.6 | n/a |
| #1910 e7ae7b1 | 3 / 0 | 0 → 53 | 0 → 10 | 5.7 | 9.0 | n/a |

Lane times are this PR's predictions. Without the budget (the first
commit alone), the Vitest lane would have been 9.8 min for #2000, 16.3
min for #1910 and 6.6 min for #2291, and #2291's browser lane 18.3 min.
`--budget=Infinity` reproduces the first commit's output byte for byte
on all four source PRs. In the first commit's replay, a spec-only PR
(#2557) and a docs-only PR (#1694) selected exactly what the old rule
selected.

- `node --test tests/integration/flake-check.test.mjs`: 20 pass. Biome
is clean, including the table.
- Local `select` time on a laptop: #2000 takes 49 s with a cold Vitest
cache and 6 s warm (13 s before); #1361 takes 63 s cold and 8 s warm (19
s before). CI runs cold, within the select job's 5-minute limit. These
are local measurements, not hosted ones.

## Cost and known gaps

- **Runner-minutes grow about 7.5×.** In a replay of the 248 PRs merged
Nov 2–4, flake runner-minutes per run averaged about 11.4 under the old
rule and about 84 with related selection. Almost every source PR fills
the 40/20 caps, and the caps are what bound the cost. The budget limits
wall time only: against the first commit it moves the mean from 84.6 to
84.0 and the max from 381 to 319 (#2291). Lane wall time stays about 6–7
minutes for most PRs, and the highest predicted lanes fall from 16.7 to
13.7 minutes (browser) and from 16.3 to 10.4 (Vitest). All 11 Vitest
lanes still over 9 minutes are over on edited tests alone. The lanes
stay advisory, and repetitions, timeouts and workers are unchanged.
- **Not yet run in hosted CI.** This PR's own run will be the first.
- **Cost table drift.** Timings go stale until someone reruns
`flake-check costs`.
- **Model error.** Predicted/actual ranges from 0.73 to 1.23 (p10–p90)
for the browser lane and from 0.75 to 1.58 for Vitest. A Vitest
`file:line` filter is charged for its whole file.
- **Full-app journeys.** Journeys that load the full app rather than a
fixture page are reached only through helpers. This misses #1729, #2108
and #1969, which a coverage-map follow-up would address. A patched
package reaches browser specs but not Vitest files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Matt Toohey <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>