Skip to content

ci(release): the half-hourly nightly check stops paying for an 8 vCPU runner - #28304

Closed
only21mil wants to merge 1 commit into
pingdotgg:mainfrom
only21mil:fix/audit-m027
Closed

only21mil wants to merge 1 commit into
pingdotgg:mainfrom
only21mil:fix/audit-m027

Conversation

@only21mil

@only21mil only21mil commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The release workflow runs on a schedule twice an hour (cron: "8,38 * * * *"), 48 times a day, and every run starts resolve_commit on blacksmith-8vcpu-ubuntu-2404. That job does a sparse checkout of .github/scripts and a few GitHub API calls, and most runs stop there with has_changes=false. So the workflow pays for 48 short 8 vCPU jobs a day, on days with no commits too, for work that needs one core.

Change

resolve_commit runs on blacksmith-2vcpu-ubuntu-2404, the runner the repo already uses for its light jobs, such as changes in ci.yml and deploy_web and deploy_marketing in this workflow. Preflight and the build jobs keep their runners.

Scope

A fix for wasted CI spend, not a product bug: 48 short 8 vCPU jobs a day for work that needs one core. No issue or discussion: this is an obvious one-line runner-size fix that only lowers runner spend. It changes no product behavior and no job output. The 8 vCPU label was never chosen for this work: it came over from the old check_changes job when #10410 renamed it to resolve_commit. t3dotgg's merged #33141 lists the runner size among the things it leaves unchanged and points to this PR for it: "and the runner size (that is #28304)".

Verification

No product runtime path: the evidence is upstream's own job timings and a 2-CPU stand-in.

"Resolve release commit" in the 30 newest completed scheduled runs on main (2026-11-11 16:40 to 2026-11-12 07:24 UTC), from the public Actions API:

Jobs Job time Longest step Went past this job
30, all success on 8 vCPU median 4 s, range 3 to 7 s the script step, 5 s 2 of 30

Stand-in, my own Linux machine limited to 2 CPUs, not the 2 vCPU runner: the checkout's git commands and the step's script at 4e0b528, three times, read-only, took 2.0 to 2.2 s per run, each has_changes=false, the same skip upstream logged.

Not checked:

  • The job on blacksmith-2vcpu-ubuntu-2404 itself. It runs only in this repository, so only a maintainer can run it.
  • Blacksmith billing for this job.
Steps, stand-in method and timing output

Because of the 6-hour release gap, most scheduled runs stop at resolve_commit. The closest runner match is changes in ci.yml, which also does a sparse checkout and runs one script.

On every trigger the job only makes API calls: assertReleaseSource and shouldReleaseNightly on schedule, resolveLatestNightlyCommit and assertCommitOnDefaultBranch for a manual stable release, and the source check alone for tag pushes and other manual runs. Nothing in it installs, builds or compiles, and it keeps its 5-minute timeout.

A GitHub-hosted ubuntu-24.04 runner would cost nothing on a public repo, but the repo's other light jobs run on Blacksmith 2 vCPU, so this follows that.

Steps: Set up job, Set up runner, Checkout (sparse, .github/scripts), Resolve release commit (github-script, GitHub API reads only), Post Checkout, Complete runner, Complete job. Each step's median is 1 s or less.

Stand-in. I ran the checkout's git commands and the step's script at 4e0b528 three times, read-only, under taskset -c 0,1 and a 4 GiB cap: 2.0 to 2.2 s per run, each has_changes=false, the same skip upstream logged. Differences from the runner: Node 22 with @actions/github 6.0.1 in place of actions/github-script on Node 24, no retries, and no runner setup, which needs the Blacksmith image.

jobs: 30 ("Resolve release commit", scheduled runs on main)
window: 2026-11-11T16:40:26Z to 2026-11-12T07:24:14Z
runs-on labels: blacksmith-8vcpu-ubuntu-2404; conclusions: success
job duration (started_at to completed_at, s): median 4, min 3, max 7
step (median / max seconds across jobs; GitHub records step times to the second):
  1 Set up job: 0 / 2
  2 Set up runner: 0 / 1
  3 Checkout: 1 / 1
  4 Resolve release commit: 1 / 5
  7 Post Checkout: 0 / 1
  8 Complete runner: 0 / 0
  9 Complete job: 0 / 0
longest single step: "Resolve release commit" 5 s

limits: nproc=2 cpus_allowed=0-1 memory.max=4294967296 node=v22.22.1 git=2.53.0
candidate sha (upstream main when run): 4e0b52806be83e66f80839e1e372398307346425
run 1 checkout: 4e0b528 files=32049
run 1 resolve: Nightly v0.1.10-nightly.20261112.4564 was published less than six hours ago. Skipping.
run 1 resolve: output has_changes=false
run 1 resolve: output ref=4e0b52806be83e66f80839e1e372398307346425
run 1 seconds: checkout 1.10, resolve 1.12, both 2.22
run 2 checkout: 4e0b528 files=32049
run 2 resolve: Nightly v0.1.10-nightly.20261112.4564 was published less than six hours ago. Skipping.
run 2 resolve: output has_changes=false
run 2 resolve: output ref=4e0b52806be83e66f80839e1e372398307346425
run 2 seconds: checkout 1.08, resolve 0.91, both 1.99
run 3 checkout: 4e0b528 files=32049
run 3 resolve: Nightly v0.1.10-nightly.20261112.4564 was published less than six hours ago. Skipping.
run 3 resolve: output has_changes=false
run 3 resolve: output ref=4e0b52806be83e66f80839e1e372398307346425
run 3 seconds: checkout 1.09, resolve 0.91, both 2.00
rc=0
2026-11-12T07:37:18Z

Prior PRs

Theo Browne's #7284 (closed) right-sized six runners in one bundle, three scale-ups and three trims, and was closed because the bundle as a whole added spend. It did not touch this job, then named check_changes. This PR changes one runner and only lowers spend.

Generated with Claude Opus 5.5 in T3 Code Nightly

@github-actions github-actions Bot added vouch:unvouched Author is not yet trusted in the VOUCHED list. size:XS 0-9 changed lines (additions + deletions). labels Oct 31, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 44c37b8

Macroscope's review found this PR approvable — Moves the lightweight release-resolution job to a smaller runner while leaving builds, publishing and release safeguards unchanged. The limited operational downside is a delayed or failed gate, recoverable by restoring the runner and rerunning.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The resolve_commit job in the release workflow now uses blacksmith-2vcpu-ubuntu-2404 instead of blacksmith-8vcpu-ubuntu-2404. Its timeout and release-commit resolution steps are unchanged.

Changes: Release workflow

Layer / File(s) Summary
Resolve commit runner
.github/workflows/release.yml
The resolve_commit job now uses the 2-vCPU runner. Its timeout and release-commit resolution steps are unchanged.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Suggested reviewers: t3dotgg, juliusmarminge

Merge Risk: Minimal · up to 44c37

The release commit resolver uses a smaller runner. No actionable merge-blocking risk is established.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly identifies the main change: reducing the release workflow runner from 8 vCPUs to 2 vCPUs. It is concise and uses a conventional commit format.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template and fills every required section. Problem states that resolve_commit runs twice an hour on blacksmith-8vcpu-ubuntu-2404 and does light API work. Change …
Full details: Description check

Explanation

The description follows the repository template and fills every required section. Problem states that resolve_commit runs twice an hour on blacksmith-8vcpu-ubuntu-2404 and does light API work. Change explains the switch to blacksmith-2vcpu-ubuntu-2404 and names the light-job runner the repo already uses. Scope gives a bug-fix rationale and explains why no issue is linked. Verification reports 30 upstream job timings (median 4 s, max 7 s), a 2-CPU local stand-in run three times, and what was not checked. The author also names the model used, as the template asks. The Verification section is detailed and the Scope section gives a clear reason for having no issue. The template's conventional commit title is not visible in the description body, so the title cannot be confirmed from the provided context.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 15fe283 and 44c37b8.

Files selected for processing: 1

  • .github/workflows/release.yml

Limit details: You’ve used all 10 included reviews currently available.

@only21mil

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

I am closing this: it changes what a CI job costs and does not fix a bug, and this repository now takes only bug fixes from outside contributors.

@only21mil only21mil closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS 0-9 changed lines (additions + deletions). vouch:unvouched Author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant