|
| 1 | +--- |
| 2 | +name: release-captain |
| 3 | +description: Run the weekly OpenShell release captain checklist. Monday checks the pre-release train, triages failures and the compatibility review, and drafts the coordination post. Tuesday promotes the qualified pre-release after approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release. |
| 4 | +metadata: |
| 5 | + internal: true |
| 6 | +--- |
| 7 | + |
| 8 | +# Release Captain |
| 9 | + |
| 10 | +OpenShell publishes a stable release every Tuesday from a qualified |
| 11 | +pre-release ([RFC 0014](../../../rfc/0014-release-stability/README.md)). The |
| 12 | +release captain keeps the week's pre-release train green, coordinates the last |
| 13 | +merges, cuts the release, and starts the next train. |
| 14 | + |
| 15 | +## How to use this skill |
| 16 | + |
| 17 | +| When | Ask | What happens | |
| 18 | +|---|---|---| |
| 19 | +| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). Then check each weekday that Auto-Tag's pre-release ran and passed (`gh run list --workflow release-prerelease.yml --limit 3`), and fix failures as they happen (Monday step 3) instead of on Monday. | |
| 20 | +| Monday | "Run the release captain Monday checklist." | Monday steps 1–4. | |
| 21 | +| A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 3 and Recovery. | |
| 22 | +| Tuesday | "Cut the release." | Tuesday steps 1–6, ending with starting the next train. | |
| 23 | + |
| 24 | +## Guardrails |
| 25 | + |
| 26 | +Get explicit approval, with the exact command and target SHA, before pushing a |
| 27 | +tag, dispatching, rerunning, or cancelling a workflow, merging or approving a |
| 28 | +PR, or posting in Slack or GitHub. Never repoint a stable tag, delete a `-pre.N` |
| 29 | +tag, or start a second `Pre-release` run for a tag; it has no concurrency guard. |
| 30 | + |
| 31 | +## How the train works |
| 32 | + |
| 33 | +- A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A |
| 34 | + patch train can't take breaking changes to Stable interfaces. |
| 35 | +- `Release Auto-Tag` runs on weekdays at 14:00 Europe/Zurich. If `main` has |
| 36 | + commits since the latest `-pre.N`, it tags the next one and dispatches |
| 37 | + `Pre-release` (`release-prerelease.yml`, about 45 minutes). It skips when the |
| 38 | + train has no `-pre.1` or the base version is already stable. |
| 39 | +- `Pre-release` builds, then publishes and qualifies in parallel, and writes a |
| 40 | + `release-notes` preview artifact. Pushing a `-pre.N` tag by hand starts it. |
| 41 | +- `Release` (`release.yml`) is dispatched by hand with the stable version. It |
| 42 | + picks the highest `-pre.N` on `main` whose whole `Pre-release` run passed, |
| 43 | + creates the stable tag on that commit, and publishes without requalifying. |
| 44 | + |
| 45 | +## Setup |
| 46 | + |
| 47 | +```shell |
| 48 | +export GH_REPO=NVIDIA/OpenShell |
| 49 | +git fetch origin main --tags --force |
| 50 | +PREV=$(git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1) |
| 51 | +git tag -l 'v*' --sort=-v:refname | grep -E -- '-pre\.[0-9]+$' | head -5 |
| 52 | +``` |
| 53 | + |
| 54 | +Find the `Pre-release` run for a tag. A tag pushed by hand runs on the tag; an |
| 55 | +Auto-Tag run is dispatched on `main`: |
| 56 | + |
| 57 | +```shell |
| 58 | +TAG=vX.Y.Z-pre.N |
| 59 | +gh run list --workflow release-prerelease.yml --branch "$TAG" |
| 60 | +gh run list --workflow release-prerelease.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")" |
| 61 | +``` |
| 62 | + |
| 63 | +Preview the candidate `Release` would promote, read-only: |
| 64 | + |
| 65 | +```shell |
| 66 | +VERSION=vX.Y.Z |
| 67 | +wt=$(mktemp -d) |
| 68 | +git worktree add -q --detach "$wt" origin/main |
| 69 | +(cd "$wt" && GITHUB_REPOSITORY=NVIDIA/OpenShell uv run -q --no-project python tasks/scripts/select-release-candidate.py "$VERSION" --branch main | jq .) |
| 70 | +git worktree remove --force "$wt" |
| 71 | +``` |
| 72 | + |
| 73 | +## Monday: pre-flight |
| 74 | + |
| 75 | +1. **Check the candidate.** Run the preview. If the latest `-pre.N` isn't the |
| 76 | + candidate, its run failed; read why: |
| 77 | + |
| 78 | + ```shell |
| 79 | + RUN=<run-id> |
| 80 | + gh run view "$RUN" --json jobs --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out") | "\(.name)\t\(.url)"' |
| 81 | + gh run view "$RUN" --log-failed |
| 82 | + ``` |
| 83 | + |
| 84 | + If the next version has no `-pre.1`, start the train (Tuesday step 6). |
| 85 | + |
| 86 | +2. **Triage the compatibility review** for the candidate's run. It's advisory, |
| 87 | + and a green job doesn't mean it found no breaks: |
| 88 | + |
| 89 | + ```shell |
| 90 | + ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt) |
| 91 | + dir=$(mktemp -d) |
| 92 | + gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir" |
| 93 | + jq '{status, verdict: .review.verdict}' "$dir/report.json" |
| 94 | + cat "$dir/report.md" |
| 95 | + ``` |
| 96 | + |
| 97 | + Only `complete` with `no_unaddressed_breaks` is clean. A real break needs a |
| 98 | + fix, a revert, a minor train, or a maintainer-approved exception with upgrade |
| 99 | + steps in the release notes. |
| 100 | + |
| 101 | +3. **Fix forward.** A failed pre-release doesn't end the train. Find the cause, |
| 102 | + and let the failed run finish rather than cancelling it. |
| 103 | + |
| 104 | + A rerun repeats the same commit and workflow, so it only helps with a |
| 105 | + transient failure (see Recovery). Anything that needs a code or workflow |
| 106 | + fix ships in a new number: that `-pre.N` stays failed, and `Release` skips |
| 107 | + it. Merge the fix, then cut the next `-pre.N` from the tip of `main` after |
| 108 | + approval: |
| 109 | + |
| 110 | + ```shell |
| 111 | + gh workflow run release-auto-tag.yml --ref main |
| 112 | + ``` |
| 113 | + |
| 114 | + Tag a specific commit by hand (as in Tuesday step 6) only when the tip of |
| 115 | + `main` isn't the commit you want. |
| 116 | + |
| 117 | +4. **Draft the coordination post.** Start a thread in the OpenShell channel of |
| 118 | + the CNCF Slack. The release ships from `main` with no release branch, so the |
| 119 | + thread is for getting ready PRs reviewed and merged before the cutoff. List |
| 120 | + approved PRs: |
| 121 | + |
| 122 | + ```shell |
| 123 | + gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"' |
| 124 | + ``` |
| 125 | + |
| 126 | + The release ships the final pre-release's commit, if it passes. Pick a |
| 127 | + cutoff before Tuesday's Auto-Tag run, or a later one after which you tag the |
| 128 | + final pre-release by hand. Don't promise that everything merged by the |
| 129 | + cutoff ships, or name the next version. |
| 130 | + |
| 131 | + ```text |
| 132 | + :thread: OpenShell vX.Y.Z release, Tuesday <date>. I'm release captain this week. |
| 133 | +
|
| 134 | + Status: vX.Y.Z-pre.N (<commit>) passed qualification. <blockers and owners, or "No blockers."> |
| 135 | +
|
| 136 | + Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is <HH:MM UTC> (<local time>) Tuesday. <"Tuesday's automatic pre-release picks up everything merged by then" | "I'll cut the final pre-release after the cutoff">, and vX.Y.Z ships from it if it passes qualification. Anything that misses it goes in the next release. |
| 137 | + ``` |
| 138 | + |
| 139 | + If the pre-release that ships was cut before the posted cutoff, say so in |
| 140 | + the thread, with its commit and time. |
| 141 | + |
| 142 | +## Tuesday: cut the release |
| 143 | + |
| 144 | +1. **Select the final candidate.** For a late cutoff, cut the final `-pre.N` |
| 145 | + after the cutoff as in Monday step 3. When it finishes, run the preview and |
| 146 | + repeat Monday step 2 for its run. |
| 147 | + |
| 148 | +2. **Request approval.** Present the candidate tag, commit, qualification run, |
| 149 | + compatibility assessment, commits since `$PREV`, and release notes preview: |
| 150 | + |
| 151 | + ```shell |
| 152 | + git log --oneline --no-merges "$PREV..<candidate_tag>" |
| 153 | + gh run download "$RUN" -n release-notes -D "$dir/notes" && cat "$dir/notes/release-notes.md" |
| 154 | + ``` |
| 155 | + |
| 156 | + The notes come from each PR's "Release Notes" section. To fix them, edit the |
| 157 | + PR, then preview with |
| 158 | + `gh workflow run release.yml --ref main -f tag="$VERSION" -f notes-only=true`. |
| 159 | + |
| 160 | +3. **Dispatch the release.** After approval: |
| 161 | + |
| 162 | + ```shell |
| 163 | + gh workflow run release.yml --ref main -f tag="$VERSION" |
| 164 | + ``` |
| 165 | + |
| 166 | +4. **Watch and verify.** Check that `Select Qualified Prerelease` names the |
| 167 | + approved candidate, the GitHub Release has assets, no job failed, and the |
| 168 | + wheel reached PyPI: |
| 169 | + |
| 170 | + ```shell |
| 171 | + gh run watch <run-id> --exit-status |
| 172 | + gh release view "$VERSION" --json isPrerelease,publishedAt,assets --jq '{isPrerelease, publishedAt, assets: (.assets | length)}' |
| 173 | + ``` |
| 174 | + |
| 175 | +5. **Close the loop.** Draft a short reply to the coordination thread. Point |
| 176 | + to the release notes rather than repeating them: |
| 177 | + |
| 178 | + ```text |
| 179 | + :white_check_mark: :ship: OpenShell vX.Y.Z is out: https://github.com/NVIDIA/OpenShell/releases/tag/vX.Y.Z |
| 180 | +
|
| 181 | + It shipped from vX.Y.Z-pre.N (<commit>). I'll be starting the <next version>-pre.* jobs shortly for next week. |
| 182 | +
|
| 183 | + Please check the release notes for upgrade notes that operators might care about. I'm also moving everything still open in the X.Y.Z milestone to <next milestone>. |
| 184 | +
|
| 185 | + Thanks everyone! |
| 186 | + ``` |
| 187 | + |
| 188 | + After approval, move the milestone's open items to the next milestone. |
| 189 | + |
| 190 | +6. **Start the next train.** This is this week's captain's last duty. Until the |
| 191 | + next `-pre.1` exists, Auto-Tag skips. Agree with the maintainers on patch |
| 192 | + (the default) or minor |
| 193 | + ([version selection](../../../rfc/0014-release-stability/release-version-selection.md)). |
| 194 | + Seed it on a commit after `$VERSION` with a passing `Release Dev` run: |
| 195 | + |
| 196 | + ```shell |
| 197 | + for sha in $(git rev-list --max-count=10 "$VERSION..origin/main"); do |
| 198 | + state=$(gh run list --workflow release-dev.yml --commit "$sha" --json status,conclusion --jq '.[0] | if .status == "completed" then .conclusion else .status end') |
| 199 | + echo "${sha:0:9} ${state:-none} $(git log -1 --format=%s "$sha")" |
| 200 | + done |
| 201 | + ``` |
| 202 | + |
| 203 | + After approval, push the tag. The `-c` and `/dev/null` keep `git` from |
| 204 | + signing the tag and opening an editor: |
| 205 | + |
| 206 | + ```shell |
| 207 | + NEXT=vX.Y.Z-pre.1 NEXT_SHA=<commit> |
| 208 | + git -c tag.gpgSign=false tag "$NEXT" "$NEXT_SHA" </dev/null |
| 209 | + git push origin "refs/tags/$NEXT" |
| 210 | + ``` |
| 211 | + |
| 212 | + You're done when that `Pre-release` run passes, or its failure is handed to |
| 213 | + next week's captain. |
| 214 | + |
| 215 | +## Recovery |
| 216 | + |
| 217 | +| Symptom | Action | |
| 218 | +|---|---| |
| 219 | +| Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun <run-id> --failed`. `Release` won't select the run until all of it passes. | |
| 220 | +| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Fix it in a new `-pre.N` (Monday step 3). | |
| 221 | +| Code defect or `Cargo Deny` advisory | Rerunning won't help. Fix it in a new `-pre.N` (Monday step 3). | |
| 222 | +| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. | |
| 223 | + |
| 224 | +## Related |
| 225 | + |
| 226 | +- `release-openshell`: the short procedure behind Tuesday steps 1–4. |
| 227 | +- `watch-github-actions`: general `gh run` monitoring and log retrieval. |
| 228 | +- `CI.md`: release workflows, qualification profile, and scanner policies. |
0 commit comments