Skip to content

Commit 0a06e5d

Browse files
afinetoothclaude
andauthored
Smoke test: run weekly and open an issue on failure (#270)
* Smoke test: add weekly schedule The existing `workflow_run` trigger only fires after releases of *this* repo, so it never catches changes underneath us -- a new coverage-reporter release, a runner-image update, or a Homebrew change. Those are precisely the drifts that break the end-user install chain silently. Concrete example: coverage-reporter v0.6.18 shipped tonight and nothing here fired. Verifying it required remembering this workflow existed and dispatching it by hand, which is exactly the kind of step that gets skipped. Adds a weekly cron as a safety net. The job's existing `if:` guard already passes for schedule events (`github.event_name` is 'schedule', so the workflow_run conclusion check is short-circuited), so no other change needed. Chose a weekly schedule over wiring a `repository_dispatch` from coverage-reporter's build.yml: cross-repo dispatch cannot use `secrets.GITHUB_TOKEN` and would require creating, storing, and rotating a PAT. A schedule needs no credentials and additionally covers drift that no coverage-reporter release would trigger -- which is where two of tonight's three breakages actually came from. Cron is set off the hour to avoid GitHub's peak-of-hour scheduling delays. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> * Smoke test: open an issue on failure Email is a weak alarm for a scheduled job. GitHub sends scheduled-run failure notifications only to whoever last edited the cron expression -- not to repo watchers -- and if that person's notification settings are wrong, the failure is silent. We would have automated the check without automating the alert, which is worse than not automating it: it looks watched when it is not. Adds a `report-failure` job that opens an issue when any smoke matrix job fails. Visible, persistent, and independent of anyone's inbox configuration. Details: - Reuses the existing open issue (via the `smoke-test-failure` label) and comments on it, so a recurring weekly failure does not open 52 issues. - Creates the label on first use, so no manual setup is required. - Scoped `permissions: issues: write` on the job rather than the workflow. - The issue body explains what the failure implies (end users are likely broken right now, even though nothing in this repo changed) and lists the usual causes, so whoever picks it up has context rather than a bare link. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 87358ce commit 0a06e5d

1 file changed

Lines changed: 67 additions & 2 deletions

File tree

‎.github/workflows/smoke-test.yml‎

Lines changed: 67 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@ name: Smoke Test (published action)
33
# Verifies the full end-user chain of the *published* v2 tag on every
44
# supported platform: action resolution -> reporter install (Homebrew on
55
# macOS, GitHub-release download on Linux/Windows) -> report upload.
6-
# Runs automatically after each release (once the v2 tag has been moved
7-
# by the Update Major Version Tag workflow).
6+
# Runs automatically after each release of *this* repo (once the v2 tag has
7+
# been moved by the Update Major Version Tag workflow), and weekly on a
8+
# schedule to catch changes underneath us that no release here would trigger.
89
#
910
# Run it manually after any change worth sanity-checking against the
1011
# published action -- e.g. a Homebrew major version, a runner-image
@@ -18,6 +19,12 @@ on:
1819
workflow_run:
1920
workflows: ["Update Major Version Tag"]
2021
types: [completed]
22+
# Weekly safety net. The workflow_run trigger above only fires on *this*
23+
# repo's releases, so it does not catch changes underneath us -- a new
24+
# coverage-reporter release, a runner-image update, or a Homebrew change.
25+
# Those are exactly the drifts that break the end-user chain silently.
26+
schedule:
27+
- cron: "23 12 * * 1" # Mondays 12:23 UTC (off the hour to dodge peak load)
2128

2229
jobs:
2330
smoke:
@@ -56,3 +63,61 @@ jobs:
5663
run: |
5764
which coveralls
5865
coveralls --version
66+
67+
# Email alone is a weak alarm for a scheduled job: GitHub sends scheduled-run
68+
# failure notifications only to whoever last edited the cron, and a wrong
69+
# notification setting fails silently. Open an issue instead -- visible,
70+
# persistent, and independent of anyone's inbox config.
71+
report-failure:
72+
needs: smoke
73+
if: failure()
74+
runs-on: ubuntu-latest
75+
permissions:
76+
issues: write
77+
steps:
78+
- name: Open or update the smoke-test failure issue
79+
env:
80+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
81+
GH_REPO: ${{ github.repository }}
82+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
83+
TRIGGER: ${{ github.event_name }}
84+
run: |
85+
set -euo pipefail
86+
87+
label="smoke-test-failure"
88+
# Ensure the label exists; harmless if it already does.
89+
gh label create "$label" --color B60205 \
90+
--description "Published action failed its end-user smoke test" 2>/dev/null || true
91+
92+
body=$(cat <<EOF
93+
The [smoke test]($RUN_URL) failed (trigger: \`$TRIGGER\`).
94+
95+
This workflow verifies the **published \`@v2\`** end-user chain on every supported
96+
platform: action resolution → reporter install (Homebrew on macOS, GitHub-release
97+
download on Linux/Windows) → report upload.
98+
99+
A failure here means **end users are likely broken right now**, even though nothing
100+
in this repository changed.
101+
102+
Usual causes:
103+
- a new \`coverage-reporter\` release
104+
- a runner-image change (e.g. \`macos-latest\` moving to a new macOS version)
105+
- a Homebrew change (tap trust, formula/linkage rules, bottle availability)
106+
107+
Run: $RUN_URL
108+
EOF
109+
)
110+
111+
# Reuse the open issue if there is one, so a recurring failure does not
112+
# open a new issue every week.
113+
existing=$(gh issue list --label "$label" --state open --limit 1 --json number --jq '.[0].number // empty')
114+
if [ -n "$existing" ]; then
115+
echo "Commenting on existing issue #$existing"
116+
gh issue comment "$existing" --body "$body"
117+
else
118+
echo "Opening a new issue"
119+
gh issue create \
120+
--title "Smoke test failed: published action is not installing correctly" \
121+
--label "$label" \
122+
--body "$body"
123+
fi

0 commit comments

Comments
 (0)