Skip to content

Commit fc46b8c

Browse files
ci(relay): add a forced manual relay deploy (pingdotgg#13550)
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 568c9bc commit fc46b8c

2 files changed

Lines changed: 26 additions & 8 deletions

File tree

‎.github/workflows/deploy-relay.yml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,15 @@ on:
44
push:
55
branches:
66
- main
7+
# Alchemy does not redeploy the Worker when only a Config value read in its
8+
# Init changes (alchemy-run/alchemy#1831), so a changed repository variable
9+
# needs a forced deploy to reach production.
10+
workflow_dispatch:
11+
inputs:
12+
force:
13+
description: Redeploy every resource, including ones with no detected changes
14+
type: boolean
15+
default: true
716

817
permissions:
918
contents: read
@@ -17,6 +26,8 @@ concurrency:
1726
jobs:
1827
deploy_relay:
1928
name: Deploy production relay
29+
# A manual run from another branch would deploy that branch to production.
30+
if: github.ref == 'refs/heads/main'
2031
runs-on: blacksmith-8vcpu-ubuntu-2404
2132
timeout-minutes: 15
2233
environment:
@@ -55,7 +66,7 @@ jobs:
5566
- --filter=t3code-relay...
5667
5768
- name: Deploy production relay stage
58-
run: vp run --filter t3code-relay deploy --stage prod --yes --no-input
69+
run: vp run --filter t3code-relay deploy --stage prod --yes --no-input ${{ inputs.force && '--force' || '' }}
5970
env:
6071
# The PublishClientConfig action writes the client env here instead
6172
# of the repo-root .env; nothing on the runner reads it.

‎docs/operations/release.md‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,11 @@ Required `production` environment secrets:
140140
- `CLERK_SECRET_KEY`
141141
- `APNS_PRIVATE_KEY`
142142

143+
The relay Worker reads these variables and secrets when it is deployed. Alchemy does not redeploy the
144+
Worker when only one of these values changes ([alchemy-run/alchemy#1831](https://github.com/alchemy-run/alchemy/issues/1831)),
145+
so a push to `main` without relay code changes leaves the old value in place. After changing one, run
146+
the **Deploy T3 Connect relay** workflow manually from `main` with **force** checked.
147+
143148
The account-scoped repository credentials are consumed by Alchemy while provisioning relay stages; they
144149
are not bound into the relay Worker. The production deployment uses an Axiom personal access token,
145150
so `AXIOM_ORG_ID` must accompany `AXIOM_TOKEN`. The `prod` stage owns the retained PlanetScale
@@ -163,18 +168,20 @@ because those builds register recovery and replace a deleted tunnel after wake.
163168
1. Deploy the relay and migration with cleanup `off`.
164169
2. Release the server build and confirm current hosts register recovery. Older hosts stay marked
165170
legacy and are never candidates.
166-
3. Set `dry-run`, deploy, and read the sweep counters (`scanned`, `wouldDelete`, `skippedLegacy`,
167-
`skippedOrphan`, `failed`, `truncated`) across several sweeps.
171+
3. Set `dry-run`, run a forced relay deploy, and read the sweep counters (`scanned`, `wouldDelete`,
172+
`skippedLegacy`, `skippedOrphan`, `failed`, `truncated`) across several sweeps. Each sweep records
173+
them, and the active `mode`, as `relay.managed_endpoint_reaper.*` attributes on its
174+
`relay.managed_endpoint_reaper.sweep` span in Axiom.
168175
4. Run the disposable-host canary below.
169176
5. Set `enabled` only after the canary recovers without a server restart.
170177

171178
The job runs every five minutes with a five-minute grace period for tunnels that lost their
172179
connector, so a candidate is usually removed five to ten minutes after it goes down. Tunnels that
173180
never connected wait an hour. One sweep attempts at most 100 deletions, so a backlog takes longer.
174-
`RELAY_TUNNEL_CLEANUP_MODE` is read at deploy time. Changing it, including turning cleanup off during
175-
an incident, needs a relay deploy.
181+
Changing `RELAY_TUNNEL_CLEANUP_MODE`, including turning cleanup off during an incident, needs a forced
182+
relay deploy. Confirm the new `mode` on the next sweep span.
176183

177-
To roll back, set cleanup to `off` and deploy the relay before downgrading any host. Keep the
184+
To roll back, set cleanup to `off` and run a forced relay deploy before downgrading any host. Keep the
178185
recovery endpoints deployed while current server builds are in use. The nullable columns can stay.
179186

180187
### Disposable-host canary
@@ -194,8 +201,8 @@ stage, test Cloudflare account, disposable host, and disposable T3 home. Keep pr
194201
and pause it with `kill -STOP <first-pid>`. Wait until Cloudflare reports it down for over five
195202
minutes.
196203
5. Confirm dry-run counts the first tunnel in `wouldDelete` and the second in `skippedLegacy`.
197-
6. Set cleanup `enabled` on the disposable stage and deploy. Confirm in the test Cloudflare account
198-
that the first tunnel is deleted and the legacy tunnel still exists.
204+
6. Set cleanup `enabled` on the disposable stage and deploy it with `--force`. Confirm in the test
205+
Cloudflare account that the first tunnel is deleted and the legacy tunnel still exists.
199206
7. Resume the first child with `kill -CONT <first-pid>`. Confirm the running server detects the
200207
repeated rejection, requests recovery, and becomes reachable at the same hostname without a
201208
restart.

0 commit comments

Comments
 (0)