Skip to content

Commit 5ec0cb8

Browse files
docs(relay): add the legacy tunnel cleanup rollout runbook
Order the client release before enabling legacy cleanup, point the dry-run at Cloudflare's totals rather than a single sweep's sample, give the Axiom query for the sweep counters, and list when to turn the switch back off. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent f7cef65 commit 5ec0cb8

1 file changed

Lines changed: 21 additions & 1 deletion

File tree

‎docs/operations/release.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,17 +203,37 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the
203203
- On an older build linked from web or mobile, the host stays offline until T3 Code on that computer
204204
is updated.
205205

206+
Ship the web and mobile builds that show the offline reason before enabling legacy cleanup, so a
207+
user whose host is affected sees what to do. The relay adds the `tunnel_released_at` allocation
208+
column in its first deploy with this change; the legacy switch stays `off` until you set it.
209+
206210
1. Run `vp run --filter t3code-relay tunnels:census` with a read-only Cloudflare token. It counts
207211
tunnels in every relay stage. The reaper only sees its own stage's tunnels, so clean up the rest
208212
by hand.
209213
2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`,
210-
`totalDown`, and `totalInactive` on the sweep spans for a day.
214+
`totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the
215+
tunnels a sweep inspected, at most 500 per status, so use `totalDown` and `totalInactive` for the
216+
size of the backlog.
211217
3. Run the legacy steps of the disposable-host canary below.
212218
4. Set the legacy mode to `enabled`. One sweep deletes at most 100 tunnels, four at a time, and
213219
stops starting new deletions after 90 seconds. A backlog of 20,000 takes about 17 hours if each
214220
sweep finishes its 100. Watch `deletedLegacy`, `attempted`, `failed`, and `truncated`; an
215221
`attempted` well under 100 means sweeps are running out of time.
216222

223+
In Axiom, filter the relay traces dataset on `name == "relay.managed_endpoint_reaper.sweep"` and
224+
chart the `attributes.custom.relay.managed_endpoint_reaper.*` fields over time.
225+
226+
Set the legacy mode back to `off` and deploy if any of these happen:
227+
228+
- `failed` stays above a few per sweep. Read the warning log for the Cloudflare error.
229+
- Users report an environment that is offline with the update message after they have updated T3
230+
Code on that computer and restarted it.
231+
- Relay request errors rise while sweeps run. Deletions share the Postgres connection pool with
232+
request handlers.
233+
234+
Turning the switch off stops new deletions. Deleted tunnels stay deleted; their hosts recover as
235+
described above.
236+
217237
### Disposable-host canary
218238

219239
This test has not been run against a real Cloudflare account. Run it against a disposable relay

0 commit comments

Comments
 (0)