Skip to content

Commit 31b0e4f

Browse files
t3dotggclaude
andcommitted
fix(server): reuse the shared relay retry and response helpers
Main added relayResponse.ts with one place that maps relay status codes and decides which failures retry. The recovery and registration paths now use it instead of inline predicates, so a 403 is permanent there too. The docs sections for cleanup and recovery are restored on top of the rewritten connect, remote access, and release pages. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent eb662c4 commit 31b0e4f

6 files changed

Lines changed: 91 additions & 17 deletions

File tree

‎apps/server/src/cloud/http.test.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -993,8 +993,8 @@ describe("releaseManagedTunnelOnShutdown", () => {
993993

994994
it.effect.each([
995995
{ status: 401, errorTag: "EnvironmentHttpUnauthorizedError" },
996-
{ status: 403, errorTag: "EnvironmentHttpUnauthorizedError" },
997-
{ status: 409, errorTag: "EnvironmentHttpConflictError" },
996+
{ status: 403, errorTag: "EnvironmentHttpForbiddenError" },
997+
{ status: 409, errorTag: "EnvironmentHttpBadRequestError" },
998998
])("preserves a permanent $status relay recovery failure", ({ status, errorTag }) => {
999999
const { store } = makeMemorySecretStore([
10001000
[CLOUD_ENDPOINT_RUNTIME_CONFIG, "old-config"],

‎apps/server/src/cloud/http.ts‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ import {
9696
import * as CliTokenManager from "./CliTokenManager.ts";
9797
import { getOrCreateEnvironmentKeyPairFromSecretStore } from "./environmentKeys.ts";
9898
import { traceRelayRequest } from "./traceRelayRequest.ts";
99-
import { filterRelayResponse, relayRequestError } from "./relayResponse.ts";
99+
import { filterRelayResponse, relayRequestError, shouldRetryCloudLink } from "./relayResponse.ts";
100100

101101
const CLOUD_MINT_NONCE_PREFIX = "cloud-mint-nonce-";
102102
const CLOUD_MINT_JTI_PREFIX = "cloud-mint-jti-";
@@ -699,9 +699,8 @@ const cloudRelayConfigHandler = Effect.fn("environment.cloud.relayConfig")(
699699
Effect.retry({
700700
times: 2,
701701
while: (error) =>
702-
error._tag !== "EnvironmentCloudEndpointUnavailableError" &&
703-
error._tag !== "EnvironmentHttpUnauthorizedError" &&
704-
error._tag !== "EnvironmentHttpConflictError",
702+
shouldRetryCloudLink(error) &&
703+
error._tag !== "EnvironmentCloudEndpointUnavailableError",
705704
}),
706705
);
707706
if (registration.status === "superseded") {

‎apps/server/src/server.ts‎

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -722,9 +722,7 @@ export const makeServerLayer = Layer.unwrap(
722722
}).pipe(
723723
Effect.retry({
724724
while: (error) =>
725-
error._tag !== "EnvironmentHttpBadRequestError" &&
726-
error._tag !== "EnvironmentHttpUnauthorizedError" &&
727-
error._tag !== "EnvironmentHttpConflictError" &&
725+
shouldRetryCloudLink(error) &&
728726
error._tag !== "EnvironmentCloudEndpointUnavailableError",
729727
schedule: Schedule.exponential("1 second").pipe(
730728
Schedule.modifyDelay(({ duration }) =>
@@ -772,10 +770,8 @@ export const makeServerLayer = Layer.unwrap(
772770
retryRuntimeFailures: true,
773771
}),
774772
(error) =>
775-
error._tag !== "EnvironmentCloudEndpointUnavailableError" &&
776-
error._tag !== "EnvironmentHttpBadRequestError" &&
777-
error._tag !== "EnvironmentHttpUnauthorizedError" &&
778-
error._tag !== "EnvironmentHttpConflictError",
773+
shouldRetryCloudLink(error) &&
774+
error._tag !== "EnvironmentCloudEndpointUnavailableError",
779775
).pipe(
780776
Effect.tap((result) =>
781777
result.status === "ready"
@@ -808,10 +804,7 @@ export const makeServerLayer = Layer.unwrap(
808804
localOrigin,
809805
).pipe(
810806
Effect.retry({
811-
while: (error) =>
812-
error._tag !== "EnvironmentHttpBadRequestError" &&
813-
error._tag !== "EnvironmentHttpUnauthorizedError" &&
814-
error._tag !== "EnvironmentHttpConflictError",
807+
while: shouldRetryCloudLink,
815808
schedule: Schedule.exponential("1 second").pipe(
816809
Schedule.modifyDelay(({ duration }) =>
817810
Effect.succeed(Duration.min(duration, Duration.seconds(30))),

‎docs/internals/t3-connect.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,35 @@ teardown, because a database failure must leave the active link usable. Failed
6565
teardown retains enough state to retry. See the
6666
[managed endpoint lifecycle](../../infra/relay/src/environments/ManagedEndpointProvider.ts).
6767

68+
## Idle tunnels are reclaimed and recovered
69+
70+
Cloudflare bills a tunnel whether or not a connector is attached, so a laptop
71+
that sleeps with a linked environment leaves a paid tunnel behind. The relay's
72+
five-minute maintenance job can reclaim those tunnels. `RELAY_TUNNEL_CLEANUP_MODE`
73+
selects `off`, `dry-run`, or `enabled`, with `off` as the default. A candidate is
74+
a same-stage tunnel that Cloudflare reports down for at least five minutes, or
75+
one that never connected and is at least that old.
76+
77+
Cleanup deletes only tunnels whose host has registered recovery, or tunnels with
78+
no allocation record at all. Allocations without recovery registration belong to
79+
hosts that cannot replace a deleted tunnel and are left alone. Allocations with
80+
no recorded tunnel ID, or a different tunnel ID, are skipped because a provision
81+
may own them. Each sweep is bounded: at most ten list requests, 100 deletions, a
82+
two-minute deadline, and an early stop on a Cloudflare rate limit. See the
83+
[reaper](../../infra/relay/src/environments/ManagedEndpointReaper.ts).
84+
85+
A host registers recovery at startup by sending its tunnel ID and loopback
86+
origin with a short-lived signature from the environment key. Registration
87+
touches Cloudflare only when the local port changed. The host stores a
88+
confirmed-origin marker with the connector config, and a later boot starts the
89+
connector before registration only when that marker matches the current config
90+
and port. If the connector exits, or `cloudflared` reports repeated tunnel
91+
authorization rejections, the host asks the relay for a replacement. The relay
92+
provisions under the same allocation, so the hostname and DNS record survive
93+
and clients keep their bindings. Every mutation on an allocation bumps its
94+
`generation`, and deletion locks the row at the generation it claimed, so a
95+
host that reconnects mid-sweep wins.
96+
6897
## OAuth traps
6998

7099
Interactive clients and the headless CLI use the same Clerk application but

‎docs/operations/release.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,53 @@ Developers deploy personal stages locally rather than through pull-request autom
113113
vp run --filter t3code-relay deploy -- --stage "$USER" --env-file .env.local
114114
```
115115

116+
### Managed tunnel cleanup rollout
117+
118+
Keep `RELAY_TUNNEL_CLEANUP_MODE=off` for the first production deploy. That deploy applies the
119+
nullable allocation migration and adds the recovery endpoints. Web and mobile clients need no
120+
coordinated release. CLI and desktop server builds must reach users before cleanup is enabled,
121+
because those builds register recovery and replace a deleted tunnel after wake.
122+
123+
1. Deploy the relay and migration with cleanup `off`.
124+
2. Release the server build and confirm current hosts register recovery. Older hosts stay marked
125+
legacy and are never candidates.
126+
3. Set `dry-run`, deploy, and read the sweep counters (`scanned`, `wouldDelete`, `skippedLegacy`,
127+
`failed`, `truncated`) across several sweeps.
128+
4. Run the disposable-host canary below.
129+
5. Set `enabled` only after the canary recovers without a server restart.
130+
131+
The job runs every five minutes with a five-minute grace period, so a candidate is usually removed
132+
five to ten minutes after it goes down. One sweep attempts at most 100 deletions, so a backlog takes
133+
longer.
134+
135+
To roll back, set cleanup to `off` and deploy the relay before downgrading any host. Keep the
136+
recovery endpoints deployed while current server builds are in use. The nullable columns can stay.
137+
138+
### Disposable-host canary
139+
140+
This test has not been run against a real Cloudflare account. Run it against a disposable relay
141+
stage, test Cloudflare account, disposable host, and disposable T3 home. Keep production cleanup at
142+
`off` or `dry-run` until it passes. Do not stop a daily-use T3 server.
143+
144+
1. Deploy the disposable stage with cleanup `dry-run`. Link a first disposable environment through
145+
web or mobile settings and confirm its tunnel is healthy and recovery is registered.
146+
2. Stop that host and restart the same T3 home on a different local port. Confirm the public
147+
hostname reaches the new port and sends nothing to the old one.
148+
3. Link a second disposable environment with a server build that predates recovery registration.
149+
Capture its managed `cloudflared` child PID, confirm it belongs to that host, and pause only that
150+
child with `kill -STOP <legacy-pid>`. Wait until Cloudflare reports it down for over five minutes.
151+
4. Capture the first environment's `cloudflared` child PID from its server logs, confirm ownership,
152+
and pause it with `kill -STOP <first-pid>`. Wait until Cloudflare reports it down for over five
153+
minutes.
154+
5. Confirm dry-run counts the first tunnel in `wouldDelete` and the second in `skippedLegacy`.
155+
6. Set cleanup `enabled` on the disposable stage and deploy. Confirm in the test Cloudflare account
156+
that the first tunnel is deleted and the legacy tunnel still exists.
157+
7. Resume the first child with `kill -CONT <first-pid>`. Confirm the running server detects the
158+
repeated rejection, requests recovery, and becomes reachable at the same hostname without a
159+
restart.
160+
8. Resume the legacy child with `kill -CONT <legacy-pid>` and confirm its tunnel reconnects.
161+
9. Repeat with a physical sleep and wake cycle on a disposable laptop before broad rollout.
162+
116163
## Marketing site deployment
117164

118165
After a nightly release is published, the release workflow deploys the same commit

‎docs/user/remote-access.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,12 @@ page, or **Settings → T3 Connect** on mobile, and choose **Deregister**. This
157157
revokes its cloud access and frees its host space even when the environment is
158158
offline or has been wiped.
159159

160+
When idle tunnel cleanup is enabled, T3 Connect removes a linked environment's
161+
tunnel after it stays offline for several minutes. The environment stays linked
162+
and keeps the same address. When the host starts again or wakes, T3 Connect
163+
creates a replacement tunnel on its own. You do not need to pair again. Cleanup
164+
usually runs five to ten minutes after the tunnel goes down.
165+
160166
On a command-line host, `t3 connect unlink` disables exposure while retaining
161167
your login; `t3 connect logout` also clears that login. Background-service
162168
[removal](./background-service.md#manage-the-service) is separate.

0 commit comments

Comments
 (0)