Skip to content

Commit 11de284

Browse files
radroidclaude
andcommitted
feat(t3x): detect a push connection that is open but dead
The failure this guards is not a connection that errors — that case announces itself and a reconnect ladder handles it. It is the connection that survives laptop sleep, a NAT rebind or a Tailscale route change and comes back with the socket open, no error event, and no bytes ever again. Nothing in a reconnect-on-error design ever fires for it; the app just stops hearing about updates while reporting itself connected. This fork has already lived that exact shape once: issue #41, 103 minutes dark, silent throughout. Two independent mechanisms, because one is not trustworthy enough: A watchdog at 2.5 missed heartbeats makes detection fast. It only means anything because the relay sends `: ping` on a fixed interval — without those bytes, a genuinely idle channel is indistinguishable from a corpse, and any watchdog built on it either fires constantly or never. A floor poll of /latest every 15 minutes makes the worst case BOUNDED even if the watchdog is wrong. That is the whole difference between "we might be a bit stale" and an outage of unbounded length, so the test asserts the two intervals stay in the right order rather than leaving it to a comment. Reconnect backoff uses full jitter: every app on the fleet wakes on the same events — a relay deploy, Monday morning lids opening — and lockstep reconnects would turn a blip into a thundering herd against one Durable Object. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 8751020 commit 11de284

2 files changed

Lines changed: 179 additions & 0 deletions

File tree

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
import { describe, expect, it } from "vite-plus/test";
2+
3+
import {
4+
assessConnection,
5+
FLOOR_POLL_INTERVAL_MS,
6+
reconnectDelayMs,
7+
RECONNECT_MAX_DELAY_MS,
8+
SERVER_HEARTBEAT_INTERVAL_MS,
9+
WATCHDOG_TIMEOUT_MS,
10+
} from "./connectionHealth.ts";
11+
12+
const NOW = 1_800_000_000_000;
13+
14+
describe("assessConnection", () => {
15+
it("is healthy while heartbeats keep arriving", () => {
16+
expect(assessConnection({ lastByteAtMs: NOW - 5_000, nowMs: NOW })).toEqual({ kind: "healthy" });
17+
});
18+
19+
it("tolerates a single dropped heartbeat", () => {
20+
// One missed ping under load must not churn a working connection.
21+
const silentFor = SERVER_HEARTBEAT_INTERVAL_MS * 2;
22+
expect(assessConnection({ lastByteAtMs: NOW - silentFor, nowMs: NOW })).toEqual({
23+
kind: "healthy",
24+
});
25+
});
26+
27+
it("declares the stream stale once it passes the watchdog timeout", () => {
28+
// THE case this module exists for: socket still open, no error ever raised, no bytes since
29+
// the laptop woke up. Nothing else in the system notices this.
30+
const silentFor = WATCHDOG_TIMEOUT_MS + 1;
31+
expect(assessConnection({ lastByteAtMs: NOW - silentFor, nowMs: NOW })).toEqual({
32+
kind: "stale",
33+
silentForMs: silentFor,
34+
});
35+
});
36+
37+
it("treats a long sleep as stale", () => {
38+
const silentFor = 4 * 60 * 60_000;
39+
expect(assessConnection({ lastByteAtMs: NOW - silentFor, nowMs: NOW }).kind).toBe("stale");
40+
});
41+
42+
it("does not fire exactly at the boundary", () => {
43+
expect(assessConnection({ lastByteAtMs: NOW - WATCHDOG_TIMEOUT_MS, nowMs: NOW }).kind).toBe(
44+
"healthy",
45+
);
46+
});
47+
48+
it("survives a clock that jumps backwards", () => {
49+
// NTP correction or a timezone change can make `now` precede the last byte. A negative
50+
// silence must never be read as "very stale" and trigger a reconnect storm.
51+
expect(assessConnection({ lastByteAtMs: NOW + 60_000, nowMs: NOW }).kind).toBe("healthy");
52+
});
53+
});
54+
55+
describe("watchdog and floor-poll relationship", () => {
56+
it("detects a dead stream long before the floor poll would", () => {
57+
// Both must exist. The watchdog makes detection fast; the floor poll makes the worst case
58+
// BOUNDED even if the watchdog is somehow wrong. Losing the floor poll is what turns a missed
59+
// update into an outage of unbounded length.
60+
expect(WATCHDOG_TIMEOUT_MS).toBeLessThan(FLOOR_POLL_INTERVAL_MS);
61+
});
62+
63+
it("gives the server at least two heartbeats before giving up", () => {
64+
expect(WATCHDOG_TIMEOUT_MS).toBeGreaterThanOrEqual(SERVER_HEARTBEAT_INTERVAL_MS * 2);
65+
});
66+
});
67+
68+
describe("reconnectDelayMs", () => {
69+
it("grows with each attempt", () => {
70+
const noJitter = () => 1;
71+
expect(reconnectDelayMs(0, noJitter)).toBeLessThan(reconnectDelayMs(3, noJitter));
72+
});
73+
74+
it("is capped", () => {
75+
expect(reconnectDelayMs(50, () => 1)).toBe(RECONNECT_MAX_DELAY_MS);
76+
});
77+
78+
it("applies full jitter", () => {
79+
// Every desktop app on the fleet wakes on the same events — a relay deploy, Monday morning
80+
// lids opening. Unjittered backoff would reconnect them in lockstep against a single Durable
81+
// Object.
82+
expect(reconnectDelayMs(5, () => 0)).toBe(0);
83+
expect(reconnectDelayMs(5, () => 0.5)).toBeLessThan(reconnectDelayMs(5, () => 1));
84+
});
85+
86+
it("never returns a negative delay for a negative attempt", () => {
87+
expect(reconnectDelayMs(-3, () => 1)).toBeGreaterThan(0);
88+
});
89+
});
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
/**
2+
* Deciding when a push connection that *looks* alive is actually dead.
3+
*
4+
* This is the most important reliability rule in update delivery, and it is deliberately pure so
5+
* it can be tested without a socket.
6+
*
7+
* The motivating failure is not "the connection errored" — that case is easy and self-announcing.
8+
* It is the connection that survives laptop sleep, or a NAT rebind, or a Tailscale route change,
9+
* and comes back with the socket open, no error event, and no bytes ever again. A
10+
* reconnect-on-error ladder never fires for it. The app simply stops hearing about updates, for an
11+
* unbounded length of time, while reporting itself connected.
12+
*
13+
* This fork has already lived that failure once: issue #41, 103 minutes dark, silent throughout.
14+
*/
15+
16+
/** Server sends `: ping` at this interval. Must match `HEARTBEAT_INTERVAL_MS` in the relay. */
17+
export const SERVER_HEARTBEAT_INTERVAL_MS = 20_000;
18+
19+
/**
20+
* Declare the stream dead after 2.5 missed heartbeats.
21+
*
22+
* Two would be too eager — one dropped ping under load would churn the connection. Three or more
23+
* pushes detection past a minute for no benefit, given reconnecting is nearly free.
24+
*/
25+
export const WATCHDOG_TIMEOUT_MS = SERVER_HEARTBEAT_INTERVAL_MS * 2.5;
26+
27+
/**
28+
* Poll `/latest` this often no matter how healthy the stream looks.
29+
*
30+
* The floor is what makes the worst case bounded instead of unbounded. Even if every heuristic
31+
* above is wrong, and the watchdog somehow does not fire, staleness cannot exceed this. It is the
32+
* difference between "we might miss an update for a while" and the #41 outage.
33+
*/
34+
export const FLOOR_POLL_INTERVAL_MS = 15 * 60_000;
35+
36+
/** Reconnect backoff. Capped low because the relay closes streams every 15 min by design. */
37+
export const RECONNECT_BASE_DELAY_MS = 1_000;
38+
export const RECONNECT_MAX_DELAY_MS = 60_000;
39+
40+
export type ConnectionVerdict =
41+
| { readonly kind: "healthy" }
42+
| { readonly kind: "stale"; readonly silentForMs: number };
43+
44+
/**
45+
* Any byte counts, including a heartbeat comment.
46+
*
47+
* The heartbeat exists solely so that "no bytes" means something. Without it, a genuinely idle
48+
* channel — no releases for an hour — is indistinguishable from a dead socket, and any watchdog
49+
* built on it would either fire constantly or never.
50+
*/
51+
export function assessConnection(args: {
52+
readonly lastByteAtMs: number;
53+
readonly nowMs: number;
54+
readonly timeoutMs?: number;
55+
}): ConnectionVerdict {
56+
const timeoutMs = args.timeoutMs ?? WATCHDOG_TIMEOUT_MS;
57+
const silentForMs = args.nowMs - args.lastByteAtMs;
58+
return silentForMs > timeoutMs ? { kind: "stale", silentForMs } : { kind: "healthy" };
59+
}
60+
61+
/**
62+
* Exponential backoff with full jitter.
63+
*
64+
* Jittered because every desktop app on the fleet is woken by the same events — a relay deploy, a
65+
* laptop lid opening on a Monday morning — and unjittered backoff would reconnect them all in
66+
* lockstep, turning a blip into a thundering herd against a single Durable Object.
67+
*/
68+
export function reconnectDelayMs(attempt: number, random: () => number): number {
69+
const exponential = Math.min(RECONNECT_BASE_DELAY_MS * 2 ** Math.max(0, attempt), RECONNECT_MAX_DELAY_MS);
70+
return Math.round(exponential * random());
71+
}
72+
73+
/**
74+
* Every one of these re-reads `/latest` rather than trusting the stream.
75+
*
76+
* `power-resume` and `network-online` earn their place for the same reason the watchdog does:
77+
* after a sleep the stream may already be a corpse nobody has noticed, and waiting out the
78+
* watchdog to discover that is time spent knowingly stale. Being told by the OS that the world
79+
* changed beats inferring it from silence.
80+
*
81+
* There is no variant that trusts the stream instead, which is why this is a list rather than a
82+
* predicate — if a trigger is worth waking for, it is worth reconciling for.
83+
*/
84+
export type ReconciliationTrigger =
85+
| "startup"
86+
| "floor-poll"
87+
| "watchdog-fired"
88+
| "stream-closed"
89+
| "power-resume"
90+
| "network-online";

0 commit comments

Comments
 (0)