Skip to content

Commit 4abafa4

Browse files
t3dotggclaude
andcommitted
fix(mobile): harden background resume and reconnection
Five targeted fixes for mobile clients returning from iOS suspension: - Jitter the supervisor backoff ladder (equal jitter, [rung/2, rung]) so clients sharing an environment stop reconnecting in lockstep. - A failed foreground liveness probe now replaces the session immediately instead of paying the first backoff delay for a failure the probe just diagnosed. - New advisory `network-path-changed` wakeup: WiFi<->cellular keeps isConnected true while invalidating the socket's path, so the mobile adapter now emits a wakeup on interface-type changes while active and the supervisor probes the session instead of waiting ~5-10s for the ping timeout. - Report client activity on every newly connected session generation; the AppState-triggered report races the reconnect and is dropped, which left the server lease stale (provider/VCS work paused) for up to 25s after a resume. - Shell resumes a warm cache via afterSequence without the previously unconditional HTTP snapshot fetch, mirroring the thread-detail guard; foreground returns no longer pay an HTTP round trip. Supporting changes: wire the patched RPC onPingTimeout hook to logging so zombie-socket disconnects are distinguishable from ordinary closes, and set the server websocket idleTimeout to 30s (Bun defaults to 120s) so a suspended phone's half-dead socket and its server-side buffers are released four times sooner; live clients ping every 5s and never idle. Co-Authored-By: Claude Fable 5 <[email protected]>
1 parent 491219b commit 4abafa4

10 files changed

Lines changed: 218 additions & 65 deletions

File tree

‎apps/mobile/src/connection/background-activity.ts‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { EnvironmentRegistry } from "@t3tools/client-runtime/connection";
1+
import { EnvironmentRegistry, EnvironmentSupervisor } from "@t3tools/client-runtime/connection";
22
import { EnvironmentRpcSubscriptionObserver, request } from "@t3tools/client-runtime/rpc";
33
import {
44
type BackgroundScope,
@@ -103,6 +103,41 @@ export const mobileBackgroundActivityReporterLayer = Layer.effectDiscard(
103103
Stream.runForEach(() => Effect.sync(requestReport)),
104104
Effect.forkScoped,
105105
);
106+
// Re-report on every newly connected session generation. The AppState
107+
// report races the reconnect (the RPC fails while the socket is down and
108+
// is ignored), so without this the server's lease can lag a resume by up
109+
// to REPORT_INTERVAL_MS, keeping provider/VCS work paused.
110+
const connectedGenerations = (environmentId: EnvironmentId) =>
111+
registry
112+
.followStream(
113+
environmentId,
114+
Stream.unwrap(
115+
Effect.map(EnvironmentSupervisor, (supervisor) =>
116+
SubscriptionRef.changes(supervisor.state),
117+
),
118+
),
119+
)
120+
.pipe(
121+
Stream.filter((state) => state.phase === "connected"),
122+
Stream.map((state) => `${environmentId}:${state.generation}`),
123+
Stream.changes,
124+
);
125+
yield* Stream.concat(
126+
Stream.fromEffect(SubscriptionRef.get(registry.entries)),
127+
SubscriptionRef.changes(registry.entries),
128+
).pipe(
129+
Stream.map((entries) => [...entries.keys()].sort()),
130+
Stream.changesWith((a, b) => a.join(",") === b.join(",")),
131+
Stream.switchMap(
132+
(environmentIds) =>
133+
Stream.mergeAll(environmentIds.map(connectedGenerations), {
134+
concurrency: "unbounded",
135+
}),
136+
{ concurrency: "unbounded" },
137+
),
138+
Stream.runForEach(() => Effect.sync(requestReport)),
139+
Effect.forkScoped,
140+
);
106141
yield* Stream.fromQueue(reportRequests).pipe(
107142
Stream.debounce("250 millis"),
108143
Stream.runForEach(() => report),

‎apps/mobile/src/connection/platform.ts‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -87,11 +87,13 @@ const connectivityLayer = Connectivity.layer({
8787

8888
const wakeupsLayer = Wakeups.layer({
8989
changes: Stream.merge(
90-
Stream.callback<"application-active-probe" | "application-active-reconnect">((queue) =>
90+
Stream.callback<
91+
"application-active-probe" | "application-active-reconnect" | "network-path-changed"
92+
>((queue) =>
9193
Effect.acquireRelease(
9294
Effect.sync(() => {
9395
let backgroundedAtMs = AppState.currentState === "background" ? Date.now() : null;
94-
return AppState.addEventListener("change", (state) => {
96+
const appStateSubscription = AppState.addEventListener("change", (state) => {
9597
if (state === "background") {
9698
backgroundedAtMs = Date.now();
9799
return;
@@ -101,6 +103,32 @@ const wakeupsLayer = Wakeups.layer({
101103
backgroundedAtMs = null;
102104
}
103105
});
106+
// WiFi <-> cellular keeps isConnected true while invalidating the
107+
// socket's path, so the coarse online/offline signal never fires.
108+
// Emit an advisory wakeup on interface-type changes while active;
109+
// the supervisor probes the session rather than blindly replacing
110+
// it, which keeps flapping paths cheap.
111+
let networkType: string | null = null;
112+
const networkSubscription = Network.addNetworkStateListener((state) => {
113+
const nextType = state.type ?? null;
114+
const previousType = networkType;
115+
networkType = nextType;
116+
if (
117+
previousType !== null &&
118+
nextType !== null &&
119+
nextType !== previousType &&
120+
state.isConnected === true &&
121+
AppState.currentState === "active"
122+
) {
123+
Queue.offerUnsafe(queue, "network-path-changed");
124+
}
125+
});
126+
return {
127+
remove: () => {
128+
appStateSubscription.remove();
129+
networkSubscription.remove();
130+
},
131+
};
104132
}),
105133
(subscription) => Effect.sync(() => subscription.remove()),
106134
).pipe(Effect.asVoid),

‎packages/client-runtime/src/connection/supervisor.test.ts‎

Lines changed: 64 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -358,15 +358,18 @@ describe("EnvironmentSupervisor", () => {
358358
);
359359
expect(yield* Ref.get(harness.prepareCount)).toBe(1);
360360

361-
for (const [index, delay] of [1_000, 2_000, 4_000, 8_000, 16_000, 16_000].entries()) {
362-
yield* TestClock.adjust(delay);
361+
// Jitter puts each delay in [rung/2, rung], so exact rung boundaries
362+
// are not observable; advancing by the 16s cap always covers at least
363+
// one more retry and proves the ladder keeps going past its last rung.
364+
for (const [index] of [1_000, 2_000, 4_000, 8_000, 16_000, 16_000].entries()) {
365+
yield* TestClock.adjust(16_000);
363366
yield* eventuallyState(
364367
supervisor.state,
365-
(state) => state.phase === "backoff" && state.attempt === index + 2,
368+
(state) => state.phase === "backoff" && state.attempt >= index + 2,
366369
);
367370
}
368371

369-
expect(yield* Ref.get(harness.prepareCount)).toBe(7);
372+
expect(yield* Ref.get(harness.prepareCount)).toBeGreaterThanOrEqual(7);
370373
}).pipe(Effect.provide(TestClock.layer())),
371374
);
372375

@@ -539,9 +542,10 @@ describe("EnvironmentSupervisor", () => {
539542
);
540543
expect(yield* Ref.get(harness.prepareCount)).toBe(3);
541544

542-
yield* TestClock.adjust("999 millis");
545+
// Jittered first-rung delay lands in [500ms, 1000ms].
546+
yield* TestClock.adjust("499 millis");
543547
expect(yield* Ref.get(harness.prepareCount)).toBe(3);
544-
yield* TestClock.adjust("1 milli");
548+
yield* TestClock.adjust("501 millis");
545549
yield* eventuallyState(
546550
supervisor.state,
547551
(state) => state.phase === "backoff" && state.attempt === 2,
@@ -873,6 +877,55 @@ describe("EnvironmentSupervisor", () => {
873877
}),
874878
);
875879

880+
it.effect("probes the active session when the network path changes", () =>
881+
Effect.gen(function* () {
882+
const probeCount = yield* Ref.make(0);
883+
const probeCalled = yield* Deferred.make<void>();
884+
const harness = yield* makeHarness({
885+
probe: () =>
886+
Ref.update(probeCount, (count) => count + 1).pipe(
887+
Effect.andThen(Deferred.succeed(probeCalled, undefined)),
888+
),
889+
});
890+
const supervisor = yield* EnvironmentSupervisor.make(TARGET_ENTRY, {
891+
initiallyDesired: true,
892+
}).pipe(Effect.provide(harness.dependencies));
893+
894+
yield* awaitState(supervisor.state, (state) => state.phase === "connected");
895+
yield* harness.wake("network-path-changed");
896+
yield* Deferred.await(probeCalled);
897+
898+
expect(yield* Ref.get(probeCount)).toBe(1);
899+
expect(yield* Ref.get(harness.sessionCount)).toBe(1);
900+
expect(yield* Ref.get(harness.releaseCount)).toBe(0);
901+
expect((yield* SubscriptionRef.get(supervisor.state)).phase).toBe("connected");
902+
}),
903+
);
904+
905+
it.effect("replaces the session when a probe after a network path change fails", () =>
906+
Effect.gen(function* () {
907+
const harness = yield* makeHarness({
908+
probe: (attempt) =>
909+
attempt === 1
910+
? Effect.fail(transient("The path changed under the socket."))
911+
: Effect.void,
912+
});
913+
const supervisor = yield* EnvironmentSupervisor.make(TARGET_ENTRY, {
914+
initiallyDesired: true,
915+
}).pipe(Effect.provide(harness.dependencies));
916+
917+
yield* awaitState(supervisor.state, (state) => state.phase === "connected");
918+
yield* harness.wake("network-path-changed");
919+
yield* awaitState(
920+
supervisor.state,
921+
(state) => state.phase === "connected" && state.generation === 2,
922+
);
923+
924+
expect(yield* Ref.get(harness.sessionCount)).toBe(2);
925+
expect(yield* Ref.get(harness.releaseCount)).toBe(1);
926+
}),
927+
);
928+
876929
it.effect("immediately replaces a mobile session after a long background resume", () =>
877930
Effect.gen(function* () {
878931
const probeCount = yield* Ref.make(0);
@@ -937,9 +990,9 @@ describe("EnvironmentSupervisor", () => {
937990

938991
yield* awaitState(supervisor.state, (state) => state.phase === "connected");
939992
yield* harness.wake("application-active");
940-
yield* awaitState(supervisor.state, (state) => state.phase === "backoff");
941-
yield* TestClock.adjust("1 second");
942-
yield* eventuallyState(
993+
// A failed probe already proves the session is dead: the supervisor
994+
// replaces it immediately without entering backoff.
995+
yield* awaitState(
943996
supervisor.state,
944997
(state) => state.phase === "connected" && state.generation === 2,
945998
);
@@ -961,11 +1014,8 @@ describe("EnvironmentSupervisor", () => {
9611014
yield* awaitState(supervisor.state, (state) => state.phase === "connected");
9621015
yield* harness.wake("application-active-probe");
9631016
yield* TestClock.adjust("3 seconds");
964-
yield* awaitState(
965-
supervisor.state,
966-
(state) => state.phase === "backoff" && state.lastFailure?.reason === "timeout",
967-
);
968-
yield* TestClock.adjust("1 second");
1017+
// The 3s mobile probe timeout counts as a failed probe: the stale
1018+
// session is replaced immediately without a backoff delay.
9691019
yield* eventuallyState(
9701020
supervisor.state,
9711021
(state) => state.phase === "connected" && state.generation === 2,

‎packages/client-runtime/src/connection/supervisor.ts‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import * as Fiber from "effect/Fiber";
88
import * as Layer from "effect/Layer";
99
import * as Option from "effect/Option";
1010
import * as Queue from "effect/Queue";
11+
import * as Random from "effect/Random";
1112
import * as Ref from "effect/Ref";
1213
import * as Scope from "effect/Scope";
1314
import * as Stream from "effect/Stream";
@@ -101,8 +102,11 @@ export interface EnvironmentSupervisorOptions {
101102
readonly initiallyDesired?: boolean;
102103
}
103104

104-
function retryDelayMs(failureCount: number): number {
105-
return RETRY_DELAYS_MS[Math.min(failureCount, RETRY_DELAYS_MS.length - 1)] ?? 16_000;
105+
// Equal jitter: each retry waits between half and all of its ladder rung so
106+
// clients sharing an environment don't reconnect in lockstep after a restart.
107+
function retryDelayMs(failureCount: number, random: number): number {
108+
const base = RETRY_DELAYS_MS[Math.min(failureCount, RETRY_DELAYS_MS.length - 1)] ?? 16_000;
109+
return Math.round(base / 2 + random * (base / 2));
106110
}
107111

108112
function annotateTarget(target: ConnectionTarget) {
@@ -414,13 +418,17 @@ export const make = Effect.fn("EnvironmentSupervisor.make")(function* (
414418
// replaces that lease and starts a fresh attempt without backoff.
415419
return true;
416420
}
417-
if (next.reason === "application-active" || next.reason === "application-active-probe") {
421+
if (
422+
next.reason === "application-active" ||
423+
next.reason === "application-active-probe" ||
424+
next.reason === "network-path-changed"
425+
) {
418426
const probe = yield* lease.session.probe.pipe(
419427
Effect.timeoutOrElse({
420428
duration:
421-
next.reason === "application-active-probe"
422-
? MOBILE_CONNECTION_PROBE_TIMEOUT
423-
: CONNECTION_PROBE_TIMEOUT,
429+
next.reason === "application-active"
430+
? CONNECTION_PROBE_TIMEOUT
431+
: MOBILE_CONNECTION_PROBE_TIMEOUT,
424432
orElse: () =>
425433
Effect.fail(
426434
new ConnectionTransientError({
@@ -441,7 +449,12 @@ export const make = Effect.fn("EnvironmentSupervisor.make")(function* (
441449
),
442450
);
443451
if (probeEvent._tag === "ProbeCompleted") {
444-
yield* probeEvent.exit;
452+
if (Exit.isFailure(probeEvent.exit)) {
453+
// A failed health check already proves the session is dead;
454+
// replace it immediately instead of paying the retry ladder
455+
// for a failure the probe just diagnosed.
456+
return true;
457+
}
445458
break;
446459
}
447460
switch (probeEvent.signal._tag) {
@@ -710,7 +723,7 @@ export const make = Effect.fn("EnvironmentSupervisor.make")(function* (
710723
}
711724

712725
failureCount += 1;
713-
const delayMs = retryDelayMs(failureCount - 1);
726+
const delayMs = retryDelayMs(failureCount - 1, yield* Random.next);
714727
pendingRetry = Option.map(attemptSpan, (previousAttempt) => ({
715728
previousAttempt,
716729
failureCount,

‎packages/client-runtime/src/connection/wakeups.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ export type ConnectionWakeup =
66
| "application-active"
77
| "application-active-probe"
88
| "application-active-reconnect"
9+
// Advisory: the network path changed shape (e.g. WiFi to cellular) while
10+
// still nominally online. The old socket often survives in appearance only,
11+
// so the supervisor probes it instead of waiting for a ping timeout.
12+
| "network-path-changed"
913
| "credentials-changed";
1014

1115
export function isApplicationActiveWakeup(reason: ConnectionWakeup): boolean {

‎packages/client-runtime/src/rpc/session.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,14 @@ export const make = Effect.gen(function* () {
9191
),
9292
Effect.asVoid,
9393
),
94+
// Distinguishes zombie-socket disconnects (missed pong after suspension
95+
// or a silent path change) from ordinary closes in logs and telemetry.
96+
onPingTimeout: Effect.logInfo("Connection ping timed out; treating the socket as dead.").pipe(
97+
Effect.annotateLogs({
98+
environmentId: connection.environmentId,
99+
connectionLabel: connection.label,
100+
}),
101+
),
94102
});
95103
const socketLayer = Socket.layerWebSocket(connection.socketUrl, {
96104
openTimeout: SOCKET_OPEN_TIMEOUT,

0 commit comments

Comments
 (0)