Skip to content

Commit cf2e183

Browse files
donjor-agent[bot]donjor
authored andcommitted
fix(server): a stop during the migration backup no longer kills the update (#1873)
## Problem A stop during the pre-migration backup queued behind the copy. systemd killed the launcher at 90 s, and the next start copied the whole database again. Meanwhile clients showed "Restarting… Ns" for the whole copy, with no hint of the cause. ## Change **Stop:** - **Abort:** `stop()` kills the backup `cp` - **Seam:** fork file carries the signal - **No fallback:** no `copyFile` after an abort - **Staging:** removed by the existing catch - **Pending kept:** no trial, no rollback - **Resume:** next start backs up and trials The guarantee holds: no trial starts without a complete backup. **A1 override:** an in-flight fsync or `copyFile` still delays a stop; Node can't cancel either, and a SIGKILL there keeps the update pending to resume. **Label:** - **Server:** reports DB size when a backup will run - **Rule:** same as the launcher's skip rule - **Contract:** optional `databaseBackupBytesFork` - **Web and mobile:** shared label helper - **Shows:** the label below ```text Backing up database (20.9 GB) and restarting… 42s elapsed ``` Older servers omit the field; clients keep "Restarting". ## Scope and approval Closes #1705 Approved option a: stop fix plus backup label. Cheaper backup stays out of scope. **Follow-up:** re-measure the backup after `fork-sync` takes upstream `pingdotgg#22630`. **Caveat:** the service runs its install-time launcher (#1865). The stop fix reaches an install after `t3 update` or `t3 service install`. ## Close conditions - [x] **Backup:** skipped without pending migrations - [x] **Reconnect:** projects and threads stay visible - [x] **Route:** survives the server restart - [x] **Progress:** visible through the restart - [x] **Dialog delay:** explained or fixed - [x] **Re-run:** large DB shows the gains - [ ] **Stop:** exits promptly during the backup - [ ] **Resume:** the update resumes on next start - [ ] **Label:** the client names the backup ## Verification - [x] **Stop test:** FIFO DB holds `cp` open - [x] **Asserts:** `pending`; no trial or backup - [x] **Red first:** failed before the fix - [x] **Resume test:** next start backs up, trials - [x] **Size test:** only with pending migrations - [x] **Label test:** size format; fallback otherwise - [x] `vp run check:changed` → exit 1 - [x] `vp run fork:ci` → exit 0 **check:changed:** 1 failing test outside this diff, `GitManager.test.ts` (CLAUDE.md writer). **Not shown:** no before/after capture of the label. Reaching it needs a real boot-service update on a large database. Claude Opus 5.5 in Claude Code, via T3 Code. ## Fork trailers Fork-Domain: upstream-fixes Fork-Tier: bugfix Fork-Upstreamable: yes Co-authored-by: donjor <[email protected]>
1 parent 8bd2820 commit cf2e183

14 files changed

Lines changed: 212 additions & 7 deletions

‎apps/mobile/src/features/settings/SettingsEnvironmentDetailRouteScreen.tsx‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import {
1010
import { settlePromise, squashAtomCommandFailure } from "@t3tools/client-runtime/state/runtime";
1111
import * as Cause from "effect/Cause";
1212
import * as Exit from "effect/Exit";
13+
import { serverUpdateRestartLabelFork } from "@t3tools/client-runtime/state/server-update-backup"; // fork-hook: upstream-fixes/migration-backup-label
1314
import { AsyncResult } from "effect/reactivity";
1415
import { useRef, useState } from "react";
1516
import { View } from "react-native";
@@ -235,7 +236,10 @@ function EnvironmentDetail({ environmentId }: { readonly environmentId: Environm
235236
{running ? (
236237
<Text className="text-sm text-foreground-muted">
237238
{updateState.stage === "resuming"
238-
? "Restarting and reconnecting…"
239+
? serverUpdateRestartLabelFork(
240+
updateState,
241+
"Restarting and reconnecting…",
242+
) // fork-hook: upstream-fixes/migration-backup-label
239243
: "Downloading update…"}
240244
</Text>
241245
) : updateState.status === "failed" ? (
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import * as NodeServices from "@effect/platform-node/NodeServices";
2+
import { expect, it } from "@effect/vitest";
3+
import * as Effect from "effect/Effect";
4+
import * as FileSystem from "effect/FileSystem";
5+
import * as Path from "effect/Path";
6+
7+
import { databaseBackupBytesFork } from "./databaseBackupBytes.fork.ts";
8+
import { SERVICE_LAUNCHER_PROTOCOL } from "./serviceProtocol.ts";
9+
10+
it.layer(NodeServices.layer)("database backup size", (it) => {
11+
it.effect("is reported unless the preflight rules out pending migrations", () =>
12+
Effect.gen(function* () {
13+
const fs = yield* FileSystem.FileSystem;
14+
const path = yield* Path.Path;
15+
const dbPath = path.join(yield* fs.makeTempDirectoryScoped(), "state.sqlite");
16+
yield* fs.writeFileString(dbPath, "1234");
17+
yield* fs.writeFileString(`${dbPath}-wal`, "56");
18+
const ready = {
19+
status: "ready",
20+
version: "1.1.0",
21+
launcherProtocol: SERVICE_LAUNCHER_PROTOCOL,
22+
} as const;
23+
const bytes = (pendingMigrationsFork?: boolean) =>
24+
databaseBackupBytesFork(fs, dbPath, {
25+
...ready,
26+
...(pendingMigrationsFork === undefined ? {} : { pendingMigrationsFork }),
27+
}).pipe(Effect.map((result) => result.databaseBackupBytesFork));
28+
expect(yield* bytes(true)).toBe(6);
29+
expect(yield* bytes(undefined)).toBe(6);
30+
expect(yield* bytes(false)).toBeUndefined();
31+
}),
32+
);
33+
});
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import * as Effect from "effect/Effect";
2+
import * as FileSystem from "effect/FileSystem";
3+
import * as Option from "effect/Option";
4+
5+
import type { ServicePreflightResult } from "./servicePreflight.ts";
6+
7+
/**
8+
* Size of the copy the service launcher makes before a migrating trial, so
9+
* clients can name that wait instead of showing a bare restart. Mirrors the
10+
* launcher's skip rule: only a preflight reporting no pending migrations skips it.
11+
*/
12+
export const databaseBackupBytesFork = Effect.fn("cloud.server_self_update.backup_bytes_fork")(
13+
function* (fs: FileSystem.FileSystem, dbPath: string, preflight?: ServicePreflightResult) {
14+
if (preflight?.status === "ready" && preflight.pendingMigrationsFork === false) return {};
15+
let bytes = 0;
16+
for (const suffix of ["", "-wal", "-shm"]) {
17+
const info = yield* fs.stat(`${dbPath}${suffix}`).pipe(Effect.option);
18+
if (Option.isSome(info)) bytes += Number(info.value.size);
19+
}
20+
return bytes > 0 ? { databaseBackupBytesFork: bytes } : {};
21+
},
22+
);

‎apps/server/src/cloud/selfUpdate.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,10 @@ import {
3737
PinnedRuntimePreflightBlockedError,
3838
prunePinnedRuntimes,
3939
} from "./pinnedRuntime.ts";
40+
import { databaseBackupBytesFork } from "./databaseBackupBytes.fork.ts"; // fork-hook: upstream-fixes/migration-backup-label
4041
import { repointLaunchersFork } from "./launcherRepoint.fork.ts"; // fork-hook: upstream-fixes/launcher-repoint-import
4142
import { decodeServicePreflightResult } from "./servicePreflight.ts";
43+
import type { ServicePreflightResult } from "./servicePreflight.ts"; // fork-hook: upstream-fixes/migration-backup-label
4244
import * as ServiceLauncherClient from "./serviceLauncherClient.ts";
4345
import { isExactServiceVersion, SERVICE_LAUNCHER_PROTOCOL } from "./serviceProtocol.ts";
4446

@@ -255,6 +257,7 @@ export const make = Effect.fn("cloud.server_self_update.make")(function* () {
255257
}
256258

257259
return yield* Effect.gen(function* () {
260+
let preflightFork: ServicePreflightResult | undefined; // fork-hook: upstream-fixes/migration-backup-label
258261
yield* reportProgress("downloading");
259262
const paths = yield* ensurePinnedRuntimeInstalled({
260263
baseDir: serverConfig.baseDir,
@@ -317,6 +320,7 @@ export const make = Effect.fn("cloud.server_self_update.make")(function* () {
317320
);
318321
}
319322
const preflight = decodeServicePreflightResult(parsed);
323+
preflightFork = preflight; // fork-hook: upstream-fixes/migration-backup-label
320324
if (preflight === undefined || preflight.version !== targetVersion) {
321325
return Effect.fail(
322326
new PinnedRuntimeInstallError({
@@ -343,6 +347,7 @@ export const make = Effect.fn("cloud.server_self_update.make")(function* () {
343347
),
344348
);
345349

350+
const backupFork = yield* databaseBackupBytesFork(fs, serverConfig.dbPath, preflightFork); // fork-hook: upstream-fixes/migration-backup-label
346351
yield* reportProgress("installing");
347352
const updateId = yield* Effect.uninterruptible(
348353
launcher.requestUpdate({ targetVersion, dbPath: serverConfig.dbPath }).pipe(
@@ -364,7 +369,7 @@ export const make = Effect.fn("cloud.server_self_update.make")(function* () {
364369
targetVersion,
365370
runtimePath: paths.entryPath,
366371
});
367-
return { targetVersion, method: "boot-service" as const, updateId };
372+
return { targetVersion, method: "boot-service" as const, updateId, ...backupFork }; // fork-hook: upstream-fixes/migration-backup-label
368373
}).pipe(Effect.onError(() => Ref.set(inFlight, false)));
369374
});
370375

‎apps/server/src/cloud/servicePreflight.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,9 @@ export function decodeServicePreflightResult(value: unknown): ServicePreflightRe
5858
status: "ready",
5959
version: record.version,
6060
launcherProtocol: SERVICE_LAUNCHER_PROTOCOL,
61+
...(typeof record.pendingMigrationsFork === "boolean"
62+
? { pendingMigrationsFork: record.pendingMigrationsFork }
63+
: {}), // fork-hook: upstream-fixes/migration-backup-label
6164
};
6265
}
6366
if (

‎apps/server/src/serviceLauncher.fork.test.ts‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
// @effect-diagnostics nodeBuiltinImport:off - Exercise the launcher's native filesystem and SQLite process boundary.
2+
import * as NodeChildProcess from "node:child_process";
23
import * as NodeFSP from "node:fs/promises";
34
import * as NodeOS from "node:os";
45
import * as NodePath from "node:path";
56
import * as NodeSqlite from "node:sqlite";
67

78
import { afterEach, expect, it } from "@effect/vitest";
9+
import * as HostProcess from "@t3tools/shared/HostProcess";
810

911
import { Launcher, readServiceState, writeServiceState } from "./serviceLauncher.ts";
1012
import { backupDatabaseForMigrationsFork } from "./serviceLauncherBackup.fork.ts";
@@ -193,3 +195,69 @@ process.exit(1);
193195
expect(await NodeFSP.readFile(livePath, "utf8")).toBe("live");
194196
expect(backedUp).toBe(true);
195197
});
198+
199+
// A FIFO database keeps `cp` reading until the test closes its writer, so the stop lands mid-copy.
200+
it.runIf(HostProcess.Platform.defaultValue() === "linux")(
201+
"a stop during the backup aborts the copy and leaves the update pending",
202+
async () => {
203+
const root = await NodeFSP.mkdtemp(NodePath.join(NodeOS.tmpdir(), "t3-launcher-fork-stop-"));
204+
roots.push(root);
205+
const databasePath = NodePath.join(root, "state.sqlite");
206+
NodeChildProcess.execFileSync("mkfifo", [databasePath]);
207+
const trialStarted = NodePath.join(root, "trial-started");
208+
for (const version of ["1.0.0", "1.1.0"]) {
209+
const versionDir = NodePath.join(root, "runtime", "versions", version);
210+
await NodeFSP.mkdir(versionDir, { recursive: true });
211+
await NodeFSP.writeFile(NodePath.join(versionDir, ".install-complete"), version);
212+
await NodeFSP.writeFile(
213+
NodePath.join(versionDir, "t3"),
214+
`#!${process.execPath}
215+
if (process.argv[2] === "__service-preflight") process.exit(1);
216+
const context = JSON.parse(process.env.T3_SERVICE_LAUNCHER_CONTEXT);
217+
if (context.update?.status === "pending") require("node:fs").writeFileSync(${JSON.stringify(trialStarted)}, "");
218+
`,
219+
{ mode: 0o755 },
220+
);
221+
}
222+
const statePath = NodePath.join(root, "runtime", "service-state.json");
223+
await writeServiceState(statePath, {
224+
protocol: SERVICE_LAUNCHER_PROTOCOL,
225+
activeVersion: "1.0.0",
226+
update: {
227+
id: "update-1",
228+
fromVersion: "1.0.0",
229+
targetVersion: "1.1.0",
230+
dbPath: databasePath,
231+
status: "pending",
232+
},
233+
});
234+
const launcher = new Launcher(root, await readServiceState(statePath));
235+
const running = launcher.run();
236+
// Opening the FIFO for writing resolves once `cp` has opened it to read. Holding it open
237+
// keeps the copy blocked, so only an abort lets the stop finish.
238+
const writer = await NodeFSP.open(databasePath, "w");
239+
try {
240+
await launcher.stop("SIGTERM");
241+
await running;
242+
} finally {
243+
await writer.close();
244+
}
245+
246+
const state = await readServiceState(statePath);
247+
expect(state.activeVersion).toBe("1.0.0");
248+
expect(state.update?.status).toBe("pending");
249+
const backupDir = NodePath.join(root, "runtime", "db-backup", "update-1");
250+
for (const path of [trialStarted, backupDir, `${backupDir}.staging`]) {
251+
await expect(NodeFSP.stat(path)).rejects.toMatchObject({ code: "ENOENT" });
252+
}
253+
254+
// The next start resumes the update: a full backup, then the trial.
255+
await NodeFSP.rm(databasePath);
256+
await NodeFSP.writeFile(databasePath, "database");
257+
const resumed = new Launcher(root, await readServiceState(statePath));
258+
await expect(resumed.run()).rejects.toThrow("Active child exited unexpectedly");
259+
await NodeFSP.stat(trialStarted);
260+
expect((await readServiceState(statePath)).update?.status).toBe("rolled-back");
261+
expect(await NodeFSP.readFile(databasePath, "utf8")).toBe("database");
262+
},
263+
);

‎apps/server/src/serviceLauncher.ts‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import * as NodeFSP from "node:fs/promises";
88
import * as NodePath from "node:path";
99
import * as NodeUtil from "node:util";
1010
import { backupDatabaseForMigrationsFork } from "./serviceLauncherBackup.fork.ts"; // fork-hook: upstream-fixes/migration-backup
11+
import { abortableBackupFork, backupAbortSignalFork } from "./serviceLauncherBackupAbort.fork.ts"; // fork-hook: upstream-fixes/migration-backup-stop
1112

1213
import type {
1314
PendingServiceUpdate,
@@ -137,8 +138,11 @@ async function syncDirectory(directory: string): Promise<void> {
137138
}
138139

139140
function runCp(args: ReadonlyArray<string>): Promise<boolean> {
140-
return new Promise((resolve) => {
141-
NodeChildProcess.execFile("cp", args, (error) => resolve(error === null));
141+
const signal = backupAbortSignalFork(); // fork-hook: upstream-fixes/migration-backup-stop
142+
return new Promise((resolve, reject) => {
143+
NodeChildProcess.execFile("cp", args, { signal }, (error) =>
144+
signal?.aborted ? reject(signal.reason) : resolve(error === null),
145+
);
142146
});
143147
}
144148

@@ -156,7 +160,10 @@ async function copyDatabaseFile(source: string, destination: string): Promise<vo
156160
: platform === "linux"
157161
? ["--reflink=auto", source, destination]
158162
: undefined;
159-
if (args === undefined || !(await runCp(args))) await NodeFSP.copyFile(source, destination);
163+
if (args === undefined || !(await runCp(args))) {
164+
backupAbortSignalFork()?.throwIfAborted(); // fork-hook: upstream-fixes/migration-backup-stop
165+
await NodeFSP.copyFile(source, destination);
166+
}
160167
}
161168

162169
const GIB = 1024 ** 3;
@@ -259,6 +266,7 @@ async function backupDatabaseOnce(baseDir: string, pending: PendingServiceUpdate
259266
await copyDatabaseFile(source, destination);
260267
await syncFile(destination);
261268
}
269+
backupAbortSignalFork()?.throwIfAborted(); // fork-hook: upstream-fixes/migration-backup-stop
262270
await NodeFSP.rename(stagingDir, backupDir);
263271
await syncDirectory(NodePath.dirname(backupDir));
264272
} catch (cause) {
@@ -443,6 +451,7 @@ export class Launcher {
443451
#timer: NodeJS.Timeout | undefined;
444452
#transitions: Promise<void> = Promise.resolve();
445453
#stopRequested = false;
454+
readonly #stopAbort = new AbortController(); // fork-hook: upstream-fixes/migration-backup-stop
446455
#stopping = false;
447456
#done = false;
448457
readonly #completion = Promise.withResolvers<void>();
@@ -507,6 +516,7 @@ export class Launcher {
507516
return;
508517
}
509518
this.#stopRequested = true;
519+
this.#stopAbort.abort(); // fork-hook: upstream-fixes/migration-backup-stop
510520
this.#clearTimer();
511521
this.#enqueue(async () => {
512522
// Let in-progress updates start replacements first; replacements observe the marker and own pre-activation tunnel cleanup.
@@ -558,9 +568,10 @@ export class Launcher {
558568
this.#baseDir,
559569
pending,
560570
runtimePaths(this.#baseDir, pending.targetVersion).entryPath,
561-
backupDatabaseOnce,
571+
abortableBackupFork(this.#stopAbort.signal, backupDatabaseOnce),
562572
); // fork-hook: upstream-fixes/migration-backup
563573
} catch {
574+
if (this.#stopRequested) return; // fork-hook: upstream-fixes/migration-backup-stop
564575
await this.#returnToPrevious(pending, "failed", "db-backup-failed");
565576
return;
566577
}
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
// @effect-diagnostics nodeBuiltinImport:off - Keep the launcher independent of Effect.
2+
import * as NodeAsyncHooks from "node:async_hooks";
3+
4+
import type { PendingServiceUpdate } from "./cloud/serviceProtocol.ts";
5+
6+
// Carries the stop signal into the backup copy without widening upstream signatures.
7+
const backupAbort = new NodeAsyncHooks.AsyncLocalStorage<AbortSignal>();
8+
9+
export const backupAbortSignalFork = (): AbortSignal | undefined => backupAbort.getStore();
10+
11+
/** A stop kills `cp` and skips the rename; the launcher keeps the update pending to resume. */
12+
export const abortableBackupFork =
13+
(
14+
signal: AbortSignal,
15+
backup: (baseDir: string, pending: PendingServiceUpdate) => Promise<void>,
16+
) =>
17+
(baseDir: string, pending: PendingServiceUpdate): Promise<void> =>
18+
backupAbort.run(signal, () => backup(baseDir, pending));

‎apps/web/src/components/ServerUpdateAction.tsx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
type ServerUpdateStage,
1717
type ServerUpdateState,
1818
} from "@t3tools/client-runtime/state/server";
19+
import { serverUpdateRestartLabelFork } from "@t3tools/client-runtime/state/server-update-backup"; // fork-hook: upstream-fixes/migration-backup-label
1920
import {
2021
isAtomCommandInterrupted,
2122
settlePromise,
@@ -214,7 +215,7 @@ function useServerUpdateDetail(state: Exclude<ServerUpdateState, { status: "idle
214215
if (state.status === "failed") return state.message;
215216
if (state.status === "completed") return `Updated to t3@${state.targetVersion}.`;
216217
const elapsed = serverUpdateRestartElapsedSeconds(state, now);
217-
return `${serverUpdateStageLabel(state.stage)}${elapsed === null ? "" : ` ${elapsed}s elapsed`}`;
218+
return `${serverUpdateRestartLabelFork(state, serverUpdateStageLabel(state.stage))}${elapsed === null ? "" : ` ${elapsed}s elapsed`}`; // fork-hook: upstream-fixes/migration-backup-label
218219
}
219220

220221
/** Unknown byte counts stay indeterminate; terminal outcomes stop all activity. */

‎packages/client-runtime/package.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,10 @@
287287
"types": "./src/state/server.ts",
288288
"default": "./src/state/server.ts"
289289
},
290+
"./state/server-update-backup": {
291+
"types": "./src/state/serverUpdateBackup.fork.ts",
292+
"default": "./src/state/serverUpdateBackup.fork.ts"
293+
},
290294
"./state/outdatedServerUpdate": {
291295
"types": "./src/state/outdatedServerUpdate.ts",
292296
"default": "./src/state/outdatedServerUpdate.ts"

0 commit comments

Comments
 (0)