Skip to content

Commit ef1a0fa

Browse files
t3dotggclaude
andcommitted
fix(server): the SQLite WAL file shrinks back after large writes
PASSIVE auto-checkpoints copy frames back into the database but never make the -wal file smaller. The file keeps its largest size until the last connection closes, so one large write or a long foreign read snapshot left a multi-GB file on disk. Set journal_size_limit to 32 MiB in the shared setup. SQLite now cuts the -wal file back to that size on the first commit after a WAL reset. Normal use keeps the WAL near 4 MB, so this is a no-op there. No periodic TRUNCATE checkpoint: on the synchronous connection with a 5 s busy handler it would stall the event loop while another reader holds a snapshot. The test writes past the limit through the real layer on a temp file and checks the -wal size on disk after one more commit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 2598ce7 commit ef1a0fa

2 files changed

Lines changed: 37 additions & 1 deletion

File tree

‎apps/server/src/persistence/Layers/Sqlite.test.ts‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ import * as Effect from "effect/Effect";
1010
import * as Layer from "effect/Layer";
1111
import * as SqlClient from "effect/unstable/sql/SqlClient";
1212

13-
import { SqlitePersistenceMemory, makeSqlitePersistenceLive } from "./Sqlite.ts";
13+
import {
14+
SqlitePersistenceMemory,
15+
WAL_SIZE_LIMIT_BYTES,
16+
makeSqlitePersistenceLive,
17+
} from "./Sqlite.ts";
1418

1519
const lockHolderSource = `
1620
const { DatabaseSync } = require("node:sqlite");
@@ -57,6 +61,32 @@ it.effect("waits out a concurrent writer instead of failing with SQLITE_BUSY", (
5761
);
5862
});
5963

64+
it.effect("shrinks the WAL file back to the size limit after a large write", () => {
65+
const tempDir = NodeFS.mkdtempSync(NodePath.join(NodeOS.tmpdir(), "t3-sqlite-wal-"));
66+
const dbPath = NodePath.join(tempDir, "state.sqlite");
67+
const walFileSize = () => NodeFS.statSync(`${dbPath}-wal`).size;
68+
// About 25% more 4 KB rows than the limit holds, in one transaction.
69+
const rowCount = Math.ceil((WAL_SIZE_LIMIT_BYTES * 1.25) / 4000);
70+
71+
return Effect.gen(function* () {
72+
const sql = yield* SqlClient.SqlClient;
73+
yield* sql`CREATE TABLE wal_probe(payload BLOB)`;
74+
yield* sql`
75+
WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ${rowCount})
76+
INSERT INTO wal_probe(payload) SELECT randomblob(4000) FROM n
77+
`;
78+
assert.isAbove(walFileSize(), WAL_SIZE_LIMIT_BYTES);
79+
80+
// The auto-checkpoint after the large commit copied every frame into the
81+
// database, so the next commit restarts the WAL and cuts the file back.
82+
yield* sql`INSERT INTO wal_probe(payload) VALUES (x'00')`;
83+
assert.isAtMost(walFileSize(), WAL_SIZE_LIMIT_BYTES);
84+
}).pipe(
85+
Effect.provide(makeSqlitePersistenceLive(dbPath).pipe(Layer.provide(NodeServices.layer))),
86+
Effect.ensuring(Effect.sync(() => NodeFS.rmSync(tempDir, { recursive: true, force: true }))),
87+
);
88+
});
89+
6090
it.effect("applies busy_timeout in the shared persistence setup", () =>
6191
Effect.gen(function* () {
6292
const sql = yield* SqlClient.SqlClient;

‎apps/server/src/persistence/Layers/Sqlite.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,19 @@ import * as NodeSqliteClient from "@t3tools/shared/nodeSqliteClient";
88
import { runMigrations } from "../Migrations.ts";
99
import { ServerConfig } from "../../config.ts";
1010

11+
// Size the -wal file is cut back to on the first commit after a WAL reset.
12+
export const WAL_SIZE_LIMIT_BYTES = 32 * 1024 * 1024;
13+
1114
const setup = Layer.effectDiscard(
1215
Effect.gen(function* () {
1316
const sql = yield* SqlClient.SqlClient;
1417
// CLI and server write from separate processes; wait rather than fail with SQLITE_BUSY.
1518
yield* sql`PRAGMA busy_timeout = 5000;`;
1619
yield* sql`PRAGMA foreign_keys = ON;`;
1720
yield* sql`PRAGMA journal_mode = WAL;`;
21+
// PASSIVE checkpoints never shrink the -wal file, so it otherwise keeps its
22+
// largest size until the last connection closes.
23+
yield* sql.unsafe(`PRAGMA journal_size_limit = ${WAL_SIZE_LIMIT_BYTES};`);
1824
yield* runMigrations();
1925
}),
2026
);

0 commit comments

Comments
 (0)