Skip to content

Commit de622e4

Browse files
committed
fix(responses): verify the snapshot on disk before skipping a write
Skipping a byte-identical rewrite is the right fix for the amplification, but the cached digest describes what this process last wrote, which is not the same claim as what is on disk now. A second proxy sharing the home, or anything that rewrites the file in place, leaves the digest describing bytes that are gone. That matters more than it sounds. Before the skip existed, every flush rewrote the file and so repaired external damage silently. Skipping on the digest alone turns a self-healing snapshot into a permanently corrupt one, and nothing notices until the next restart fails to load the continuation state. Replacing the file with different bytes of the same length reproduces it: the digest still matches, the file still exists, and the flush declines to repair. The skip now verifies identity against the file itself. The cached digest is keyed to the resolved write target, so a config-dir change or a retargeted symlink is a miss rather than a false match, and the contents are compared byte-for-byte before declining to write. Size is checked first so the common mismatch costs a stat, any read failure answers "no" and the caller rewrites, and the read only happens when the digest already agreed. The amplification being fixed is the repeated 24 MiB atomic replace, not the read that avoids it. This also removes the need to trust Bun.hash for correctness. It stays a cheap first filter, but a collision can no longer produce a false skip. Tests: same-length external replacement must be rewritten, proven by reverting only the disk check. The docs line claiming graceful shutdown "always flushes" is corrected too — the flush is a disk write and can fail like any other, and writeBoundedSnapshot swallows that into a "failed" outcome the lifecycle warning cannot see. 124 pass across write-amplification and responses-state. tsc clean.
1 parent fec2957 commit de622e4

3 files changed

Lines changed: 113 additions & 7 deletions

File tree

‎docs-site/src/content/docs/troubleshooting/disk-usage-temp-files.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,9 +63,11 @@ entirely.
6363
Together these keep the write rate roughly flat as the cache grows, instead of
6464
re-serializing and replacing the whole file every two seconds.
6565

66-
A graceful shutdown always flushes, so the longer wait only widens the window in
67-
which a hard kill loses the most recent continuation entries — which are cache,
68-
as above.
66+
A graceful shutdown flushes immediately rather than waiting out the timer, so the
67+
longer wait mainly widens the window in which a hard kill loses the most recent
68+
continuation entries — which are cache, as above. That flush is still a disk
69+
write and can fail like any other, so a shutdown on a full or read-only volume
70+
can lose the same entries.
6971

7072
## Reclaiming files that already accumulated
7173

‎src/responses/state.ts‎

Lines changed: 38 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,27 @@ let stateRevision = 0;
101101
* retained: at the 24 MiB bound that would double the snapshot's memory cost. */
102102
let lastSnapshotBytes = 0;
103103
let lastSnapshotDigest: string | null = null;
104+
// The resolved file the digest above describes. Keeping it means a config-dir
105+
// change or a retargeted symlink is a miss rather than a false "unchanged".
106+
let lastSnapshotTarget: string | null = null;
107+
108+
/**
109+
* Is the snapshot on disk still byte-for-byte what we last wrote?
110+
*
111+
* The cached digest proves what this process wrote, not what is there now. Size is
112+
* checked first so the common mismatch costs a `stat`, and the content comparison
113+
* only runs when the size already agrees. Any read failure answers "no" and the
114+
* caller rewrites — the safe direction.
115+
*/
116+
async function snapshotOnDiskMatches(path: string, payload: string, payloadBytes: number): Promise<boolean> {
117+
try {
118+
const file = Bun.file(path);
119+
if (file.size !== payloadBytes) return false;
120+
return await file.text() === payload;
121+
} catch {
122+
return false;
123+
}
124+
}
104125
const spillCounters = { writes: 0, writeFailures: 0, readFailures: 0 };
105126
/**
106127
* Admission-boundary observability (test-visible). directSpills: oversized
@@ -805,18 +826,31 @@ async function writeBoundedSnapshot(path: string): Promise<SnapshotWriteOutcome>
805826
// A mutation does not always change what gets persisted: entries past the
806827
// per-entry or total byte bound are dropped from the selection, and spill
807828
// demotion moves bytes out of it. Re-writing a byte-identical 24 MiB file
808-
// buys nothing, so compare first — but only skip while the file we would be
809-
// reproducing is still there.
829+
// buys nothing, so compare first — but the cached digest describes what THIS
830+
// process last wrote, which is not the same claim as "that is what is on disk
831+
// now". A second proxy sharing the home, or anything that rewrites the file
832+
// in place, leaves the digest describing bytes that are gone. Before every
833+
// release-of-a-write, the previous behaviour rewrote unconditionally and so
834+
// repaired that silently; skipping without checking would turn a repaired
835+
// snapshot into a lost one at the next restart.
836+
//
837+
// Verify against the file itself, keyed to the resolved target so a retargeted
838+
// symlink is also a miss. Reading back a matching-size file costs far less
839+
// than the atomic replace it avoids, and only happens when the digest already
840+
// matched — the amplification this fixes is the repeated WRITE, not the read.
810841
const unchanged = lastSnapshotDigest !== null
811842
&& payloadDigest === lastSnapshotDigest
812843
&& payloadBytes === lastSnapshotBytes
813-
&& existsSync(path);
844+
&& lastSnapshotTarget === resolveWriteTarget(path)
845+
&& existsSync(path)
846+
&& await snapshotOnDiskMatches(path, payload, payloadBytes);
814847
if (!unchanged) {
815848
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
816849
try { chmodSync(dirname(path), 0o700); } catch { /* best-effort (e.g. Windows) */ }
817850
await atomicWriteFileAsync(path, payload);
818851
lastSnapshotDigest = payloadDigest;
819852
lastSnapshotBytes = payloadBytes;
853+
lastSnapshotTarget = resolveWriteTarget(path);
820854
}
821855
persistAttemptHookForTests?.();
822856
if (revision === stateRevision) return "stable";
@@ -1462,6 +1496,7 @@ export function clearResponseStateMemoryForTests(): void {
14621496
persistAttemptHookForTests = null;
14631497
lastSnapshotBytes = 0;
14641498
lastSnapshotDigest = null;
1499+
lastSnapshotTarget = null;
14651500
loaded = false;
14661501
}
14671502

‎tests/responses-state-write-amplification.test.ts‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
* of the snapshot actually being written.
99
*/
1010
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
11-
import { existsSync, mkdtempSync, rmSync, statSync, utimesSync } from "node:fs";
11+
import { existsSync, mkdtempSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync } from "node:fs";
1212
import { tmpdir } from "node:os";
1313
import { join } from "node:path";
1414
import {
@@ -108,6 +108,31 @@ describe("responses-state snapshot write amplification (#2460)", () => {
108108
expect(existsSync(snapshot)).toBe(true);
109109
});
110110

111+
// Deletion is the easy half. The cached digest describes what THIS process last
112+
// wrote, which is not the same claim as "that is what is on disk now" — a second
113+
// proxy sharing the home, or anything rewriting the file in place, leaves the
114+
// digest describing bytes that are gone. Before the skip existed, every flush
115+
// rewrote and so repaired that silently; skipping on the digest alone would turn
116+
// a self-healing snapshot into a permanently corrupt one, discovered only at the
117+
// next restart when the continuation state fails to load.
118+
//
119+
// Same-length replacement is the case that defeats a size-only check, so that is
120+
// what this drives.
121+
test("a snapshot replaced with different bytes of the same length is rewritten", async () => {
122+
remember("resp_amp_replaced", "kept");
123+
await flushResponseState();
124+
125+
const original = readFileSync(snapshot, "utf-8");
126+
writeFileSync(snapshot, "X".repeat(Buffer.byteLength(original, "utf8")));
127+
128+
// A mutation whose bounded payload is byte-identical to the last write: the
129+
// oversized entry is dropped by the per-entry bound, so the digest still matches.
130+
remember("resp_amp_replaced_oversized", "y".repeat(3 * 1024 * 1024));
131+
await flushResponseState();
132+
133+
expect(readFileSync(snapshot, "utf-8")).toBe(original);
134+
});
135+
111136
test("the scheduled debounce stays at its base value for a small snapshot", async () => {
112137
remember("resp_amp_tiny", "tiny");
113138
await flushResponseState();
@@ -133,6 +158,50 @@ describe("responses-state snapshot write amplification (#2460)", () => {
133158
expect(delay).toBeGreaterThanOrEqual(5_000);
134159
});
135160

161+
// The ceiling is the half of the scaling rule that a proportional formula gets
162+
// wrong silently: `<= 30_000` passes for any well-behaved input, so it proves
163+
// the clamp exists only when something actually reaches it. A snapshot at the
164+
// byte cap is the case that does.
165+
test("the scheduled debounce clamps to exactly 30s at the snapshot cap", async () => {
166+
// The scaling rule is 2s per MiB, so a payload at or past 15 MiB computes above
167+
// the ceiling. 900 KB of text per entry matters: the entry is serialized with the
168+
// input echoed alongside the output, so ~1.5 MB of text lands past the 2 MiB
169+
// per-entry bound and is skipped entirely, leaving an empty snapshot.
170+
for (let i = 0; i < 20; i += 1) remember(`resp_amp_cap_${i}`, "c".repeat(900 * 1024));
171+
await flushResponseState();
172+
// Guard the premise: if the snapshot did not actually get large, a passing
173+
// clamp assertion below would be proving nothing.
174+
expect(statSync(snapshot).size).toBeGreaterThan(15 * 1024 * 1024);
175+
176+
const delay = scheduledDelay(() => remember("resp_amp_cap_next", "next"));
177+
await flushResponseState();
178+
179+
expect(delay).toBe(30_000);
180+
});
181+
182+
// Shutdown is the one moment the debounce must not apply: a pending write that
183+
// waits out a 30s timer during a graceful stop is a lost turn, and the longer
184+
// the debounce grows the more there is to lose. The existing tests all flush
185+
// immediately after each change, so none of them proves the pending case.
186+
test("a graceful flush writes changes still sitting behind the debounce", async () => {
187+
remember("resp_amp_drain_seed", "seed");
188+
await flushResponseState();
189+
190+
// Two changes with NO flush between them: the debounce timer is pending and
191+
// neither has reached disk when the graceful flush arrives.
192+
remember("resp_amp_drain_late", "late");
193+
remember("resp_amp_drain_late2", "late2");
194+
195+
await flushResponseState();
196+
197+
const parsed = JSON.parse(await Bun.file(snapshot).text()) as {
198+
states: [string, Record<string, unknown>][];
199+
};
200+
const ids = parsed.states.map(([id]) => id);
201+
expect(ids).toContain("resp_amp_drain_late");
202+
expect(ids).toContain("resp_amp_drain_late2");
203+
});
204+
136205
test("the snapshot still round-trips after a skipped write", async () => {
137206
remember("resp_amp_roundtrip", "payload");
138207
await flushResponseState();

0 commit comments

Comments
 (0)