Skip to content

Commit a00a723

Browse files
ntdat812lidge-jun
andauthored
fix(responses): stop rewriting an unchanged snapshot every two seconds (lidge-jun#2476)
* fix(responses): stop rewriting an unchanged snapshot every two seconds `responses-state.json` is bounded at 24 MiB and rewritten whole on a fixed 2 s debounce, so under sustained traffic every cycle paid a complete re-serialization plus an atomic replacement of a file nothing reads until the next start. Two narrow measures, both scoped to the write path: - A flush that would reproduce the existing file byte-for-byte is skipped. A mutation does not always change what gets persisted — entries past the per-entry or total bound are dropped from the selection, and spill demotion moves bytes out of it. The comparison is a length plus a Bun.hash digest rather than the retained payload, which at the 24 MiB bound would double the snapshot's memory cost. The skip is conditional on the file still existing, so a snapshot deleted underneath the process is restored. - The debounce scales with the size of the last snapshot written: base 2 s below 1 MiB, linear above it, clamped at 30 s. The write rate is then roughly flat as the cache grows instead of growing with it. Durability is unchanged for a graceful shutdown, which flushes; a longer debounce only widens the window in which a hard kill loses the most recent continuation entries, which are cache. Journal / incremental store deliberately not attempted here. Refs lidge-jun#2460 * docs(troubleshooting): say the debounce follows the last snapshot written The section read as though the cadence tracked the pending snapshot. It tracks the size of the last snapshot actually written, so a cache that has only just grown still takes the short wait once. Review feedback on lidge-jun#2476. * fix(responses): verify the snapshot on disk before skipping a write Skipping a byte-identical rewrite is the right fix for the amplification, but the cached digest describes what this process last wrote, which is not the same claim as what is on disk now. A second proxy sharing the home, or anything that rewrites the file in place, leaves the digest describing bytes that are gone. That matters more than it sounds. Before the skip existed, every flush rewrote the file and so repaired external damage silently. Skipping on the digest alone turns a self-healing snapshot into a permanently corrupt one, and nothing notices until the next restart fails to load the continuation state. Replacing the file with different bytes of the same length reproduces it: the digest still matches, the file still exists, and the flush declines to repair. The skip now verifies identity against the file itself. The cached digest is keyed to the resolved write target, so a config-dir change or a retargeted symlink is a miss rather than a false match, and the contents are compared byte-for-byte before declining to write. Size is checked first so the common mismatch costs a stat, any read failure answers "no" and the caller rewrites, and the read only happens when the digest already agreed. The amplification being fixed is the repeated 24 MiB atomic replace, not the read that avoids it. This also removes the need to trust Bun.hash for correctness. It stays a cheap first filter, but a collision can no longer produce a false skip. Tests: same-length external replacement must be rewritten, proven by reverting only the disk check. The docs line claiming graceful shutdown "always flushes" is corrected too — the flush is a disk write and can fail like any other, and writeBoundedSnapshot swallows that into a "failed" outcome the lifecycle warning cannot see. 124 pass across write-amplification and responses-state. tsc clean. --------- Co-authored-by: bitkyc08-arch <[email protected]>
1 parent 77c081d commit a00a723

3 files changed

Lines changed: 320 additions & 4 deletions

File tree

‎docs-site/src/content/docs/troubleshooting/disk-usage-temp-files.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,25 @@ running process can own those.
5050
The safety rules are unchanged: a file younger than 15 minutes is never removed,
5151
and the proxy never removes a file it is writing itself.
5252

53+
## How often the snapshot is written
54+
55+
Writes are debounced, and the debounce is derived from the size of the **last
56+
snapshot actually written**: while that file is small the next write is scheduled
57+
about two seconds after a change, and once it is near the 24 MB bound the wait
58+
stretches to at most thirty seconds. A cache that has only just grown therefore
59+
still takes the short wait once — the longer cadence applies from the write after
60+
it. A flush that would reproduce the existing file byte-for-byte is skipped
61+
entirely.
62+
63+
Together these keep the write rate roughly flat as the cache grows, instead of
64+
re-serializing and replacing the whole file every two seconds.
65+
66+
A graceful shutdown flushes immediately rather than waiting out the timer, so the
67+
longer wait mainly widens the window in which a hard kill loses the most recent
68+
continuation entries — which are cache, as above. That flush is still a disk
69+
write and can fail like any other, so a shutdown on a full or read-only volume
70+
can lose the same entries.
71+
5372
## Reclaiming files that already accumulated
5473

5574
If the proxy runs, this happens automatically within a minute or two.

‎src/responses/state.ts‎

Lines changed: 83 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@ import {
1818
const MAX_STORED_RESPONSES = 1_000;
1919
const RESPONSE_TTL_MS = 60 * 60 * 1_000;
2020
const SNAPSHOT_DEBOUNCE_MS = 2_000;
21+
/** Snapshot size below which the debounce stays at its base value. */
22+
const SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES = 1 * 1024 * 1024;
23+
/** Ceiling for the stretched debounce. Continuation state is only read after a
24+
* restart, and a graceful shutdown flushes, so the exposure a longer debounce adds
25+
* is bounded by a hard kill — paid against rewriting the whole snapshot every 2 s. */
26+
const SNAPSHOT_DEBOUNCE_MAX_MS = 30_000;
2127
/** In-memory high-water byte cap across all entries. Forced store:false retention (kiro/cursor
2228
* continuation chains) stores the full expanded input each turn — ~quadratic bytes per chain —
2329
* so a count cap alone cannot bound memory. Oldest-first eviction applies past this mark. */
@@ -90,6 +96,32 @@ let oldestResidentId: string | undefined;
9096
let oldestResidentAt: number | null = null;
9197
let byteCapOverride: number | null = null;
9298
let stateRevision = 0;
99+
/** Byte length and digest of the last snapshot actually written, for the
100+
* identical-payload skip and the size-scaled debounce. The payload itself is not
101+
* retained: at the 24 MiB bound that would double the snapshot's memory cost. */
102+
let lastSnapshotBytes = 0;
103+
let lastSnapshotDigest: string | null = null;
104+
// The resolved file the digest above describes. Keeping it means a config-dir
105+
// change or a retargeted symlink is a miss rather than a false "unchanged".
106+
let lastSnapshotTarget: string | null = null;
107+
108+
/**
109+
* Is the snapshot on disk still byte-for-byte what we last wrote?
110+
*
111+
* The cached digest proves what this process wrote, not what is there now. Size is
112+
* checked first so the common mismatch costs a `stat`, and the content comparison
113+
* only runs when the size already agrees. Any read failure answers "no" and the
114+
* caller rewrites — the safe direction.
115+
*/
116+
async function snapshotOnDiskMatches(path: string, payload: string, payloadBytes: number): Promise<boolean> {
117+
try {
118+
const file = Bun.file(path);
119+
if (file.size !== payloadBytes) return false;
120+
return await file.text() === payload;
121+
} catch {
122+
return false;
123+
}
124+
}
93125
const spillCounters = { writes: 0, writeFailures: 0, readFailures: 0 };
94126
/**
95127
* Admission-boundary observability (test-visible). directSpills: oversized
@@ -788,9 +820,38 @@ async function writeBoundedSnapshot(path: string): Promise<SnapshotWriteOutcome>
788820
entries.push(persistEntry);
789821
}
790822
entries.reverse();
791-
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
792-
try { chmodSync(dirname(path), 0o700); } catch { /* best-effort (e.g. Windows) */ }
793-
await atomicWriteFileAsync(path, JSON.stringify({ version: 2, states: entries }));
823+
const payload = JSON.stringify({ version: 2, states: entries });
824+
const payloadBytes = Buffer.byteLength(payload, "utf8");
825+
const payloadDigest = Bun.hash(payload).toString(36);
826+
// A mutation does not always change what gets persisted: entries past the
827+
// per-entry or total byte bound are dropped from the selection, and spill
828+
// demotion moves bytes out of it. Re-writing a byte-identical 24 MiB file
829+
// buys nothing, so compare first — but the cached digest describes what THIS
830+
// process last wrote, which is not the same claim as "that is what is on disk
831+
// now". A second proxy sharing the home, or anything that rewrites the file
832+
// in place, leaves the digest describing bytes that are gone. Before every
833+
// release-of-a-write, the previous behaviour rewrote unconditionally and so
834+
// repaired that silently; skipping without checking would turn a repaired
835+
// snapshot into a lost one at the next restart.
836+
//
837+
// Verify against the file itself, keyed to the resolved target so a retargeted
838+
// symlink is also a miss. Reading back a matching-size file costs far less
839+
// than the atomic replace it avoids, and only happens when the digest already
840+
// matched — the amplification this fixes is the repeated WRITE, not the read.
841+
const unchanged = lastSnapshotDigest !== null
842+
&& payloadDigest === lastSnapshotDigest
843+
&& payloadBytes === lastSnapshotBytes
844+
&& lastSnapshotTarget === resolveWriteTarget(path)
845+
&& existsSync(path)
846+
&& await snapshotOnDiskMatches(path, payload, payloadBytes);
847+
if (!unchanged) {
848+
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
849+
try { chmodSync(dirname(path), 0o700); } catch { /* best-effort (e.g. Windows) */ }
850+
await atomicWriteFileAsync(path, payload);
851+
lastSnapshotDigest = payloadDigest;
852+
lastSnapshotBytes = payloadBytes;
853+
lastSnapshotTarget = resolveWriteTarget(path);
854+
}
794855
persistAttemptHookForTests?.();
795856
if (revision === stateRevision) return "stable";
796857
}
@@ -809,11 +870,26 @@ function drainPendingSpillUnlinks(): void {
809870
}
810871
}
811872

873+
/**
874+
* Debounce scaled by the size of the last snapshot written.
875+
*
876+
* The whole snapshot is re-serialized and atomically replaced on every flush, so at
877+
* the 24 MiB bound a fixed 2 s debounce is up to ~12 MB/s of write amplification for
878+
* state nothing reads until the next start (#2460). Small snapshots keep the base
879+
* cadence; the stretch is linear in size and clamped, so the write rate is roughly
880+
* flat instead of growing with the file.
881+
*/
882+
function snapshotDebounceMs(): number {
883+
if (lastSnapshotBytes <= SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES) return SNAPSHOT_DEBOUNCE_MS;
884+
const scaled = Math.round(SNAPSHOT_DEBOUNCE_MS * (lastSnapshotBytes / SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES));
885+
return Math.min(scaled, SNAPSHOT_DEBOUNCE_MAX_MS);
886+
}
887+
812888
function schedulePersistAt(path: string, replace = false): void {
813889
if (persistTimer && !replace) return;
814890
if (persistTimer) clearTimeout(persistTimer);
815891
pendingPersistPath = path;
816-
persistTimer = setTimeout(() => { void persistNow(path); }, SNAPSHOT_DEBOUNCE_MS);
892+
persistTimer = setTimeout(() => { void persistNow(path); }, snapshotDebounceMs());
817893
(persistTimer as { unref?: () => void }).unref?.();
818894
}
819895

@@ -1418,6 +1494,9 @@ export function clearResponseStateMemoryForTests(): void {
14181494
replayScopeMismatchDrops = 0;
14191495
replayOverlapSkips = 0;
14201496
persistAttemptHookForTests = null;
1497+
lastSnapshotBytes = 0;
1498+
lastSnapshotDigest = null;
1499+
lastSnapshotTarget = null;
14211500
loaded = false;
14221501
}
14231502

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
/**
2+
* #2460 — `responses-state.json` was rewritten in full on a fixed 2 s debounce.
3+
*
4+
* The snapshot is bounded at 24 MiB, so under sustained traffic every debounce
5+
* paid a complete re-serialization plus an atomic replacement of a file nothing
6+
* reads until the next start. Two narrow measures are covered here: a
7+
* byte-identical payload is not rewritten, and the debounce scales with the size
8+
* of the snapshot actually being written.
9+
*/
10+
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
11+
import { existsSync, mkdtempSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync } from "node:fs";
12+
import { tmpdir } from "node:os";
13+
import { join } from "node:path";
14+
import {
15+
clearResponseStateForTests,
16+
clearResponseStateMemoryForTests,
17+
flushResponseState,
18+
rememberResponseState,
19+
setResponseStateByteCapForTests,
20+
} from "../src/responses/state";
21+
22+
function remember(id: string, text: string): void {
23+
rememberResponseState(
24+
{ model: "test/model", input: text, store: false },
25+
{ id, output: [{ type: "message", role: "assistant", content: text }], status: "completed" },
26+
undefined,
27+
{ force: true },
28+
);
29+
}
30+
31+
describe("responses-state snapshot write amplification (#2460)", () => {
32+
let home: string;
33+
const priorHome = process.env["OPENCODEX_HOME"];
34+
let snapshot: string;
35+
36+
beforeEach(() => {
37+
home = mkdtempSync(join(tmpdir(), "ocx-state-amp-"));
38+
process.env["OPENCODEX_HOME"] = home;
39+
snapshot = join(home, "responses-state.json");
40+
clearResponseStateMemoryForTests();
41+
});
42+
43+
afterEach(() => {
44+
setResponseStateByteCapForTests(null);
45+
clearResponseStateForTests();
46+
rmSync(home, { recursive: true, force: true });
47+
if (priorHome === undefined) delete process.env["OPENCODEX_HOME"];
48+
else process.env["OPENCODEX_HOME"] = priorHome;
49+
});
50+
51+
/** Record the delay the store hands to setTimeout when it schedules its next write. */
52+
function scheduledDelay(schedule: () => void): number {
53+
const realSetTimeout = globalThis.setTimeout;
54+
const delays: number[] = [];
55+
globalThis.setTimeout = ((handler: TimerHandler, ms?: number, ...rest: unknown[]) => {
56+
delays.push(ms ?? 0);
57+
return (realSetTimeout as (...a: unknown[]) => unknown)(handler, ms, ...rest);
58+
}) as unknown as typeof setTimeout;
59+
try {
60+
schedule();
61+
} finally {
62+
globalThis.setTimeout = realSetTimeout;
63+
}
64+
expect(delays).toHaveLength(1);
65+
return delays[0]!;
66+
}
67+
68+
/** Backdate the snapshot so "was it rewritten?" is a mtime comparison, not a clock race. */
69+
function backdate(): number {
70+
const past = new Date(Date.now() - 60_000);
71+
utimesSync(snapshot, past, past);
72+
return statSync(snapshot).mtimeMs;
73+
}
74+
75+
test("a flush that would reproduce the same bytes does not rewrite the file", async () => {
76+
remember("resp_amp_small", "kept");
77+
await flushResponseState();
78+
expect(existsSync(snapshot)).toBe(true);
79+
const before = backdate();
80+
81+
// An entry past the 2 MiB per-entry bound is dropped from the selection, so
82+
// recording it mutates state without changing a single persisted byte.
83+
remember("resp_amp_oversized", "x".repeat(3 * 1024 * 1024));
84+
await flushResponseState();
85+
86+
expect(statSync(snapshot).mtimeMs).toBe(before);
87+
});
88+
89+
test("a flush that changes the payload still rewrites the file", async () => {
90+
remember("resp_amp_first", "first");
91+
await flushResponseState();
92+
const before = backdate();
93+
94+
remember("resp_amp_second", "second");
95+
await flushResponseState();
96+
97+
expect(statSync(snapshot).mtimeMs).toBeGreaterThan(before);
98+
});
99+
100+
test("a snapshot deleted underneath us is rewritten even when the payload matches", async () => {
101+
remember("resp_amp_restore", "kept");
102+
await flushResponseState();
103+
rmSync(snapshot, { force: true });
104+
105+
remember("resp_amp_oversized_2", "y".repeat(3 * 1024 * 1024));
106+
await flushResponseState();
107+
108+
expect(existsSync(snapshot)).toBe(true);
109+
});
110+
111+
// Deletion is the easy half. The cached digest describes what THIS process last
112+
// wrote, which is not the same claim as "that is what is on disk now" — a second
113+
// proxy sharing the home, or anything rewriting the file in place, leaves the
114+
// digest describing bytes that are gone. Before the skip existed, every flush
115+
// rewrote and so repaired that silently; skipping on the digest alone would turn
116+
// a self-healing snapshot into a permanently corrupt one, discovered only at the
117+
// next restart when the continuation state fails to load.
118+
//
119+
// Same-length replacement is the case that defeats a size-only check, so that is
120+
// what this drives.
121+
test("a snapshot replaced with different bytes of the same length is rewritten", async () => {
122+
remember("resp_amp_replaced", "kept");
123+
await flushResponseState();
124+
125+
const original = readFileSync(snapshot, "utf-8");
126+
writeFileSync(snapshot, "X".repeat(Buffer.byteLength(original, "utf8")));
127+
128+
// A mutation whose bounded payload is byte-identical to the last write: the
129+
// oversized entry is dropped by the per-entry bound, so the digest still matches.
130+
remember("resp_amp_replaced_oversized", "y".repeat(3 * 1024 * 1024));
131+
await flushResponseState();
132+
133+
expect(readFileSync(snapshot, "utf-8")).toBe(original);
134+
});
135+
136+
test("the scheduled debounce stays at its base value for a small snapshot", async () => {
137+
remember("resp_amp_tiny", "tiny");
138+
await flushResponseState();
139+
140+
const delay = scheduledDelay(() => remember("resp_amp_tiny_2", "tiny"));
141+
await flushResponseState();
142+
143+
expect(delay).toBe(2_000);
144+
});
145+
146+
test("the scheduled debounce stretches once the snapshot is large", async () => {
147+
// Four ~800 KiB entries: each under the 2 MiB per-entry bound, so all four are
148+
// persisted and the payload lands well past the 1 MiB scaling threshold.
149+
for (let i = 0; i < 4; i += 1) remember(`resp_amp_big_${i}`, "z".repeat(800 * 1024));
150+
await flushResponseState();
151+
152+
const delay = scheduledDelay(() => remember("resp_amp_big_next", "next"));
153+
await flushResponseState();
154+
155+
expect(delay).toBeGreaterThan(2_000);
156+
expect(delay).toBeLessThanOrEqual(30_000);
157+
// Roughly proportional to size: ~3.2 MiB of payload is ~6 s, not ~2 s.
158+
expect(delay).toBeGreaterThanOrEqual(5_000);
159+
});
160+
161+
// The ceiling is the half of the scaling rule that a proportional formula gets
162+
// wrong silently: `<= 30_000` passes for any well-behaved input, so it proves
163+
// the clamp exists only when something actually reaches it. A snapshot at the
164+
// byte cap is the case that does.
165+
test("the scheduled debounce clamps to exactly 30s at the snapshot cap", async () => {
166+
// The scaling rule is 2s per MiB, so a payload at or past 15 MiB computes above
167+
// the ceiling. 900 KB of text per entry matters: the entry is serialized with the
168+
// input echoed alongside the output, so ~1.5 MB of text lands past the 2 MiB
169+
// per-entry bound and is skipped entirely, leaving an empty snapshot.
170+
for (let i = 0; i < 20; i += 1) remember(`resp_amp_cap_${i}`, "c".repeat(900 * 1024));
171+
await flushResponseState();
172+
// Guard the premise: if the snapshot did not actually get large, a passing
173+
// clamp assertion below would be proving nothing.
174+
expect(statSync(snapshot).size).toBeGreaterThan(15 * 1024 * 1024);
175+
176+
const delay = scheduledDelay(() => remember("resp_amp_cap_next", "next"));
177+
await flushResponseState();
178+
179+
expect(delay).toBe(30_000);
180+
});
181+
182+
// Shutdown is the one moment the debounce must not apply: a pending write that
183+
// waits out a 30s timer during a graceful stop is a lost turn, and the longer
184+
// the debounce grows the more there is to lose. The existing tests all flush
185+
// immediately after each change, so none of them proves the pending case.
186+
test("a graceful flush writes changes still sitting behind the debounce", async () => {
187+
remember("resp_amp_drain_seed", "seed");
188+
await flushResponseState();
189+
190+
// Two changes with NO flush between them: the debounce timer is pending and
191+
// neither has reached disk when the graceful flush arrives.
192+
remember("resp_amp_drain_late", "late");
193+
remember("resp_amp_drain_late2", "late2");
194+
195+
await flushResponseState();
196+
197+
const parsed = JSON.parse(await Bun.file(snapshot).text()) as {
198+
states: [string, Record<string, unknown>][];
199+
};
200+
const ids = parsed.states.map(([id]) => id);
201+
expect(ids).toContain("resp_amp_drain_late");
202+
expect(ids).toContain("resp_amp_drain_late2");
203+
});
204+
205+
test("the snapshot still round-trips after a skipped write", async () => {
206+
remember("resp_amp_roundtrip", "payload");
207+
await flushResponseState();
208+
remember("resp_amp_roundtrip_oversized", "w".repeat(3 * 1024 * 1024));
209+
await flushResponseState();
210+
211+
const parsed = JSON.parse(await Bun.file(snapshot).text()) as {
212+
version: number;
213+
states: [string, Record<string, unknown>][];
214+
};
215+
expect(parsed.version).toBe(2);
216+
expect(parsed.states.map(([id]) => id)).toContain("resp_amp_roundtrip");
217+
});
218+
});

0 commit comments

Comments
 (0)