Skip to content

Commit 2f72a47

Browse files
committed
fix(responses): stop rewriting an unchanged snapshot every two seconds
`responses-state.json` is bounded at 24 MiB and rewritten whole on a fixed 2 s debounce, so under sustained traffic every cycle paid a complete re-serialization plus an atomic replacement of a file nothing reads until the next start. Two narrow measures, both scoped to the write path: - A flush that would reproduce the existing file byte-for-byte is skipped. A mutation does not always change what gets persisted — entries past the per-entry or total bound are dropped from the selection, and spill demotion moves bytes out of it. The comparison is a length plus a Bun.hash digest rather than the retained payload, which at the 24 MiB bound would double the snapshot's memory cost. The skip is conditional on the file still existing, so a snapshot deleted underneath the process is restored. - The debounce scales with the size of the last snapshot written: base 2 s below 1 MiB, linear above it, clamped at 30 s. The write rate is then roughly flat as the cache grows instead of growing with it. Durability is unchanged for a graceful shutdown, which flushes; a longer debounce only widens the window in which a hard kill loses the most recent continuation entries, which are cache. Journal / incremental store deliberately not attempted here. Refs #2460
1 parent c44e43f commit 2f72a47

3 files changed

Lines changed: 211 additions & 4 deletions

File tree

‎docs-site/src/content/docs/troubleshooting/disk-usage-temp-files.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,20 @@ running process can own those.
5050
The safety rules are unchanged: a file younger than 15 minutes is never removed,
5151
and the proxy never removes a file it is writing itself.
5252

53+
## How often the snapshot is written
54+
55+
Writes are debounced, and the debounce scales with how large the snapshot
56+
actually is: a small cache is written about two seconds after a change, while one
57+
near the 24 MB bound waits up to thirty seconds. A flush that would reproduce the
58+
existing file byte-for-byte is skipped entirely.
59+
60+
Together these keep the write rate roughly flat as the cache grows, instead of
61+
re-serializing and replacing the whole file every two seconds.
62+
63+
A graceful shutdown always flushes, so the longer wait only widens the window in
64+
which a hard kill loses the most recent continuation entries — which are cache,
65+
as above.
66+
5367
## Reclaiming files that already accumulated
5468

5569
If the proxy runs, this happens automatically within a minute or two.

‎src/responses/state.ts‎

Lines changed: 48 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@ import {
1818
const MAX_STORED_RESPONSES = 1_000;
1919
const RESPONSE_TTL_MS = 60 * 60 * 1_000;
2020
const SNAPSHOT_DEBOUNCE_MS = 2_000;
21+
/** Snapshot size below which the debounce stays at its base value. */
22+
const SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES = 1 * 1024 * 1024;
23+
/** Ceiling for the stretched debounce. Continuation state is only read after a
24+
* restart, and a graceful shutdown flushes, so the exposure a longer debounce adds
25+
* is bounded by a hard kill — paid against rewriting the whole snapshot every 2 s. */
26+
const SNAPSHOT_DEBOUNCE_MAX_MS = 30_000;
2127
/** In-memory high-water byte cap across all entries. Forced store:false retention (kiro/cursor
2228
* continuation chains) stores the full expanded input each turn — ~quadratic bytes per chain —
2329
* so a count cap alone cannot bound memory. Oldest-first eviction applies past this mark. */
@@ -90,6 +96,11 @@ let oldestResidentId: string | undefined;
9096
let oldestResidentAt: number | null = null;
9197
let byteCapOverride: number | null = null;
9298
let stateRevision = 0;
99+
/** Byte length and digest of the last snapshot actually written, for the
100+
* identical-payload skip and the size-scaled debounce. The payload itself is not
101+
* retained: at the 24 MiB bound that would double the snapshot's memory cost. */
102+
let lastSnapshotBytes = 0;
103+
let lastSnapshotDigest: string | null = null;
93104
const spillCounters = { writes: 0, writeFailures: 0, readFailures: 0 };
94105
/**
95106
* Admission-boundary observability (test-visible). directSpills: oversized
@@ -788,9 +799,25 @@ async function writeBoundedSnapshot(path: string): Promise<SnapshotWriteOutcome>
788799
entries.push(persistEntry);
789800
}
790801
entries.reverse();
791-
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
792-
try { chmodSync(dirname(path), 0o700); } catch { /* best-effort (e.g. Windows) */ }
793-
await atomicWriteFileAsync(path, JSON.stringify({ version: 2, states: entries }));
802+
const payload = JSON.stringify({ version: 2, states: entries });
803+
const payloadBytes = Buffer.byteLength(payload, "utf8");
804+
const payloadDigest = Bun.hash(payload).toString(36);
805+
// A mutation does not always change what gets persisted: entries past the
806+
// per-entry or total byte bound are dropped from the selection, and spill
807+
// demotion moves bytes out of it. Re-writing a byte-identical 24 MiB file
808+
// buys nothing, so compare first — but only skip while the file we would be
809+
// reproducing is still there.
810+
const unchanged = lastSnapshotDigest !== null
811+
&& payloadDigest === lastSnapshotDigest
812+
&& payloadBytes === lastSnapshotBytes
813+
&& existsSync(path);
814+
if (!unchanged) {
815+
mkdirSync(dirname(path), { recursive: true, mode: 0o700 });
816+
try { chmodSync(dirname(path), 0o700); } catch { /* best-effort (e.g. Windows) */ }
817+
await atomicWriteFileAsync(path, payload);
818+
lastSnapshotDigest = payloadDigest;
819+
lastSnapshotBytes = payloadBytes;
820+
}
794821
persistAttemptHookForTests?.();
795822
if (revision === stateRevision) return "stable";
796823
}
@@ -809,11 +836,26 @@ function drainPendingSpillUnlinks(): void {
809836
}
810837
}
811838

839+
/**
840+
* Debounce scaled by the size of the last snapshot written.
841+
*
842+
* The whole snapshot is re-serialized and atomically replaced on every flush, so at
843+
* the 24 MiB bound a fixed 2 s debounce is up to ~12 MB/s of write amplification for
844+
* state nothing reads until the next start (#2460). Small snapshots keep the base
845+
* cadence; the stretch is linear in size and clamped, so the write rate is roughly
846+
* flat instead of growing with the file.
847+
*/
848+
function snapshotDebounceMs(): number {
849+
if (lastSnapshotBytes <= SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES) return SNAPSHOT_DEBOUNCE_MS;
850+
const scaled = Math.round(SNAPSHOT_DEBOUNCE_MS * (lastSnapshotBytes / SNAPSHOT_DEBOUNCE_SCALE_FROM_BYTES));
851+
return Math.min(scaled, SNAPSHOT_DEBOUNCE_MAX_MS);
852+
}
853+
812854
function schedulePersistAt(path: string, replace = false): void {
813855
if (persistTimer && !replace) return;
814856
if (persistTimer) clearTimeout(persistTimer);
815857
pendingPersistPath = path;
816-
persistTimer = setTimeout(() => { void persistNow(path); }, SNAPSHOT_DEBOUNCE_MS);
858+
persistTimer = setTimeout(() => { void persistNow(path); }, snapshotDebounceMs());
817859
(persistTimer as { unref?: () => void }).unref?.();
818860
}
819861

@@ -1418,6 +1460,8 @@ export function clearResponseStateMemoryForTests(): void {
14181460
replayScopeMismatchDrops = 0;
14191461
replayOverlapSkips = 0;
14201462
persistAttemptHookForTests = null;
1463+
lastSnapshotBytes = 0;
1464+
lastSnapshotDigest = null;
14211465
loaded = false;
14221466
}
14231467

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
/**
2+
* #2460 — `responses-state.json` was rewritten in full on a fixed 2 s debounce.
3+
*
4+
* The snapshot is bounded at 24 MiB, so under sustained traffic every debounce
5+
* paid a complete re-serialization plus an atomic replacement of a file nothing
6+
* reads until the next start. Two narrow measures are covered here: a
7+
* byte-identical payload is not rewritten, and the debounce scales with the size
8+
* of the snapshot actually being written.
9+
*/
10+
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
11+
import { existsSync, mkdtempSync, rmSync, statSync, utimesSync } from "node:fs";
12+
import { tmpdir } from "node:os";
13+
import { join } from "node:path";
14+
import {
15+
clearResponseStateForTests,
16+
clearResponseStateMemoryForTests,
17+
flushResponseState,
18+
rememberResponseState,
19+
setResponseStateByteCapForTests,
20+
} from "../src/responses/state";
21+
22+
function remember(id: string, text: string): void {
23+
rememberResponseState(
24+
{ model: "test/model", input: text, store: false },
25+
{ id, output: [{ type: "message", role: "assistant", content: text }], status: "completed" },
26+
undefined,
27+
{ force: true },
28+
);
29+
}
30+
31+
describe("responses-state snapshot write amplification (#2460)", () => {
32+
let home: string;
33+
const priorHome = process.env["OPENCODEX_HOME"];
34+
let snapshot: string;
35+
36+
beforeEach(() => {
37+
home = mkdtempSync(join(tmpdir(), "ocx-state-amp-"));
38+
process.env["OPENCODEX_HOME"] = home;
39+
snapshot = join(home, "responses-state.json");
40+
clearResponseStateMemoryForTests();
41+
});
42+
43+
afterEach(() => {
44+
setResponseStateByteCapForTests(null);
45+
clearResponseStateForTests();
46+
rmSync(home, { recursive: true, force: true });
47+
if (priorHome === undefined) delete process.env["OPENCODEX_HOME"];
48+
else process.env["OPENCODEX_HOME"] = priorHome;
49+
});
50+
51+
/** Record the delay the store hands to setTimeout when it schedules its next write. */
52+
function scheduledDelay(schedule: () => void): number {
53+
const realSetTimeout = globalThis.setTimeout;
54+
const delays: number[] = [];
55+
globalThis.setTimeout = ((handler: TimerHandler, ms?: number, ...rest: unknown[]) => {
56+
delays.push(ms ?? 0);
57+
return (realSetTimeout as (...a: unknown[]) => unknown)(handler, ms, ...rest);
58+
}) as unknown as typeof setTimeout;
59+
try {
60+
schedule();
61+
} finally {
62+
globalThis.setTimeout = realSetTimeout;
63+
}
64+
expect(delays).toHaveLength(1);
65+
return delays[0]!;
66+
}
67+
68+
/** Backdate the snapshot so "was it rewritten?" is a mtime comparison, not a clock race. */
69+
function backdate(): number {
70+
const past = new Date(Date.now() - 60_000);
71+
utimesSync(snapshot, past, past);
72+
return statSync(snapshot).mtimeMs;
73+
}
74+
75+
test("a flush that would reproduce the same bytes does not rewrite the file", async () => {
76+
remember("resp_amp_small", "kept");
77+
await flushResponseState();
78+
expect(existsSync(snapshot)).toBe(true);
79+
const before = backdate();
80+
81+
// An entry past the 2 MiB per-entry bound is dropped from the selection, so
82+
// recording it mutates state without changing a single persisted byte.
83+
remember("resp_amp_oversized", "x".repeat(3 * 1024 * 1024));
84+
await flushResponseState();
85+
86+
expect(statSync(snapshot).mtimeMs).toBe(before);
87+
});
88+
89+
test("a flush that changes the payload still rewrites the file", async () => {
90+
remember("resp_amp_first", "first");
91+
await flushResponseState();
92+
const before = backdate();
93+
94+
remember("resp_amp_second", "second");
95+
await flushResponseState();
96+
97+
expect(statSync(snapshot).mtimeMs).toBeGreaterThan(before);
98+
});
99+
100+
test("a snapshot deleted underneath us is rewritten even when the payload matches", async () => {
101+
remember("resp_amp_restore", "kept");
102+
await flushResponseState();
103+
rmSync(snapshot, { force: true });
104+
105+
remember("resp_amp_oversized_2", "y".repeat(3 * 1024 * 1024));
106+
await flushResponseState();
107+
108+
expect(existsSync(snapshot)).toBe(true);
109+
});
110+
111+
test("the scheduled debounce stays at its base value for a small snapshot", async () => {
112+
remember("resp_amp_tiny", "tiny");
113+
await flushResponseState();
114+
115+
const delay = scheduledDelay(() => remember("resp_amp_tiny_2", "tiny"));
116+
await flushResponseState();
117+
118+
expect(delay).toBe(2_000);
119+
});
120+
121+
test("the scheduled debounce stretches once the snapshot is large", async () => {
122+
// Four ~800 KiB entries: each under the 2 MiB per-entry bound, so all four are
123+
// persisted and the payload lands well past the 1 MiB scaling threshold.
124+
for (let i = 0; i < 4; i += 1) remember(`resp_amp_big_${i}`, "z".repeat(800 * 1024));
125+
await flushResponseState();
126+
127+
const delay = scheduledDelay(() => remember("resp_amp_big_next", "next"));
128+
await flushResponseState();
129+
130+
expect(delay).toBeGreaterThan(2_000);
131+
expect(delay).toBeLessThanOrEqual(30_000);
132+
// Roughly proportional to size: ~3.2 MiB of payload is ~6 s, not ~2 s.
133+
expect(delay).toBeGreaterThanOrEqual(5_000);
134+
});
135+
136+
test("the snapshot still round-trips after a skipped write", async () => {
137+
remember("resp_amp_roundtrip", "payload");
138+
await flushResponseState();
139+
remember("resp_amp_roundtrip_oversized", "w".repeat(3 * 1024 * 1024));
140+
await flushResponseState();
141+
142+
const parsed = JSON.parse(await Bun.file(snapshot).text()) as {
143+
version: number;
144+
states: [string, Record<string, unknown>][];
145+
};
146+
expect(parsed.version).toBe(2);
147+
expect(parsed.states.map(([id]) => id)).toContain("resp_amp_roundtrip");
148+
});
149+
});

0 commit comments

Comments
 (0)