Skip to content

Commit c1b051b

Browse files
fix(checkpoints): sync journal markers before rename, guard and merge marker entries
The two marker writers staged a file with fs.writeFile and renamed it without ever syncing it, so a crash could leave the marker name in place over content that never reached it. Both writers now take a write-mode handle on the staging file, write the whole payload, sync once, close, rename, and then sync the containing directory best effort - the order appendChange already uses for the journal itself. On this Windows box that directory sync opens with "r" and lands in appendChange's own EPERM tolerance list, so it degrades to a warning there; the content sync is what carries the guarantee, and the directory part is no stronger than the journal path's. reconcilePendingJournal read entry.path straight off marker.entries. A marker that parses can still carry a null or a half-shaped element, and the TypeError escaped loadChanges with the marker left in place: every later write-bearing checkpoint save then failed on the same corruption. The fold now runs entries through the same isChangeJournalEntry guard the incomplete fold at the sibling reconciler already uses. writeIncompleteJournal renamed over whatever marker was already on disk, so a second failed ready-promotion dropped the first batch of an already-committed checkpoint. It now unions with the entries the marker carries under the same path\0operation\0checkpointId key, preserving first-seen order: the earlier marker's entries keep their positions, the new batch is appended after them, and a repeated key is stored once. ApplyPatchTool recorded a move's destination write only after the source unlink and its error report, so a rejected task.say stranded a write that was already on disk and left the catch-path flush with nothing to record. The record now happens right after the destination write, before the unlink attempt and before bookkeeping. The index.ts comment that called the pre-commit 'awaiting' marker the recovery record for the next read is corrected: it carries no entries and reconcilePendingJournal clears it, so that branch cannot fall back on it. Tests first, each red against the pre-fix code and green after it, with the implementation reverted to its HEAD blob and restored from a recorded blob sha: reconcile with [valid, null, half-shaped] appends the valid entry and still clears the marker (was TypeError: Cannot read properties of null (reading 'path')); staged content synced before the rename and the directory after it, for both marker writers (was "expected [] to have a length of 1 but got +0"); a second incomplete write keeps the first batch (was "expected ['first'] to deeply equal ['first', 'second']"); a move whose unlink and error report both reject still journals the destination (was "expected \"vi.fn()\" to be called once, but got 0 times"). Two further cases characterise behaviour this commit deliberately does not change: the next read discards an awaiting marker left behind by a host exit between commit and promotion, and checkpointSave resolves undefined when the pre-commit marker write fails so the tool catch path reports only its own error. Counts from this same git diff --cached --numstat run: +110/-21 changeJournal.ts, +7/-4 index.ts, +8/-5 ApplyPatchTool.ts, +157/-1 changeJournal.spec.ts, +29/-0 checkpointJournal.test.ts, +51/-0 checkpointJournalPromotion.test.ts, +33/-0 applyPatchTool.execute.spec.ts.
1 parent e12e961 commit c1b051b

7 files changed

Lines changed: 395 additions & 31 deletions

File tree

‎src/core/checkpoints/__tests__/changeJournal.spec.ts‎

Lines changed: 157 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,16 @@ import path from "path"
44

55
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"
66

7-
import { appendChange, journalPath, loadChanges, type ChangeJournalEntry } from "../changeJournal"
7+
import {
8+
appendChange,
9+
incompleteJournalPath,
10+
journalPath,
11+
loadChanges,
12+
pendingJournalPath,
13+
writeIncompleteJournal,
14+
writePendingJournal,
15+
type ChangeJournalEntry,
16+
} from "../changeJournal"
817

918
describe("changeJournal", () => {
1019
const taskId = "test-task"
@@ -186,6 +195,153 @@ describe("changeJournal", () => {
186195
})
187196
})
188197

198+
describe("marker durability (the two staging+rename writers)", () => {
199+
const markerEntry: ChangeJournalEntry = { path: "src/foo.ts", operation: "create", checkpointId: "abc12345" }
200+
201+
/**
202+
* Replace fs/promises.open with a double that labels every handle by what it is for
203+
* (a ".staging" file gets "content", anything else is the containing "directory") and
204+
* records the order of open / sync / rename. The three journal writers only ever use
205+
* write, sync and close on a handle, so the double implements exactly those three and
206+
* delegates them to the real handle - the durability being asserted is the order the
207+
* writers call them in, which is what a crash actually leaves behind.
208+
*/
209+
function instrument(eventList: string[]) {
210+
const realOpen = fs.open
211+
const realRename = fs.rename
212+
const openSpy = vi.spyOn(fs, "open").mockImplementation(async (pathish, flags) => {
213+
const openAs = realOpen as (p: string | URL, f?: string | number) => Promise<fs.FileHandle>
214+
const handle = await openAs(pathish as string | URL, flags as string | number | undefined)
215+
const label = String(pathish).endsWith(".staging") ? "content" : "directory"
216+
eventList.push("open:" + label)
217+
const double = {
218+
write: (data: Buffer, offset?: number, length?: number) => handle.write(data, offset, length),
219+
sync: async () => {
220+
eventList.push("sync:" + label)
221+
return handle.sync()
222+
},
223+
close: () => handle.close(),
224+
} as unknown as fs.FileHandle
225+
return double
226+
})
227+
const renameSpy = vi.spyOn(fs, "rename").mockImplementation(async (fromish, toish) => {
228+
eventList.push("rename")
229+
return realRename(fromish, toish)
230+
})
231+
return () => {
232+
openSpy.mockRestore()
233+
renameSpy.mockRestore()
234+
}
235+
}
236+
237+
it("syncs the staged pending marker before renaming it into place", async () => {
238+
const events: string[] = []
239+
const restore = instrument(events)
240+
try {
241+
await writePendingJournal(tmpRoot, taskId, { state: "ready", createdAt: 1, entries: [markerEntry] })
242+
} finally {
243+
restore()
244+
}
245+
// Counted before ordered, so a writer that syncs nothing fails with a count
246+
// message rather than an index comparison against -1.
247+
expect(events.filter((e) => e === "open:content")).toHaveLength(1)
248+
expect(events.filter((e) => e === "sync:content")).toHaveLength(1)
249+
expect(events.filter((e) => e === "rename")).toHaveLength(1)
250+
// Renaming an un-synced staging file is the whole durability question: the
251+
// rename can land in a different order from the content it carries.
252+
expect(events.indexOf("sync:content")).toBeLessThan(events.indexOf("rename"))
253+
// The marker itself is still written, so the durability work changed nothing
254+
// about the state the recovery reader classifies.
255+
const marker = JSON.parse(await fs.readFile(pendingJournalPath(tmpRoot, taskId), "utf8"))
256+
expect(marker).toEqual({ state: "ready", createdAt: 1, entries: [markerEntry] })
257+
})
258+
259+
it("syncs the containing directory after the rename so the marker entry survives", async () => {
260+
const events: string[] = []
261+
const restore = instrument(events)
262+
try {
263+
await writePendingJournal(tmpRoot, taskId, { state: "awaiting-checkpoint", createdAt: 1 })
264+
} finally {
265+
restore()
266+
}
267+
// The marker's directory entry is what makes the marker findable after a
268+
// crash, and the writer creates that directory itself, so the directory must
269+
// be synced once, after the rename that first publishes the marker.
270+
expect(events.filter((e) => e === "open:directory")).toHaveLength(1)
271+
expect(events.filter((e) => e === "sync:directory")).toHaveLength(1)
272+
expect(events.indexOf("rename")).toBeLessThan(events.indexOf("sync:directory"))
273+
})
274+
275+
it("tolerates platforms that cannot sync the marker directory", async () => {
276+
const dir = path.dirname(pendingJournalPath(tmpRoot, taskId))
277+
const realOpen = fs.open
278+
// Captured locally rather than read back from the spy: mockRestore() clears a
279+
// spy's call history, and this test restores before it asserts.
280+
const warnings: string[] = []
281+
const warnSpy = vi.spyOn(console, "warn").mockImplementation((...args: unknown[]) => {
282+
warnings.push(String(args[0]))
283+
})
284+
const openSpy = vi.spyOn(fs, "open").mockImplementation(async (pathish, flags) => {
285+
if (String(pathish) === dir) {
286+
throw Object.assign(new Error("EPERM: operation not permitted"), { code: "EPERM" })
287+
}
288+
const openAs = realOpen as (p: string | URL, f?: string | number) => Promise<fs.FileHandle>
289+
return openAs(pathish as string | URL, flags as string | number | undefined)
290+
})
291+
try {
292+
await writePendingJournal(tmpRoot, taskId, { state: "ready", createdAt: 1, entries: [markerEntry] })
293+
} finally {
294+
openSpy.mockRestore()
295+
warnSpy.mockRestore()
296+
}
297+
// A platform without directory sync degrades to a warning; the marker itself
298+
// is durable through the content sync and must still be readable.
299+
const marker = JSON.parse(await fs.readFile(pendingJournalPath(tmpRoot, taskId), "utf8"))
300+
expect(marker.state).toBe("ready")
301+
expect(warnings.some((w) => w.includes("directory sync unsupported"))).toBe(true)
302+
})
303+
304+
it("syncs the staged incomplete marker before replacing the marker file", async () => {
305+
const events: string[] = []
306+
const restore = instrument(events)
307+
try {
308+
await writeIncompleteJournal(tmpRoot, taskId, [markerEntry])
309+
} finally {
310+
restore()
311+
}
312+
expect(events.filter((e) => e === "sync:content")).toHaveLength(1)
313+
expect(events.filter((e) => e === "rename")).toHaveLength(1)
314+
expect(events.indexOf("sync:content")).toBeLessThan(events.indexOf("rename"))
315+
// Same directory guarantee as the pending marker: this file is the recovery
316+
// record for a committed checkpoint whose entries never made the journal.
317+
expect(events.filter((e) => e === "sync:directory")).toHaveLength(1)
318+
expect(events.indexOf("rename")).toBeLessThan(events.indexOf("sync:directory"))
319+
const marker = JSON.parse(await fs.readFile(incompleteJournalPath(tmpRoot, taskId), "utf8"))
320+
expect(marker).toEqual({ entries: [markerEntry] })
321+
})
322+
})
323+
324+
describe("incomplete marker preservation", () => {
325+
it("preserves the entries of an existing incomplete marker instead of replacing them", async () => {
326+
// rename replaces its destination, and the only reader that folds this marker
327+
// back is reconcileIncompleteJournal, reached from loadChanges. Two failed
328+
// ready-promotions in the same session, with no load between them, therefore
329+
// used to lose the first batch: the second write replaced the first marker.
330+
await writeIncompleteJournal(tmpRoot, taskId, [entry({ checkpointId: "first" })])
331+
await writeIncompleteJournal(tmpRoot, taskId, [entry({ checkpointId: "second", path: "src/bar.ts" })])
332+
await writeIncompleteJournal(tmpRoot, taskId, [entry({ checkpointId: "first" })])
333+
334+
const entries = await loadChanges(tmpRoot, taskId)
335+
// First-seen order is what is preserved: the earlier marker's entries keep
336+
// their positions, later batches are appended in their own order, and a
337+
// repeated key is not written twice.
338+
expect(entries.map((e) => e.checkpointId)).toEqual(["first", "second"])
339+
expect(entries.map((e) => e.path)).toEqual(["src/foo.ts", "src/bar.ts"])
340+
// The read still consumes the marker once every entry is durable.
341+
await expect(fs.stat(incompleteJournalPath(tmpRoot, taskId))).rejects.toMatchObject({ code: "ENOENT" })
342+
})
343+
})
344+
189345
describe("task ID containment", () => {
190346
it("rejects task IDs that could escape the tasks directory", async () => {
191347
for (const bad of ["../../outside", "a/b", "a\\b", "", ".", ".."]) {

‎src/core/checkpoints/__tests__/checkpointJournal.test.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,35 @@ describe("checkpoint/journal transaction recovery (B2)", () => {
271271
await expect(fs.stat(pendingJournalPath(tmpStorageDir, TASK_ID))).rejects.toMatchObject({ code: "ENOENT" })
272272
})
273273

274+
it("appends only well-formed entries from a ready marker and still clears it", async () => {
275+
// The incomplete-marker path already filters its entries through isChangeJournalEntry;
276+
// this is the same corruption arriving through the pending marker: an array that parses
277+
// but carries a null and a half-shaped element. Reading entry.path off the null is what
278+
// used to throw here, which left the marker in place and made every later write-bearing
279+
// checkpoint save return without a checkpoint.
280+
const markerPath = pendingJournalPath(tmpStorageDir, TASK_ID)
281+
await fs.mkdir(path.dirname(markerPath), { recursive: true })
282+
await fs.writeFile(
283+
markerPath,
284+
JSON.stringify({
285+
state: "ready",
286+
createdAt: 1,
287+
entries: [
288+
{ path: "src/a.ts", operation: "create", checkpointId: "c1" },
289+
null,
290+
{ path: "src/b.ts", operation: "update" },
291+
],
292+
}),
293+
)
294+
295+
const entries = await loadChanges(tmpStorageDir, TASK_ID)
296+
expect(entries).toEqual([{ path: "src/a.ts", operation: "create", checkpointId: "c1" }])
297+
// The malformed entries are dropped rather than thrown on, for the same reason
298+
// loadChanges drops a corrupt middle line: keeping the marker would make the
299+
// corruption permanent and block every later checkpoint.
300+
await expect(fs.stat(markerPath)).rejects.toMatchObject({ code: "ENOENT" })
301+
})
302+
274303
it("drops an awaiting-checkpoint marker without inventing entries", async () => {
275304
await writePendingJournal(tmpStorageDir, TASK_ID, { state: "awaiting-checkpoint", createdAt: Date.now() })
276305

‎src/core/checkpoints/__tests__/checkpointJournalPromotion.test.ts‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,12 +43,14 @@ type JournalControls = {
4343
incompleteFails: boolean
4444
reconcileCalls: number
4545
reconcileFailAfter: number
46+
awaitingFails: boolean
4647
}
4748
const controls = vi.hoisted<JournalControls>(() => ({
4849
readyFailures: 0,
4950
incompleteFails: false,
5051
reconcileCalls: 0,
5152
reconcileFailAfter: -1,
53+
awaitingFails: false,
5254
}))
5355

5456
// The journal helpers are delegated to the real implementation; only the failure
@@ -58,6 +60,9 @@ vi.mock("../changeJournal", async (importOriginal) => {
5860
return {
5961
...actual,
6062
writePendingJournal: vi.fn(async (dir: string, taskId: string, state: { state: string }) => {
63+
if (state.state === "awaiting-checkpoint" && controls.awaitingFails) {
64+
throw new Error("awaiting marker write failed")
65+
}
6166
if (state.state === "ready" && controls.readyFailures > 0) {
6267
controls.readyFailures--
6368
throw new Error("ready marker write failed")
@@ -100,6 +105,7 @@ describe("checkpointSave journal promotion failure (B2)", () => {
100105
controls.incompleteFails = false
101106
controls.reconcileCalls = 0
102107
controls.reconcileFailAfter = -1
108+
controls.awaitingFails = false
103109
tmpStorageDir = await fs.mkdtemp(path.join(os.tmpdir(), "b2-journal-promotion-"))
104110
saveCheckpointSpy = vi.fn().mockResolvedValue({ commit: COMMIT })
105111
const mockProvider = {
@@ -190,4 +196,49 @@ describe("checkpointSave journal promotion failure (B2)", () => {
190196
expect.anything(),
191197
)
192198
})
199+
200+
it("discards the awaiting marker on the next read when the host exits before promotion", async () => {
201+
// The Git commit landed and then the extension host went away before the ready
202+
// promotion. The on-disk state of that scenario is an awaiting marker, which by
203+
// construction carries no entries, and this test pins what the next read really
204+
// does with it: reconcilePendingJournal accepts the awaiting state, skips the fold
205+
// (that block is gated on "ready"), and clears the marker - so a committed
206+
// checkpoint's writes never reach the journal. index.ts calls this record the
207+
// "recovery record for the next read"; if a later unit makes the pre-commit record
208+
// carry the entries plus a transaction identity, this is the assertion that has to
209+
// change, deliberately rather than by accident.
210+
controls.readyFailures = 99
211+
controls.incompleteFails = true
212+
const consoleErrorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined)
213+
214+
await expect(checkpointSave(mockTask as unknown as Task, false, true, write)).resolves.toBeUndefined()
215+
216+
expect(saveCheckpointSpy).toHaveBeenCalledOnce()
217+
const marker = JSON.parse(await fs.readFile(pendingJournalPath(tmpStorageDir, TASK_ID), "utf8"))
218+
expect(marker.state).toBe("awaiting-checkpoint")
219+
expect(marker.entries).toBeUndefined()
220+
221+
expect(await loadChanges(tmpStorageDir, TASK_ID)).toEqual([])
222+
await expect(fs.stat(pendingJournalPath(tmpStorageDir, TASK_ID))).rejects.toMatchObject({ code: "ENOENT" })
223+
consoleErrorSpy.mockRestore()
224+
})
225+
226+
it("resolves undefined when the pre-commit marker cannot be written, so the save never runs", async () => {
227+
// The pending marker is a hard prerequisite: the save refuses to commit at all
228+
// and resolves undefined. Characterised rather than fixed here, because the
229+
// tool-side catch path reads that undefined as "the flush succeeded" and tells the
230+
// user only its own bookkeeping error; making a journal failure surface as a tool
231+
// failure at the five awaiting call sites is a product decision of its own, not a
232+
// bug fix, so it is recorded as behaviour rather than silently changed.
233+
controls.awaitingFails = true
234+
const consoleErrorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined)
235+
236+
await expect(checkpointSave(mockTask as unknown as Task, false, true, write)).resolves.toBeUndefined()
237+
238+
// No commit was attempted, so there is nothing to journal and no marker left.
239+
expect(saveCheckpointSpy).not.toHaveBeenCalled()
240+
expect(await loadChanges(tmpStorageDir, TASK_ID)).toEqual([])
241+
await expect(fs.stat(pendingJournalPath(tmpStorageDir, TASK_ID))).rejects.toMatchObject({ code: "ENOENT" })
242+
consoleErrorSpy.mockRestore()
243+
})
193244
})

0 commit comments

Comments
 (0)