Skip to content

Commit 6378aa8

Browse files
S4b (Zoo-Code-Org#1375): create the directory the lock needs before taking the lock
proper-lockfile creates the lock artefact with a non-recursive mkdir. A guarded write into a directory that did not exist yet therefore failed inside acquireFileLock with ENOENT - before the publish primitive ran, and so before the rollback the publish owns could do anything about it - and left the empty tree behind for DiffViewProvider.ts:1843-:1845 and safeWriteText.ts:437-:443 to defend against. The directory the artefact needs is now created first, and the artefact itself is still derived only from the guarded path. The empirical answer on the alternative, measured on a real temporary tree rather than argued: writer A took <existing-dir>.lock, created the missing parent under it, and writer B - arriving after the parent existed, for the file A was about to write - acquired <existing-dir>/nested/state.json.lock while A still held its lock. Those are different locations, so B did not contend at all. Widening the lock to an ancestor means the artefact depends on what happens to exist at the moment each writer derives it, which is the divergence class the repair exists to close. With the shipped derivation the same late writer is refused: 'Lock file is already being held'. That is why the mkdir moved before the lock rather than the lock moving to the ancestor. 'Only the directories this call created' is answered by asking the filesystem to create and reading the answer, not by statting: recursive mkdir resolves with the first directory it actually made, and with nothing when every level already existed. Measured on this host, that answer arrives with an extended-length prefix (\\?\C:\...); comparing it with the chain unnormalised made a rollback climb past the tree it had built and remove directories that were already there. It is normalised before it is compared, the rollback runs deepest- first and swallows its own failures, and the acquisition error reaches the caller unchanged. The first version of this walked the chain with fs.stat, and that was wrong for a reason I only found by running the neighbours: it reddened two safeWriteJson rows, including safeWriteJson.stagingDirectory.spec.ts's 'stages anyway under the task persistence suite's blanket missing stat', whose whole point is that a write must not start asking fs.stat questions about the directory it is allowed to write into - the task-persistence suites mock fs.stat to reject for every path. The create-shape asks no such question and both rows are green again. There is deliberately no fs.stat call in the chain builder. Rows: fileLock.test.ts 2 (the parent exists before the lock is taken and the artefact stayed at lockPathFor(path); a failed acquisition gives back only what it created, and a directory that already existed survives it). fileLockContention.test.ts 2, against the real proper-lockfile on real directories (a writer arriving after the directory exists contends with the writer that created it; two writers racing into the same missing directory leave exactly one holding the path, the loser refused with 'already being held', and no lock artefact leaks). fileLockChain.test.ts 2, where the doubles hold Node's recursive-mkdir contract and the assertions read the calls: nothing is claimed when nothing was created, and only the reported levels are given back, deepest first. Controls, each failing only for the reason its row describes: skipping the create fails the two creation rows and both contention rows ('ENOENT ... mkdir <file>.lock', 'expected [] to have a length of 1'); claiming something even when the create reported nothing fails the nothing-claimed row alone ('expected vi.fn() to not be called at all, but actually been called 1 times'); claiming one level above what was created fails both only-what-it-created rows ('ENOENT ... stat <existing>/one'); keeping what was created fails the two rollback rows; moving the artefact to the parent directory fails both contention rows and the artefact-location assertion, and also the pre-existing drift-guard row, which is right to fail when the artefact stops being lockPathFor(path); comparing the create's answer without normalising it fails the claim row alone. The control that reports the rollback instead of the acquisition failure reddens six rows rather than one, because it changes the identity of every acquisition failure - the two rollback rows and the claim rows are the ones that name it, and one pre-existing safeWriteJson row reports it too; stated rather than hidden. Lanes: fileLock 4, contention 2, chain 2, safeWriteJson + stagingDirectory + lockKey 57 passed | 4 skipped, and the wide run over every utils/ and services/ lane is 190 files passed | 11 failed - the same eleven that fail at HEAD fileLock.ts on this host (5 Windows symlink EPERM rows in rules, 4 'TelemetryService not initialized' rows in MarketplaceManager, 2 rows in code-index parser), so nothing moved in either direction. DiffViewProvider 168 and guardedWrite 73 green. eslint --max-warnings=0 exit 0 on all six touched files; --prune-suppressions run afterwards and src/eslint-suppressions.json restored to HEAD and verified by an empty diff (a narrow prune rewrites 1326 of 1326 lines and drops records for files outside the selection, so it is a check and never a commit). prettier --check clean. tsc --noEmit -p . shows only activate/registerCommands.ts(88,102) TS2739, the local junction artifact. No it.runIf(win32) anywhere in the change: the rows run on every platform, and the extended-length prefix is handled by normalising what Windows reports rather than by gating a test. Review row 4507646158.
1 parent dec06df commit 6378aa8

4 files changed

Lines changed: 245 additions & 2 deletions

File tree

‎src/utils/__tests__/fileLock.test.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
import * as fsp from "fs/promises"
2+
import * as os from "os"
3+
import * as path from "path"
14
import * as lockfile from "proper-lockfile"
25
import { acquireFileLock, LOCK_STALE_MS, lockPathFor, withFileLock } from "../fileLock"
36

@@ -35,4 +38,55 @@ describe("file lock protocol", () => {
3538
).rejects.toBe(error)
3639
expect(release).toHaveBeenCalledOnce()
3740
})
41+
42+
// The lock artefact sits beside the file it guards, and proper-lockfile's own mkdir of
43+
// that artefact is not recursive. A write into a directory that did not exist yet failed
44+
// here with ENOENT - before the publish primitive, and so before any rollback it would
45+
// have owned - and left the empty tree behind. These rows run against the real filesystem
46+
// for the directory behaviour; the lock itself stays stubbed because the question here is
47+
// which directories exist afterwards, not who holds what.
48+
it("creates the directory the lock artefact needs before taking the lock", async () => {
49+
const base = await fsp.mkdtemp(path.join(os.tmpdir(), "filelock-chain-"))
50+
const target = path.join(base, "missing", "nested", "history.json")
51+
const release = vi.fn(async () => {})
52+
vi.mocked(lockfile.lock).mockResolvedValue(release)
53+
try {
54+
await acquireFileLock(target)
55+
expect((await fsp.stat(path.join(base, "missing", "nested"))).isDirectory()).toBe(true)
56+
// The artefact did not move to the ancestor that happened to exist: every writer to
57+
// this path, arriving before or after the directory, is locking the same location.
58+
expect(lockfile.lock).toHaveBeenCalledWith(
59+
expect.any(String),
60+
expect.objectContaining({ lockfilePath: lockPathFor(target) }),
61+
)
62+
await release()
63+
// Releasing does not undo the creation: the write the lock was taken for follows it.
64+
expect((await fsp.stat(path.join(base, "missing", "nested"))).isDirectory()).toBe(true)
65+
} finally {
66+
await fsp.rm(base, { recursive: true, force: true })
67+
}
68+
})
69+
70+
it("removes only the directories it created when the lock cannot be acquired", async () => {
71+
const base = await fsp.mkdtemp(path.join(os.tmpdir(), "filelock-rollback-"))
72+
const existing = path.join(base, "existing")
73+
await fsp.mkdir(existing)
74+
// A directory somebody else already made is not this call's to remove, even though it
75+
// sits inside the chain this call was walking.
76+
const madeByAnotherWriter = path.join(existing, "one")
77+
await fsp.mkdir(madeByAnotherWriter)
78+
const target = path.join(madeByAnotherWriter, "two", "history.json")
79+
const lockFailure = Object.assign(new Error("EEXIST: file already exists"), { code: "EEXIST" })
80+
vi.mocked(lockfile.lock).mockRejectedValue(lockFailure)
81+
try {
82+
// A rollback that reported its own failure instead would hide which of the two
83+
// failures the caller has to act on.
84+
await expect(acquireFileLock(target)).rejects.toBe(lockFailure)
85+
expect((await fsp.stat(existing)).isDirectory()).toBe(true)
86+
expect((await fsp.stat(madeByAnotherWriter)).isDirectory()).toBe(true)
87+
await expect(fsp.stat(path.join(madeByAnotherWriter, "two"))).rejects.toMatchObject({ code: "ENOENT" })
88+
} finally {
89+
await fsp.rm(base, { recursive: true, force: true })
90+
}
91+
})
3892
})
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import * as fsp from "fs/promises"
2+
import * as path from "path"
3+
import * as lockfile from "proper-lockfile"
4+
import { acquireFileLock } from "../fileLock"
5+
6+
// Only the two filesystem calls this module makes are doubled, and the lock itself: an
7+
// unexpected fs.stat call would fail the row instead of quietly resolving, which matters
8+
// because the directory question is the one the task-persistence suites forbid asking.
9+
vi.mock("fs/promises", () => ({ mkdir: vi.fn(), rmdir: vi.fn() }))
10+
vi.mock("proper-lockfile", () => ({ lock: vi.fn() }))
11+
12+
// The doubles hold Node's recursive-mkdir contract, measured on this host rather than
13+
// inferred from the docs: the call resolves with the first directory it actually created -
14+
// on Windows reported with an extended-length \\?\ prefix - and with nothing when every
15+
// level already existed. That answer is the only thing this module uses to decide what a
16+
// failed acquisition may give back, so these rows keep the answer fixed and read the calls
17+
// the module made, instead of asking a real filesystem to lose a race on schedule.
18+
describe("what a failed lock acquisition gives back", () => {
19+
const lockRefused = (): Error =>
20+
Object.assign(new Error("EEXIST: file already exists, lock held"), { code: "EEXIST" })
21+
22+
beforeEach(() => vi.clearAllMocks())
23+
24+
it("claims nothing when every level of the chain already existed", async () => {
25+
const refusal = lockRefused()
26+
vi.mocked(fsp.mkdir).mockResolvedValue(undefined)
27+
vi.mocked(lockfile.lock).mockRejectedValue(refusal)
28+
await expect(acquireFileLock(path.join("C:", "tree", "state.json"))).rejects.toBe(refusal)
29+
// Nothing was created, so nothing may be removed: the directories here belong to
30+
// whoever put them there, and one of them may be somebody else's working tree.
31+
expect(fsp.rmdir).not.toHaveBeenCalled()
32+
})
33+
34+
it("gives back only the levels the create reported, deepest first", async () => {
35+
const refusal = lockRefused()
36+
const firstLevel = path.join("C:", "tree", "one")
37+
// Node's answer on Windows arrives prefixed; the module normalises it before comparing.
38+
vi.mocked(fsp.mkdir).mockResolvedValue("\\\\?\\" + firstLevel)
39+
vi.mocked(lockfile.lock).mockRejectedValue(refusal)
40+
const target = path.join(firstLevel, "two", "state.json")
41+
await expect(acquireFileLock(target)).rejects.toBe(refusal)
42+
expect(vi.mocked(fsp.mkdir).mock.calls.map((call) => String(call[0]))).toEqual([path.join(firstLevel, "two")])
43+
expect(vi.mocked(fsp.rmdir).mock.calls.map((call) => String(call[0]))).toEqual([
44+
path.join(firstLevel, "two"),
45+
firstLevel,
46+
])
47+
})
48+
})
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
import * as fsp from "fs/promises"
2+
import * as os from "os"
3+
import * as path from "path"
4+
import { acquireFileLock, lockPathFor } from "../fileLock"
5+
6+
// These rows run against the real proper-lockfile on real temporary directories. The
7+
// property is which artefact two writers reach, and a stubbed lock cannot answer it:
8+
// a stub reports whatever shape the test hands it, including a shape production does
9+
// not build.
10+
describe("file lock contention where the directory has to be created first", () => {
11+
it("a writer that arrives after the directory exists contends with the writer that made it", async () => {
12+
const base = await fsp.mkdtemp(path.join(os.tmpdir(), "filelock-contend-"))
13+
const target = path.join(base, "missing", "nested", "state.json")
14+
try {
15+
const releaseFirst = await acquireFileLock(target)
16+
expect((await fsp.stat(path.dirname(target))).isDirectory()).toBe(true)
17+
// The second writer arrives while the first still holds and after the directory
18+
// appeared. Under ancestor-first locking it would have derived a different
19+
// artefact and walked straight in; the artefact is derived from the guarded path,
20+
// so it contends.
21+
// proper-lockfile reports a held artefact as its own error rather than the raw
22+
// mkdir collision, so the row names what the caller actually sees.
23+
await expect(acquireFileLock(target, { immediate: true })).rejects.toThrow(/already being held/)
24+
await releaseFirst()
25+
const releaseSecond = await acquireFileLock(target, { immediate: true })
26+
expect((await fsp.stat(lockPathFor(target))).isDirectory()).toBe(true)
27+
await releaseSecond()
28+
await expect(fsp.stat(lockPathFor(target))).rejects.toMatchObject({ code: "ENOENT" })
29+
} finally {
30+
await fsp.rm(base, { recursive: true, force: true })
31+
}
32+
})
33+
34+
it("two writers racing into the same missing directory leave exactly one holding the path", async () => {
35+
const base = await fsp.mkdtemp(path.join(os.tmpdir(), "filelock-race-"))
36+
const target = path.join(base, "missing", "nested", "state.json")
37+
const attempts = await Promise.allSettled([
38+
acquireFileLock(target, { immediate: true }),
39+
acquireFileLock(target, { immediate: true }),
40+
])
41+
const holding = attempts.filter((attempt) => attempt.status === "fulfilled")
42+
const refused = attempts.filter((attempt) => attempt.status === "rejected")
43+
try {
44+
expect(holding).toHaveLength(1)
45+
expect(refused).toHaveLength(1)
46+
expect((refused[0] as PromiseRejectedResult).reason).toBeInstanceOf(Error)
47+
expect(String((refused[0] as PromiseRejectedResult).reason)).toMatch(/already being held/)
48+
// One artefact, held by one writer: the refused attempt creates no lock of its own
49+
// and leaks nothing behind.
50+
expect((await fsp.stat(lockPathFor(target))).isDirectory()).toBe(true)
51+
await holding[0]!.value()
52+
await expect(fsp.stat(lockPathFor(target))).rejects.toMatchObject({ code: "ENOENT" })
53+
} finally {
54+
await fsp.rm(base, { recursive: true, force: true })
55+
}
56+
})
57+
})

‎src/utils/fileLock.ts‎

Lines changed: 86 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import * as fs from "fs/promises"
12
import * as path from "path"
23
import { tracePersistenceEvent } from "../core/task-persistence/persistenceTrace"
34
import * as lockfile from "proper-lockfile"
@@ -24,6 +25,74 @@ export function lockPathFor(filePath: string): string {
2425
}
2526
const RECOVERY_GRACE_MS = 4000
2627

28+
/**
29+
* Create the directory that holds a path's lock artefact, and report only the levels this
30+
* call actually created. proper-lockfile's own mkdir of the artefact is not recursive, so a
31+
* write into a directory that did not exist yet failed here with ENOENT - before the publish
32+
* primitive, and so before any rollback it would have owned - and left the empty tree behind.
33+
*
34+
* This asks the filesystem to create, and lets the answer say what was made: recursive mkdir
35+
* resolves with the first directory it actually created, and with nothing when every level
36+
* already existed. There is deliberately no fs.stat here. The task-persistence suites mock
37+
* fs.stat to reject for every path, and safeWriteJson.stagingDirectory.spec.ts pins that the
38+
* write must not start asking fs.stat questions about the directory it is allowed to write
39+
* into; a stat-driven walk would reintroduce exactly that question, and did.
40+
*/
41+
async function createLockDirectoryChain(lockfilePath: string): Promise<string[]> {
42+
const parent = path.dirname(lockfilePath)
43+
// Measured on this host, not inferred from the docs: Node reports the first directory it
44+
// created with an extended-length prefix (\\?\C:\...), so the answer is normalised before
45+
// it is compared with anything. Comparing it raw is what made a rollback reach above the
46+
// tree this call had built.
47+
const firstMade = await fs.mkdir(parent, { recursive: true })
48+
if (firstMade === undefined || firstMade === null) {
49+
return []
50+
}
51+
const made = path.resolve(withoutExtendedLengthPrefix(String(firstMade)))
52+
const tail = path.relative(made, parent)
53+
if (tail.startsWith("..")) {
54+
// The answer is not a location this call can account for; claim the parent and no
55+
// more, which is the conservative reading of "only what we created".
56+
return [parent]
57+
}
58+
const chain: string[] = [made]
59+
let cursor = made
60+
if (tail !== "") {
61+
for (const segment of tail.split(path.sep)) {
62+
cursor = path.join(cursor, segment)
63+
chain.push(cursor)
64+
}
65+
}
66+
return chain
67+
}
68+
69+
/** Strip the extended-length prefix Node reports on Windows, leaving other paths alone. */
70+
function withoutExtendedLengthPrefix(candidate: string): string {
71+
if (candidate.startsWith("\\\\?\\UNC\\")) {
72+
return "\\\\" + candidate.slice(8)
73+
}
74+
if (candidate.startsWith("\\\\?\\")) {
75+
return candidate.slice(4)
76+
}
77+
return candidate
78+
}
79+
/**
80+
* Undo the directories a failed acquisition created, deepest first and best-effort. A
81+
* directory somebody else populated is not this call's to take away, and a rollback
82+
* failure must not arrive dressed as the error that caused the rollback.
83+
*/
84+
async function removeCreatedDirectories(created: string[]): Promise<void> {
85+
for (let index = created.length - 1; index >= 0; index--) {
86+
try {
87+
await fs.rmdir(created[index])
88+
} catch (rollbackError: unknown) {
89+
// Includes ENOTEMPTY: the directory is in use, which is a reason to leave it,
90+
// not a reason to report a second failure over the first.
91+
void rollbackError
92+
}
93+
}
94+
}
95+
2796
export interface FileLockOptions {
2897
/** Recovery may wait out a crashed writer's staleness window without stealing a live lock. */
2998
waitForStaleLock?: boolean
@@ -41,11 +110,23 @@ export interface FileLockOptions {
41110
export async function acquireFileLock(filePath: string, options?: FileLockOptions): Promise<() => Promise<void>> {
42111
const absoluteFilePath = path.resolve(filePath)
43112
if (options?.waitForStaleLock) tracePersistenceEvent("json-lock-wait", { filePath: absoluteFilePath })
113+
const lockfilePath = lockPathFor(absoluteFilePath)
114+
// Created before the lock is taken, and removed again only if the acquisition
115+
// fails: see createLockDirectoryChain. A failure here is rolled back before it is
116+
// reported, so a refused write does not litter the tree it refused to enter.
117+
let created: string[] = []
118+
try {
119+
created = await createLockDirectoryChain(lockfilePath)
120+
} catch (mkdirError: unknown) {
121+
await removeCreatedDirectories(created)
122+
console.error(`Failed to prepare the lock directory for ${absoluteFilePath}:`, mkdirError)
123+
throw mkdirError
124+
}
44125
try {
45126
return await lockfile.lock(absoluteFilePath, {
46127
stale: LOCK_STALE_MS,
47128
update: 10000, // Update mtime every 10 seconds to prevent staleness if operation is long
48-
lockfilePath: lockPathFor(absoluteFilePath),
129+
lockfilePath,
49130
realpath: false, // the file may not exist yet, which is acceptable
50131
retries: options?.immediate
51132
? 0
@@ -68,7 +149,10 @@ export async function acquireFileLock(filePath: string, options?: FileLockOption
68149
throw err
69150
},
70151
})
71-
} catch (lockError) {
152+
} catch (lockError: unknown) {
153+
// The parent directories this call made are the only state it
154+
// owns at this point; give them back, then report the acquisition failure unchanged.
155+
await removeCreatedDirectories(created)
72156
console.error(`Failed to acquire lock for ${absoluteFilePath}:`, lockError)
73157
throw lockError
74158
}

0 commit comments

Comments
 (0)