Skip to content

Commit 4f0abf3

Browse files
anandgupta42claude
andcommitted
fix(drivers): make on-demand driver installs safe across processes
The managed driver directory is shared by every CLI process on the machine, and `installsInFlight` is an in-process Map — it cannot see other processes. Eight CLIs starting together each ran `npm install` over the same tree: npm install failed (exit 217) … ENOTEMPTY … rmdir /root/.local/share/altimate-code/drivers/node_modules/duckdb/… That is not benchmark-specific. Any concurrent use of the CLI hits it, and the result is a driver that appears broken on a machine where nothing is wrong. `withInstallLock` takes a cross-process lock before mutating the directory. `mkdir` is atomic and fails EEXIST when the directory exists on both POSIX and Windows, which makes a lock directory the portable primitive; it lives beside the install directory so npm never sees it as stray package content. Stale locks are broken two ways, because neither alone is sufficient: a dead owner on this host, and age, which is the only signal available for a lock left by another host sharing a home directory. After acquiring, readiness is re-checked. The peer that held the lock has usually just installed the very thing we queued for, so most contenders return "already present" rather than running a second npm over the same tree. On timeout the install proceeds unlocked rather than failing: a racing install is recoverable and the readiness check afterwards is authoritative, whereas refusing to install because a peer is slow turns contention into a hard error. Also adds cwd/execPath and a cwd-prefix note to driver load failures. Two separate investigations have now diagnosed a load failure from the error text alone and got it wrong, because the text named a path without saying what the process's own view of the filesystem was. The exclusion claim is about separate processes, so the test spawns separate processes — an in-process test cannot establish it. The control confirms the test bites: the same four processes without the lock interleave completely (four enters before any exit). 10/10 repeat runs green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
1 parent e56cd6b commit 4f0abf3

2 files changed

Lines changed: 329 additions & 2 deletions

File tree

‎packages/drivers/src/resolve.ts‎

Lines changed: 170 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -460,10 +460,37 @@ function loadFailure(driver: DriverName, where: string, error: unknown): Error {
460460
function ambientLoadFailure(driver: DriverName, error: unknown, detail: string): Error {
461461
return new Error(
462462
`${DRIVER_LABELS[driver]} driver failed to load from the default module resolution: ` +
463-
`${error instanceof Error ? error.message : String(error)}\n${detail}`,
463+
`${error instanceof Error ? error.message : String(error)}\n${detail}${loadDiagnostics(error)}`,
464464
)
465465
}
466466

467+
/**
468+
* Context a reader needs to tell a resolution fault apart from a broken
469+
* package, appended to load failures.
470+
*
471+
* Driver-load failures have twice been diagnosed from the error text alone and
472+
* twice been diagnosed wrong, because the text named a path without saying what
473+
* the process's own view of the filesystem was. The expensive question each
474+
* time was "is this absolute path being re-anchored to the working directory?"
475+
* — which is answerable on the spot, and only from inside the failing process.
476+
*/
477+
function loadDiagnostics(error: unknown): string {
478+
const lines = [`cwd=${process.cwd()}`, `execPath=${process.execPath}`]
479+
const message = error instanceof Error ? error.message : String(error)
480+
for (const match of message.matchAll(/['"`]((?:\/|[A-Za-z]:[\\/])[^'"`\n]+)['"`]/g)) {
481+
const named = match[1]
482+
if (!named) continue
483+
const repaired = repairCwdPrefixedPath(named)
484+
if (repaired) {
485+
lines.push(
486+
`NOTE: "${named}" does not exist, but "${repaired}" does — the working ` +
487+
`directory appears to have been concatenated onto an absolute path.`,
488+
)
489+
}
490+
}
491+
return `\n(${lines.join("; ")})`
492+
}
493+
467494
function describeSearched(searched: readonly string[]): string {
468495
return searched.length
469496
? `It was not found in any searchable location. Searched ${searched.length} ` +
@@ -818,7 +845,16 @@ async function installOptionalDriverInternal(
818845
if (!options.force && installed(driver)) {
819846
return { driver, packages, dir, installed: true, alreadyPresent: true }
820847
}
821-
return performInstall(driver, packages, dir, options)
848+
// Take the cross-process lock, then check readiness again. The peer that
849+
// held it has usually just installed the very thing we queued for, so
850+
// most contenders return "already present" instead of running a second
851+
// npm over the same tree — which is what produced the ENOTEMPTY races.
852+
return withInstallLock(dir, async (acquired) => {
853+
if (acquired && !options.force && installed(driver)) {
854+
return { driver, packages, dir, installed: true, alreadyPresent: true }
855+
}
856+
return performInstall(driver, packages, dir, options)
857+
})
822858
})
823859
installsInFlight.set(dir, run)
824860
try {
@@ -831,6 +867,138 @@ async function installOptionalDriverInternal(
831867
/** In-flight installs keyed by target directory (see the note above). */
832868
const installsInFlight = new Map<string, Promise<InstallResult>>()
833869

870+
// ---------------------------------------------------------------------------
871+
// Cross-process install lock
872+
// ---------------------------------------------------------------------------
873+
874+
/**
875+
* `installsInFlight` serialises installs inside one process. It cannot see
876+
* other processes, and the managed driver directory is shared by all of them,
877+
* so N CLIs starting together each run `npm install` over the same tree:
878+
*
879+
* npm install failed (exit 217) … ENOTEMPTY …
880+
* rmdir /root/.local/share/altimate-code/drivers/node_modules/duckdb/…
881+
*
882+
* That is not benchmark-specific — any concurrent use of the CLI hits it.
883+
*
884+
* `mkdir` is atomic and fails with EEXIST when the directory exists, on both
885+
* POSIX and Windows, which makes a lock directory the portable primitive here.
886+
* The lock lives beside the install directory rather than inside it so npm
887+
* never sees it as stray package content.
888+
*/
889+
function installLockPath(dir: string): string {
890+
return `${dir.replace(/[\\/]+$/, "")}.lock`
891+
}
892+
893+
interface LockHolder {
894+
pid: number
895+
hostname: string
896+
/** Absent when the owner file is malformed; age then falls back to mtime. */
897+
startedAt?: number
898+
}
899+
900+
function readLockHolder(lockDir: string): LockHolder | undefined {
901+
try {
902+
const raw = fs.readFileSync(path.join(lockDir, "owner.json"), "utf8")
903+
const parsed: unknown = JSON.parse(raw)
904+
if (!parsed || typeof parsed !== "object") return undefined
905+
const holder = parsed as Partial<LockHolder>
906+
if (typeof holder.pid !== "number") return undefined
907+
return {
908+
pid: holder.pid,
909+
hostname: typeof holder.hostname === "string" ? holder.hostname : "",
910+
startedAt: typeof holder.startedAt === "number" ? holder.startedAt : undefined,
911+
}
912+
} catch {
913+
return undefined
914+
}
915+
}
916+
917+
/**
918+
* True when a lock cannot belong to a live install any more: its owner is gone,
919+
* or it has outlived any plausible npm run. Both checks are needed — a killed
920+
* process leaves no signal beyond its absence, and a lock from another host
921+
* (shared home directory) can only be judged by age.
922+
*/
923+
function isStaleLock(lockDir: string, holder: LockHolder | undefined, maxAgeMs: number): boolean {
924+
if (holder && holder.hostname === os.hostname() && !processExists(holder.pid)) return true
925+
const startedAt = holder?.startedAt
926+
if (typeof startedAt === "number" && Date.now() - startedAt > maxAgeMs) return true
927+
try {
928+
return Date.now() - fs.statSync(lockDir).mtimeMs > maxAgeMs
929+
} catch {
930+
// Vanished between checks — someone else released it, so it is not stale.
931+
return false
932+
}
933+
}
934+
935+
/**
936+
* Run `fn` while holding an exclusive lock on `dir`, across processes.
937+
*
938+
* On timeout the work runs anyway rather than failing. A driver install that
939+
* races is recoverable — npm is largely idempotent here and the readiness check
940+
* afterwards is authoritative — whereas refusing to install because a lock
941+
* could not be taken turns a slow peer into a hard failure.
942+
*/
943+
export async function withInstallLock<T>(
944+
dir: string,
945+
fn: (acquired: boolean) => Promise<T>,
946+
options: { timeoutMs?: number; staleAfterMs?: number; pollMs?: number } = {},
947+
): Promise<T> {
948+
const lockDir = installLockPath(dir)
949+
const timeoutMs = options.timeoutMs ?? 240_000
950+
const staleAfterMs = options.staleAfterMs ?? 300_000
951+
const pollMs = options.pollMs ?? 100
952+
const deadline = Date.now() + timeoutMs
953+
let acquired = false
954+
955+
for (;;) {
956+
try {
957+
fs.mkdirSync(lockDir, { recursive: false })
958+
acquired = true
959+
break
960+
} catch (e) {
961+
// Anything but "already held" — an unwritable parent, say — means we
962+
// cannot lock at all, so proceed unlocked rather than block forever.
963+
const code = e && typeof e === "object" && "code" in e ? (e as { code?: unknown }).code : undefined
964+
if (code !== "EEXIST") break
965+
if (isStaleLock(lockDir, readLockHolder(lockDir), staleAfterMs)) {
966+
try {
967+
fs.rmSync(lockDir, { recursive: true, force: true })
968+
} catch {
969+
// Another process won the cleanup; fall through and retry.
970+
}
971+
continue
972+
}
973+
if (Date.now() >= deadline) break
974+
await sleep(pollMs)
975+
}
976+
}
977+
978+
if (acquired) {
979+
try {
980+
fs.writeFileSync(
981+
path.join(lockDir, "owner.json"),
982+
JSON.stringify({ pid: process.pid, hostname: os.hostname(), startedAt: Date.now() } satisfies LockHolder),
983+
)
984+
} catch {
985+
// Diagnostics only — the lock is the directory, not the file in it.
986+
}
987+
}
988+
989+
try {
990+
return await fn(acquired)
991+
} finally {
992+
if (acquired) {
993+
try {
994+
fs.rmSync(lockDir, { recursive: true, force: true })
995+
} catch {
996+
// Leaving it behind is safe: the next contender ages it out as stale.
997+
}
998+
}
999+
}
1000+
}
1001+
8341002
async function performInstall(
8351003
driver: DriverName,
8361004
packages: readonly string[],
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
2+
import fs from "node:fs"
3+
import os from "node:os"
4+
import path from "node:path"
5+
import { fileURLToPath } from "node:url"
6+
7+
import { withInstallLock } from "../src/resolve"
8+
9+
// The managed driver directory is shared by every CLI process on the machine,
10+
// and `installsInFlight` only serialises within one process. Eight CLIs
11+
// starting together each ran `npm install` over the same tree:
12+
//
13+
// npm install failed (exit 217) … ENOTEMPTY …
14+
// rmdir /root/.local/share/altimate-code/drivers/node_modules/duckdb/…
15+
//
16+
// The exclusion claim is about separate processes, so the central test spawns
17+
// separate processes. An in-process test cannot establish it.
18+
19+
const resolveModule = fileURLToPath(new URL("../src/resolve.ts", import.meta.url))
20+
21+
let dir = ""
22+
23+
beforeEach(() => {
24+
dir = fs.mkdtempSync(path.join(os.tmpdir(), "install-lock-"))
25+
})
26+
27+
afterEach(() => {
28+
if (dir) fs.rmSync(dir, { recursive: true, force: true })
29+
fs.rmSync(`${dir}.lock`, { recursive: true, force: true })
30+
})
31+
32+
describe("cross-process install lock", () => {
33+
test("excludes concurrent processes from the critical section", async () => {
34+
const target = path.join(dir, "drivers")
35+
fs.mkdirSync(target, { recursive: true })
36+
const log = path.join(dir, "log.txt")
37+
38+
// Each child holds the lock briefly and brackets its critical section.
39+
// Overlapping brackets in the log mean the lock did not hold.
40+
const child = path.join(dir, "child.ts")
41+
fs.writeFileSync(
42+
child,
43+
`import fs from "node:fs"
44+
import { withInstallLock } from ${JSON.stringify(resolveModule)}
45+
const [target, log, id] = process.argv.slice(2)
46+
await withInstallLock(target, async (acquired) => {
47+
if (!acquired) { fs.appendFileSync(log, \`timeout \${id}\\n\`); return }
48+
fs.appendFileSync(log, \`enter \${id}\\n\`)
49+
await new Promise((r) => setTimeout(r, 120))
50+
fs.appendFileSync(log, \`exit \${id}\\n\`)
51+
}, { timeoutMs: 30000 })
52+
process.exit(0)
53+
`,
54+
)
55+
56+
const kids = Array.from({ length: 4 }, (_, i) =>
57+
Bun.spawn(["bun", child, target, log, String(i)], { stdout: "ignore", stderr: "ignore" }),
58+
)
59+
const codes = await Promise.all(kids.map((k) => k.exited))
60+
expect(codes).toEqual([0, 0, 0, 0])
61+
62+
const events = fs.readFileSync(log, "utf8").trim().split("\n").filter(Boolean)
63+
expect(events).not.toContain("timeout 0")
64+
// Every enter must be followed by its own exit before the next enter.
65+
let inside: string | undefined
66+
for (const line of events) {
67+
const [kind, id] = line.split(" ")
68+
if (kind === "enter") {
69+
expect(inside).toBeUndefined()
70+
inside = id
71+
} else if (kind === "exit") {
72+
expect(inside).toBe(id)
73+
inside = undefined
74+
}
75+
}
76+
expect(events.filter((e) => e.startsWith("enter")).length).toBe(4)
77+
}, 60_000)
78+
79+
test("reports the section ran unlocked when the lock cannot be taken in time", async () => {
80+
const target = path.join(dir, "drivers")
81+
fs.mkdirSync(target, { recursive: true })
82+
// Hold the lock with a live owner so it cannot be judged stale.
83+
fs.mkdirSync(`${target}.lock`)
84+
fs.writeFileSync(
85+
path.join(`${target}.lock`, "owner.json"),
86+
JSON.stringify({ pid: process.pid, hostname: os.hostname(), startedAt: Date.now() }),
87+
)
88+
89+
let sawAcquired: boolean | undefined
90+
await withInstallLock(
91+
target,
92+
async (acquired) => {
93+
sawAcquired = acquired
94+
},
95+
{ timeoutMs: 200, pollMs: 20 },
96+
)
97+
// The work still runs — refusing to install because a peer is slow would
98+
// turn contention into a hard failure — but it knows it was unlocked.
99+
expect(sawAcquired).toBe(false)
100+
// A lock we did not take must not be deleted on the way out.
101+
expect(fs.existsSync(`${target}.lock`)).toBe(true)
102+
})
103+
104+
test("breaks a lock whose owner is gone", async () => {
105+
const target = path.join(dir, "drivers")
106+
fs.mkdirSync(target, { recursive: true })
107+
fs.mkdirSync(`${target}.lock`)
108+
fs.writeFileSync(
109+
path.join(`${target}.lock`, "owner.json"),
110+
// PID 0x7FFFFFFF is not a live process on any platform we run on.
111+
JSON.stringify({ pid: 0x7fffffff, hostname: os.hostname(), startedAt: Date.now() }),
112+
)
113+
114+
let sawAcquired: boolean | undefined
115+
await withInstallLock(
116+
target,
117+
async (acquired) => {
118+
sawAcquired = acquired
119+
},
120+
{ timeoutMs: 5000, pollMs: 20 },
121+
)
122+
expect(sawAcquired).toBe(true)
123+
})
124+
125+
test("breaks a lock that has outlived any plausible install", async () => {
126+
const target = path.join(dir, "drivers")
127+
fs.mkdirSync(target, { recursive: true })
128+
fs.mkdirSync(`${target}.lock`)
129+
fs.writeFileSync(
130+
path.join(`${target}.lock`, "owner.json"),
131+
JSON.stringify({ pid: process.pid, hostname: os.hostname(), startedAt: Date.now() - 10 * 60_000 }),
132+
)
133+
134+
let sawAcquired: boolean | undefined
135+
await withInstallLock(
136+
target,
137+
async (acquired) => {
138+
sawAcquired = acquired
139+
},
140+
{ timeoutMs: 5000, staleAfterMs: 60_000, pollMs: 20 },
141+
)
142+
expect(sawAcquired).toBe(true)
143+
})
144+
145+
test("releases the lock when the critical section throws", async () => {
146+
const target = path.join(dir, "drivers")
147+
fs.mkdirSync(target, { recursive: true })
148+
let thrown = ""
149+
try {
150+
await withInstallLock(target, async () => {
151+
throw new Error("boom")
152+
})
153+
} catch (e) {
154+
thrown = e instanceof Error ? e.message : String(e)
155+
}
156+
expect(thrown).toBe("boom")
157+
expect(fs.existsSync(`${target}.lock`)).toBe(false)
158+
})
159+
})

0 commit comments

Comments
 (0)