Skip to content

Commit 2e50c99

Browse files
committed
fix(drivers): address PR review threads on the install lock and resolver
Resolves 17 of the 28 open review threads on this PR (chatgpt-codex-connector, cubic-dev-ai, kilo-code-bot, coderabbitai), leaves 7 open with a design question or a duplicate pointer, and folds in #1205's empty-searched-roots error-message fix (that PR was closed as redundant with this one). Fixed: - `isWorkspaceRoot()` compared a harvested root against ancestor `node_modules` by exact equality, so a *nested* dependency root under an ancestor (`.../node_modules/host/node_modules/duckdb`) slipped past the exclusion when the CLI started below the project root — a real workspace-boundary bypass (P1). Now uses containment via a new `isWithin` helper. - `isRequireOfEsm()` did not recognize Node's `ERR_REQUIRE_ASYNC_MODULE` or Bun's equivalent message, so an ESM-only driver with top-level `await` was reported as broken instead of falling through to dynamic import. Verified the exact Node 22 error shape empirically. - `LockHolder` now carries an optional `budgetMs`, published from the holder's own effective `hardStaleAfterMs` (itself now derived from the configured install `timeoutMs`, not a fixed constant). `isStaleLock` prefers a live holder's declared budget over a contender's own generic backstop, so a default-configured contender no longer reclaims a lock whose owner explicitly configured a longer install timeout. - `isFilesystemRoot()` now recognizes extended-length UNC (`\\?\UNC\server\share`) and forward-slash UNC (`//server/share`) roots, alongside the plain backslash UNC form already handled. - `DriverNotInstalledError` named no location when the searched-roots list was empty (bare "Searched 0 locations:"); now names the expected managed `node_modules` directory. This is #1205's fix, folded in here since that PR was closed as redundant. - Two detached JSDoc blocks reattached to the functions they document. - Test-quality fixes: `resolve-chdir.test.ts` beforeEach now restores cwd so tests don't depend on execution order; the cwd-prefix-repair test now actually constructs a concatenated path instead of one already resolvable without the repair; `install-lock.test.ts`'s multi-peer test gained a start barrier so it can't pass vacuously; `resolve-argv-isolation.test.ts` tests marked `test.serial` against a future concurrent test run. Left open (design questions, replied in-thread rather than resolved unilaterally): - Stale-lock reclamation and release are built on comparing owner records across a directory rename, which several reviewers (from different angles — ownerless-lock races, inode reuse, TOCTOU on stale-claim restore, non-atomic release) converge on as needing a different primitive: an `O_EXCL` sentinel carrying an acquisition token, not the current rename-and-compare protocol. Four of these were already triaged and left open in-thread before this pass; seven more converging on the same conclusion are now replied to and left open as duplicates or extensions of that same finding. - A contender's wait deadline is still derived only from its own `timeoutMs`, not a holder's declared budget — attempted an unconditional extension and reverted it after it blocked a short-timeout caller for a holder's entire remaining budget (over two hours in testing). Left open with two bounded-design options rather than picking one. - Canonicalizing `ALTIMATE_DRIVER_DIR` before deriving the lock path (so a symlink and its real path share a lock) conflicts with the explicitly handled cold-start case where the directory does not exist yet. Left open with two options. Gates: `bun test packages/drivers` 289/289 green (stable across repeated runs), `bun run typecheck` clean across all 15 typechecked packages, upstream marker check clean (drivers is not upstream-shared).
1 parent 730f798 commit 2e50c99

6 files changed

Lines changed: 319 additions & 54 deletions

File tree

‎packages/drivers/src/resolve.ts‎

Lines changed: 120 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -91,11 +91,20 @@ export class DriverNotInstalledError extends Error {
9191

9292
constructor(driver: DriverName, packages: readonly string[], searched: readonly string[]) {
9393
const label = DRIVER_LABELS[driver]
94+
const installDir = driverInstallDir()
95+
// `driverSearchRoots()` only returns directories that exist, so a compiled
96+
// first run — before anything has ever been installed — can have no
97+
// searchable roots at all. The old message then ended in a bare "Searched 0
98+
// locations:" with nothing after the colon. Describe only what the empty
99+
// list actually proves, and name the managed location users need.
100+
const searchedLine = searched.length
101+
? `Searched ${searched.length} location${searched.length === 1 ? "" : "s"}: ${searched.join(", ")}`
102+
: `No searchable driver locations were found. Expected managed location: ${path.join(installDir, "node_modules")}.`
94103
super(
95104
`${label} driver not installed.\n` +
96105
`Install it with the warehouse_install_driver tool, or run:\n` +
97-
` npm install --prefix ${shellQuote(driverInstallDir())} ${packages.join(" ")}\n` +
98-
`Searched ${searched.length} location${searched.length === 1 ? "" : "s"}: ${searched.join(", ")}`,
106+
` npm install --prefix ${shellQuote(installDir)} ${packages.join(" ")}\n` +
107+
searchedLine,
99108
)
100109
this.name = "DriverNotInstalledError"
101110
this.driver = driver
@@ -268,37 +277,33 @@ export function enclosingNodeModulesRoots(candidate: string, sep: string = path.
268277
return roots
269278
}
270279

280+
/** True when `candidate` is `base` itself, or nested anywhere inside it. */
281+
function isWithin(base: string, candidate: string): boolean {
282+
const rel = path.relative(base, candidate)
283+
return rel === "" || (!rel.startsWith("..") && !path.isAbsolute(rel))
284+
}
285+
271286
/** True when `root` is workspace-controlled and must not be imported from. */
272287
function isWorkspaceRoot(root: string, scope: { cwd: string | undefined; ancestors: string[] }): boolean {
273288
// Real paths on both sides: a symlink pointing into the workspace must not
274289
// slip past a purely lexical comparison.
275290
const resolved = realPath(root)
276-
if (scope.ancestors.includes(resolved)) return true
291+
// Containment, not exact equality. A quoted path often names a *nested*
292+
// dependency root — .../node_modules/host/node_modules/duckdb/... — whose
293+
// outer node_modules is one of `scope.ancestors` but whose inner one is not:
294+
// it is a descendant of that ancestor, not the ancestor itself. Comparing by
295+
// equality let that inner root slip past the exclusion when the CLI started
296+
// below the project root, admitting workspace-controlled code across the
297+
// permission boundary this check exists to enforce.
298+
if (scope.ancestors.some((ancestor) => isWithin(ancestor, resolved))) return true
277299
// Fail closed. With no working directory there is nothing to compare against,
278300
// and treating that as "not workspace content" would admit every root an error
279301
// happens to name — turning the one case where the process cannot see its own
280302
// filesystem into the case with no boundary at all.
281303
if (!scope.cwd) return true
282-
const rel = path.relative(scope.cwd, resolved)
283-
return rel === "" || (!rel.startsWith("..") && !path.isAbsolute(rel))
304+
return isWithin(scope.cwd, resolved)
284305
}
285306

286-
/**
287-
* `node_modules` directories named by an error the runtime raised while trying
288-
* to resolve a package itself.
289-
*
290-
* When ambient resolution fails it often names the exact absolute location it
291-
* was reaching for. That location is better evidence than anything we can
292-
* infer, so it is worth searching — after repairing a concatenated working
293-
* directory, which is the failure this exists for. Returns roots only when they
294-
* exist on disk, so a nonsense path contributes nothing.
295-
*
296-
* Workspace-controlled roots are never returned. `driverSearchRoots()` refuses
297-
* project and ancestor `node_modules` because importing a matching SDK during a
298-
* warehouse read/test would bypass the permission boundary and can expose
299-
* resolved credentials; mining a path out of an error message must not become a
300-
* way around that invariant.
301-
*/
302307
/**
303308
* Absolute paths a runtime quoted inside an error message.
304309
*
@@ -317,6 +322,22 @@ export function quotedAbsolutePaths(message: string): string[] {
317322
return found
318323
}
319324

325+
/**
326+
* `node_modules` directories named by an error the runtime raised while trying
327+
* to resolve a package itself.
328+
*
329+
* When ambient resolution fails it often names the exact absolute location it
330+
* was reaching for. That location is better evidence than anything we can
331+
* infer, so it is worth searching — after repairing a concatenated working
332+
* directory, which is the failure this exists for. Returns roots only when they
333+
* exist on disk, so a nonsense path contributes nothing.
334+
*
335+
* Workspace-controlled roots are never returned. `driverSearchRoots()` refuses
336+
* project and ancestor `node_modules` because importing a matching SDK during a
337+
* warehouse read/test would bypass the permission boundary and can expose
338+
* resolved credentials; mining a path out of an error message must not become a
339+
* way around that invariant.
340+
*/
320341
export function searchRootsFromError(error: unknown): string[] {
321342
const message = error instanceof Error ? error.message : String(error)
322343
const scope = workspaceScope()
@@ -484,15 +505,6 @@ function entryFromManifest(pkgDir: string, specifier: string, pkg: string): stri
484505
return undefined
485506
}
486507

487-
/**
488-
* Import an optional warehouse SDK.
489-
*
490-
* Tries the ambient resolver first so development, the monorepo, and any
491-
* already-working install behave exactly as before, then falls back to
492-
* searching real directories.
493-
*
494-
* @throws {DriverNotInstalledError} when the package is genuinely absent.
495-
*/
496508
/**
497509
* Load an already-resolved package from its own absolute location.
498510
*
@@ -568,9 +580,18 @@ function requireFromLocation(resolved: string): unknown {
568580
/** True when a require failed only because the target is an ES module. */
569581
function isRequireOfEsm(error: unknown): boolean {
570582
const code = error && typeof error === "object" && "code" in error ? (error as { code?: unknown }).code : undefined
571-
if (code === "ERR_REQUIRE_ESM") return true
583+
// Node's synchronous `require()` of ESM handles most modules, but not one
584+
// with top-level `await`: that cannot be represented synchronously, so Node
585+
// raises this code instead of `ERR_REQUIRE_ESM`. Bun's own message for the
586+
// same case is text, not a code. Both mean the same thing this function
587+
// exists to detect — the target is ESM, not that the load is broken — so an
588+
// otherwise-valid async-ESM-only optional package must still fall through to
589+
// the dynamic-import path below rather than being reported as unloadable.
590+
if (code === "ERR_REQUIRE_ESM" || code === "ERR_REQUIRE_ASYNC_MODULE") return true
572591
const message = error instanceof Error ? error.message : String(error)
573-
return /require\(\) of ES Module|Cannot use import statement outside a module|Unexpected token 'export'/i.test(message)
592+
return /require\(\) of ES Module|Cannot use import statement outside a module|Unexpected token 'export'|require\(\) (?:cannot be used on an ESM graph with top-level await|async module .* is unsupported)/i.test(
593+
message,
594+
)
574595
}
575596

576597
/**
@@ -588,6 +609,15 @@ async function loadFromLocation(resolved: string, importer: (spec: string) => Pr
588609
}
589610
}
590611

612+
/**
613+
* Import an optional warehouse SDK.
614+
*
615+
* Tries the ambient resolver first so development, the monorepo, and any
616+
* already-working install behave exactly as before, then falls back to
617+
* searching real directories.
618+
*
619+
* @throws {DriverNotInstalledError} when the package is genuinely absent.
620+
*/
591621
export async function loadOptionalDriver(
592622
driver: DriverName,
593623
specifier: string,
@@ -1014,7 +1044,13 @@ function runNpm(
10141044
}
10151045

10161046
/** @internal — exported only for focused lifecycle and queue unit tests. */
1017-
export const _testing = { killTree, runNpm, runTaskkill, installOptionalDriver: installOptionalDriverInternal }
1047+
export const _testing = {
1048+
killTree,
1049+
runNpm,
1050+
runTaskkill,
1051+
installOptionalDriver: installOptionalDriverInternal,
1052+
isRequireOfEsm,
1053+
}
10181054

10191055
/**
10201056
* Delete `packages` from the managed directory so a reinstall genuinely rebuilds
@@ -1078,6 +1114,13 @@ async function installOptionalDriverInternal(
10781114
// held it has usually just installed the very thing we queued for, so
10791115
// most contenders return "already present" instead of running a second
10801116
// npm over the same tree — which is what produced the ENOTEMPTY races.
1117+
const npmTimeoutMs = options.timeoutMs ?? 180_000
1118+
// Outlast one peer's install. A lock wait shorter than the install it is
1119+
// waiting on means a contender gives up while the holder's npm is still
1120+
// running and then installs unlocked over the same tree, which is the
1121+
// race this lock exists to stop. The two timeouts were independent
1122+
// constants, so raising the install timeout alone silently broke it.
1123+
const lockTimeoutMs = npmTimeoutMs + 60_000
10811124
return withInstallLock(
10821125
dir,
10831126
async (acquired) => {
@@ -1086,12 +1129,16 @@ async function installOptionalDriverInternal(
10861129
}
10871130
return performInstall(driver, packages, dir, options)
10881131
},
1089-
// Outlast one peer's install. A lock wait shorter than the install it is
1090-
// waiting on means a contender gives up while the holder's npm is still
1091-
// running and then installs unlocked over the same tree, which is the
1092-
// race this lock exists to stop. The two timeouts were independent
1093-
// constants, so raising the install timeout alone silently broke it.
1094-
{ timeoutMs: (options.timeoutMs ?? 180_000) + 60_000 },
1132+
{
1133+
timeoutMs: lockTimeoutMs,
1134+
// Published into owner.json for any contender to read: a caller
1135+
// that raised its own install timeout above the generic one-hour
1136+
// backstop must not have this holder's lock reclaimed by a
1137+
// default-configured contender partway through a legitimately long
1138+
// native build. Without this, `hardStaleAfterMs` stayed a fixed
1139+
// default independent of the very timeout this call just raised.
1140+
hardStaleAfterMs: Math.max(lockTimeoutMs, 3_600_000),
1141+
},
10951142
)
10961143
})
10971144
installsInFlight.set(dir, run)
@@ -1144,14 +1191,20 @@ export function installLockPath(dir: string): string {
11441191
* True when `candidate` — trailing separators already stripped — names a
11451192
* filesystem root rather than a directory inside one.
11461193
*
1147-
* The UNC case is the one that is easy to miss: a share root is a root in
1148-
* exactly the way a drive letter is, and appending `.lock` to it names an
1149-
* unrelated share rather than anything under the directory being locked.
1194+
* The UNC case is the one that is easy to miss, and it has more than one
1195+
* spelling: the plain share, its extended-length form (which bypasses
1196+
* `MAX_PATH` and Windows' own UNC parsing), and the equally valid
1197+
* forward-slash spelling `ALTIMATE_DRIVER_DIR` can carry. A share root is a
1198+
* root in exactly the way a drive letter is, and appending `.lock` to any of
1199+
* these forms names an unrelated share rather than anything under the
1200+
* directory being locked.
11501201
*/
11511202
function isFilesystemRoot(candidate: string): boolean {
11521203
if (candidate === "") return true // POSIX "/" strips to ""
11531204
if (/^[A-Za-z]:$/.test(candidate)) return true // "C:\" strips to "C:"
1154-
return /^\\\\[^\\/]+\\[^\\/]+$/.test(candidate) // "\\server\share"
1205+
if (/^\\\\[^\\/]+\\[^\\/]+$/.test(candidate)) return true // "\\server\share"
1206+
if (/^\\\\\?\\UNC\\[^\\/]+\\[^\\/]+$/i.test(candidate)) return true // "\\?\UNC\server\share"
1207+
return /^\/\/[^/]+\/[^/]+$/.test(candidate) // "//server/share"
11551208
}
11561209

11571210
/**
@@ -1175,8 +1228,22 @@ interface LockHolder {
11751228
startedAt?: number
11761229
/** Identifies one acquisition, so a holder only ever releases its own lock. */
11771230
token?: string
1231+
/**
1232+
* This holder's own reclaim backstop, in ms from `startedAt` — the
1233+
* `hardStaleAfterMs` it computed for itself from its own `InstallOptions`.
1234+
* A contender that never saw that option would otherwise apply its own
1235+
* default (or its own explicit override) to a lock someone else is holding
1236+
* for a reason the contender cannot see: a caller that configured a long
1237+
* install timeout for a native build. Absent means "assume the generic
1238+
* default" — e.g. a lock planted directly by an older build or a test.
1239+
*/
1240+
budgetMs?: number
11781241
}
11791242

1243+
/** Upper bound on a trusted `budgetMs`, so a corrupt or hostile owner record
1244+
* cannot wedge the lock directory indefinitely by claiming an absurd one. */
1245+
const MAX_TRUSTED_BUDGET_MS = 24 * 60 * 60_000
1246+
11801247
function readLockHolder(lockDir: string): LockHolder | undefined {
11811248
try {
11821249
const raw = fs.readFileSync(path.join(lockDir, "owner.json"), "utf8")
@@ -1189,6 +1256,7 @@ function readLockHolder(lockDir: string): LockHolder | undefined {
11891256
hostname: typeof holder.hostname === "string" ? holder.hostname : "",
11901257
startedAt: typeof holder.startedAt === "number" ? holder.startedAt : undefined,
11911258
token: typeof holder.token === "string" ? holder.token : undefined,
1259+
budgetMs: typeof holder.budgetMs === "number" && holder.budgetMs > 0 ? holder.budgetMs : undefined,
11921260
}
11931261
} catch {
11941262
return undefined
@@ -1227,7 +1295,14 @@ function isStaleLock(
12271295
// live same-host owner is protected, but only up to a backstop far beyond
12281296
// any real npm run — long enough never to interrupt an install, short
12291297
// enough that a recycled pid cannot wedge the directory permanently.
1230-
return age !== undefined && age > hardMaxAgeMs
1298+
//
1299+
// That backstop is normally this contender's own generic default. When the
1300+
// holder published a longer budget of its own — a caller that explicitly
1301+
// configured a long install timeout for a native build — that declared
1302+
// budget wins: a contender shaped by the generic default must not reclaim
1303+
// an install the holder was explicitly told it could take.
1304+
const backstop = holder.budgetMs !== undefined ? Math.min(holder.budgetMs, MAX_TRUSTED_BUDGET_MS) : hardMaxAgeMs
1305+
return age !== undefined && age > backstop
12311306
}
12321307
// No readable owner, or an owner on another host sharing a home directory.
12331308
// Liveness is not decidable, so age is the only signal there is.
@@ -1443,6 +1518,7 @@ export async function withInstallLock<T>(
14431518
hostname: os.hostname(),
14441519
startedAt: Date.now(),
14451520
token,
1521+
budgetMs: hardStaleAfterMs,
14461522
} satisfies LockHolder),
14471523
)
14481524
} catch {

‎packages/drivers/test/install-lock.test.ts‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -140,16 +140,30 @@ process.exit(0)
140140
// performInstall unlocked — the concurrent npm mutation the lock exists to
141141
// prevent. The hold below is deliberately longer than the timeout so the
142142
// test fails unless the wait is extended each time the lock changes hands.
143+
//
144+
// Without a start barrier this can pass vacuously: the 600ms budget is
145+
// measured from each child's own start, so a ~300ms startup stagger on a
146+
// loaded CI box lets a late child acquire within its own budget and the
147+
// assertion never sees UNLOCKED even if the handover-extension it exists
148+
// to verify is removed. Every child announces itself and waits until all
149+
// three are ready, so they contend from the same instant.
143150
const target = path.join(dir, "drivers")
144151
fs.mkdirSync(target, { recursive: true })
145152
const log = path.join(dir, "peers.txt")
153+
const ready = path.join(dir, "peers-ready")
154+
fs.mkdirSync(ready)
146155

147156
const child = path.join(dir, "peer.ts")
148157
fs.writeFileSync(
149158
child,
150159
`import fs from "node:fs"
151160
import { withInstallLock } from ${JSON.stringify(resolveModule)}
152-
const [target, log, id] = process.argv.slice(2)
161+
const [target, log, ready, id] = process.argv.slice(2)
162+
fs.writeFileSync(\`\${ready}/\${id}\`, "1")
163+
const deadline = Date.now() + 20000
164+
while (fs.readdirSync(ready).length < 3 && Date.now() < deadline) {
165+
await new Promise((r) => setTimeout(r, 5))
166+
}
153167
await withInstallLock(target, async (acquired) => {
154168
fs.appendFileSync(log, \`\${acquired ? "locked" : "UNLOCKED"} \${id}\\n\`)
155169
if (acquired) await new Promise((r) => setTimeout(r, 400))
@@ -159,7 +173,7 @@ process.exit(0)
159173
)
160174

161175
const kids = Array.from({ length: 3 }, (_, i) =>
162-
Bun.spawn(["bun", child, target, log, String(i)], { stdout: "ignore", stderr: "ignore" }),
176+
Bun.spawn(["bun", child, target, log, ready, String(i)], { stdout: "ignore", stderr: "ignore" }),
163177
)
164178
expect(await Promise.all(kids.map((k) => k.exited))).toEqual([0, 0, 0])
165179

@@ -278,6 +292,35 @@ process.exit(0)
278292
expect(sawAcquired).toBe(true)
279293
})
280294

295+
test("honors a live owner's declared budget over this contender's own backstop default", async () => {
296+
// A caller that explicitly configured a long install timeout for a native
297+
// build publishes that as its own budget. A different, default-configured
298+
// contender watching the same lock must respect it — its own generic
299+
// one-hour backstop must not reclaim an install the holder was told it
300+
// could take up to four hours.
301+
const target = path.join(dir, "drivers")
302+
fs.mkdirSync(target, { recursive: true })
303+
const held = plantLock(target, {
304+
pid: process.pid,
305+
hostname: os.hostname(),
306+
startedAt: Date.now() - 90 * 60_000, // 90 minutes old
307+
budgetMs: 4 * 60 * 60_000, // owner declared a 4 hour budget
308+
})
309+
310+
let sawAcquired: boolean | undefined
311+
await withInstallLock(
312+
target,
313+
async (acquired) => {
314+
sawAcquired = acquired
315+
},
316+
// This contender's own default backstop (one hour) would reclaim a
317+
// 90-minute-old lock; the owner's declared budget must win instead.
318+
{ timeoutMs: 200, staleAfterMs: 1_000, hardStaleAfterMs: 60 * 60_000, pollMs: 20 },
319+
)
320+
expect(sawAcquired).toBe(false)
321+
expect(fs.existsSync(held)).toBe(true)
322+
})
323+
281324
test("gives up by the deadline when a stale lock cannot be claimed", async () => {
282325
// A claim can fail persistently: a lock owned by another user, or a
283326
// container that permits inspection but not rename. Retrying that without

‎packages/drivers/test/resolve-argv-isolation.test.ts‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,14 @@ function importerThrowingAt(target: string) {
6565
}
6666

6767
describe("a driver load does not see the host's command line", () => {
68-
test("the loaded module cannot observe --dir", async () => {
68+
// These three tests share module-level state — process.argv, root, pkgDir —
69+
// and each afterEach removes root. Bun runs a file's tests sequentially by
70+
// default, so nothing races today, but that is an ambient property of how
71+
// this suite happens to be invoked (no `--concurrent` flag anywhere in this
72+
// repo), not a guarantee the tests themselves enforce. test.serial pins it,
73+
// so a future `bun test --concurrent` cannot make one test's argv mutation
74+
// or afterEach cleanup clobber another's.
75+
test.serial("the loaded module cannot observe --dir", async () => {
6976
process.argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project", "--print-logs"]
7077

7178
const loaded: any = await loadOptionalDriver(
@@ -82,7 +89,7 @@ describe("a driver load does not see the host's command line", () => {
8289
expect(mod.manifestPath.startsWith("/some/project")).toBe(false)
8390
})
8491

85-
test("restores the command line afterwards", async () => {
92+
test.serial("restores the command line afterwards", async () => {
8693
const argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project"]
8794
process.argv = argv
8895

@@ -91,7 +98,7 @@ describe("a driver load does not see the host's command line", () => {
9198
expect(process.argv).toEqual(argv)
9299
})
93100

94-
test("restores the command line even when the load throws", async () => {
101+
test.serial("restores the command line even when the load throws", async () => {
95102
fs.writeFileSync(path.join(pkgDir, "index.js"), "throw new Error('broken driver')\n")
96103
const argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project"]
97104
process.argv = argv

0 commit comments

Comments
 (0)