Skip to content

Commit 136c695

Browse files
anandgupta42claude
andauthored
fix(drivers): load a driver from the location the failing runtime named, and make installs concurrency-safe (#1201)
* fix(drivers): load a driver from the location the failing runtime named A globally installed CLI (`npm install -g`, package tree under `/usr/lib/node_modules/altimate-code`) run from an unrelated working directory could not load any warehouse driver — 8 of 8 trials on a cold VM: DuckDB driver found at duckdb but failed to load: ENOENT: no such file or directory, open '<cwd>/usr/lib/node_modules/altimate-code/node_modules/duckdb/package.json' The file exists at that path without the `<cwd>` prefix. Ambient resolution concatenated the working directory onto an already-absolute path, which is a runtime behaviour we cannot change — but it names the correct location in the error, and that is better evidence than anything we can infer. `searchRootsFromError` now mines the paths an ambient failure quotes, repairs a concatenated working directory, and searches the enclosing `node_modules` first. `repairCwdPrefixedPath` is deliberately conservative: it fires only when the named path is absent, is genuinely prefixed by the working directory, and the de-prefixed remainder exists on disk, so a legitimately nested `<cwd>/node_modules/…` is left alone. The message was the second half of the bug. `found at duckdb` named the bare specifier as though it were a location, so a package that had never been located anywhere read as a load failure at a known path — which is why this was diagnosed as a load bug rather than a search-coverage one. Both sites that produced it now say what actually happened: the default module resolution failed, here is where we looked, and here is the on-disk copy we also tried. Verified in the shape it occurs in, not in unit tests alone: a probe compiled with the production `Bun.build` options (`external`, `autoloadPackageJson`), laid out as a real `npm install -g` tree with a real `npm install duckdb`, run from an unrelated cwd with no `NODE_PATH` and no `ALTIMATE_BIN_DIR`. before: 8/8 fail, reproducing the reported message verbatim after: 8/8 load the driver Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): make on-demand driver installs safe across processes The managed driver directory is shared by every CLI process on the machine, and `installsInFlight` is an in-process Map — it cannot see other processes. Eight CLIs starting together each ran `npm install` over the same tree: npm install failed (exit 217) … ENOTEMPTY … rmdir /root/.local/share/altimate-code/drivers/node_modules/duckdb/… That is not benchmark-specific. Any concurrent use of the CLI hits it, and the result is a driver that appears broken on a machine where nothing is wrong. `withInstallLock` takes a cross-process lock before mutating the directory. `mkdir` is atomic and fails EEXIST when the directory exists on both POSIX and Windows, which makes a lock directory the portable primitive; it lives beside the install directory so npm never sees it as stray package content. Stale locks are broken two ways, because neither alone is sufficient: a dead owner on this host, and age, which is the only signal available for a lock left by another host sharing a home directory. After acquiring, readiness is re-checked. The peer that held the lock has usually just installed the very thing we queued for, so most contenders return "already present" rather than running a second npm over the same tree. On timeout the install proceeds unlocked rather than failing: a racing install is recoverable and the readiness check afterwards is authoritative, whereas refusing to install because a peer is slow turns contention into a hard error. Also adds cwd/execPath and a cwd-prefix note to driver load failures. Two separate investigations have now diagnosed a load failure from the error text alone and got it wrong, because the text named a path without saying what the process's own view of the filesystem was. The exclusion claim is about separate processes, so the test spawns separate processes — an in-process test cannot establish it. The control confirms the test bites: the same four processes without the lock interleave completely (four enters before any exit). 10/10 repeat runs green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): keep harvested roots inside the trust boundary, and make the install lock hold Review findings on #1201, grouped by what they actually break. **Harvested roots crossed a deliberate security boundary.** `driverSearchRoots()` refuses project and ancestor `node_modules` because importing a workspace- controlled SDK during a warehouse read/test bypasses the permission boundary and can expose resolved credentials. `searchRootsFromError()` mined any absolute path an error quoted and prepended it, so an ambient failure naming a project-local `node_modules` routed straight around that invariant. Harvested roots are now filtered through the same exclusion. **They also preempted the managed installation.** Prepending meant a stale or broken copy the runtime happened to name won over the driver we installed, which inverts the documented "managed install dir comes first" ordering. Trusted roots now come first and harvested roots are appended, which still recovers the original failure: a harvested root is reached whenever the roots ahead of it resolve nothing. **Windows never harvested anything.** The extraction regex accepts `C:/…` and `C:\…`, but the `node_modules` marker is built from `path.sep`, so a forward-slash path could never match a backslash marker. `enclosingNodeModulesRoot` normalises separators, and takes `sep` as a parameter so the Windows behaviour is tested from a POSIX host rather than asserted. `repairCwdPrefixedPath` now also handles the two Windows concatenation shapes, which have no separator to carry. **The lock silently degraded to unlocked in exactly the cold-start case it exists for.** `<dir>.lock` sits beside the managed directory, and on a fresh machine nothing has created the XDG data directory yet — `performInstall` is the first thing that does, and it runs after the lock attempt. The non-recursive `mkdir` failed ENOENT, took the "cannot lock" branch, and dropped every concurrent CLI into an unlocked install. The parent is now created first. **Stale-lock recovery could admit two installers.** Two processes could both judge a lock stale, and deleting by pathname let the loser delete the winner's fresh lock. Claiming is now a `rename`, which exactly one process can win. **A lock could be released out from under its successor.** An owner whose lock was broken as stale would, on the way out, delete the lock a peer had since taken. Release now only removes a lock still carrying its own token. **Age aged out live installs.** `isStaleLock` applied the age check unconditionally, so a live same-host owner running a slow native build past `staleAfterMs` had its lock broken and a peer entered — reintroducing the very ENOTEMPTY race. Where liveness is decidable (owner on this host) it is now the only signal; age applies only where it cannot be (no readable owner, or another host sharing a home directory). **The lock path was outside the approved permission pattern.** The tool asks for `external_directory` on `<dir>/*`, but the lock is the sibling `<dir>.lock` — so creating, writing and removing it mutated an external path the user never approved. It is now included in the request. **Two unrelated cwd faults on the failure path.** `process.cwd()` throws once the working directory is removed, and both `loadDiagnostics` and — pre-existing — `resolveOptionalPackage`'s `createRequire` anchor called it unguarded while a driver failure was being formatted, replacing the real diagnosis with an unrelated `uv_cwd` ENOENT. Both are guarded; the anchor never needed cwd, since resolution is driven by the explicit `paths`. Tests. The two end-to-end resolution tests used the specifier `duckdb`, which the repository's own `packages/drivers/node_modules` can satisfy no matter what the harvesting code does — they passed while proving nothing, and would have kept passing after the reordering above. They now use specifiers that exist nowhere but the tree the test builds, and assert on an export marker, so they establish which root actually satisfied the load. The cross-process exclusion test gained a start barrier: without one, a scheduler that ran the four children serially satisfied the no-overlap assertion even with a completely broken lock. The control confirms the barriered test still bites — four unlocked children interleave completely (enter 1, enter 2, enter 3, enter 0) and the assertion rejects on the second enter. 9 repeat runs, 72/72 green. Not verified: Windows on hardware. The separator handling is now unit-tested through the `sep` parameter, but nothing here ran on Windows. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): bound the stale-lock retry, and harvest every enclosing root Second review round. One of these is a bug the previous commit introduced. **The stale-claim branch could spin forever.** `claimStaleLock` returned without reporting failure and the loop `continue`d unconditionally, skipping both the deadline check and the sleep. A claim that fails persistently — a lock owned by another user, or a container that permits inspection but not rename — meant `mkdir` EEXIST, judged stale, claim fails, repeat, at full CPU with no timeout. The claim now reports success, and only a successful claim skips the wait; a failed one falls through to the normal deadline-and-sleep path. The regression test hangs rather than fails if that bound is lost again. **Stale recovery could still move a live lock aside.** The rename made two cleaners safe against each other, but not against the stale→fresh transition: a peer could release and re-acquire between the staleness verdict and the rename. The claim now re-reads the owner record before renaming, and — because nothing makes "read the owner" and "rename" one operation — verifies what it actually moved afterwards, restoring it if a peer had re-taken the lock. That narrows the window rather than closing it, which is stated plainly rather than implied. **Liveness-only staleness could wedge the lock permanently.** Making a live same-host owner immune to age fixed the interrupted-install bug but introduced a worse one: `processExists` answers "some process holds this pid", so a crashed owner whose pid is recycled by an unrelated long-lived process would hold the lock forever, with every later install waiting out its timeout and then running unlocked. A live owner is now protected only up to a backstop far beyond any real npm run (1h default), which bounds the wedge without interrupting an install. **Release could delete a successor's lock during a window the token cannot cover.** The directory is created before `owner.json` is written, so a successor that re-took the lock in that window holds a live lock carrying no token, and the token check let it be removed. Release now also compares the lock directory's inode, captured at acquire. **Every path this mechanism writes now sits under one approved prefix.** Stale recovery renamed the lock to a sibling of the container, which the tool's `<dir>.lock/*` permission does not cover. The atomic lock moved inside the container as `<dir>.lock/held`, and claims rename to `<dir>.lock/stale-…`, so both are covered by the pattern already brokered. **The lock wait could be shorter than the install it waits on.** The two timeouts were independent constants: raising the install timeout past the lock wait meant a contender gave up while the holder's npm was still running and then installed unlocked over the same tree. The lock wait is now derived from the install timeout. **Nested dependency paths harvested the wrong root.** A quoted path such as `/opt/node_modules/duckdb/node_modules/node-addon-api/…` yielded only the innermost `node_modules`, which holds the dependency — while the driver being looked for sits in the outer one, so it was never found. All enclosing roots are now harvested, innermost first, each subject to the same workspace exclusion. **The workspace exclusion compared lexical paths while resolution followed symlinks.** A link whose lexical path sits outside the working directory but whose target sits inside it passed the check and would have been imported. Containment now compares real paths, falling back to lexical when the link cannot be followed. Gates: typecheck 13/13 (forced, not cached); marker check ok; lint 5903 warnings and 1 error, byte-identical to `main`'s own baseline (the error is the known pre-existing `consistent-return` in `packages/http-recorder/test/record-replay.test.ts`); `packages/drivers` 255 pass; `test/altimate` 4226 pass. Lock tests 99/99 across 9 repeat runs. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): fail closed on an unavailable cwd, and keep a root path absolute Two narrow fixes from the latest review round. **The workspace exclusion failed open when `process.cwd()` was unavailable.** `isWorkspaceRoot` returned false with no working directory to compare against, so every `node_modules` root an error happened to name was admitted — turning the one case where the process cannot see its own filesystem into the case with no boundary at all. It now fails closed: no cwd means no harvested roots. This was introduced by the guard added two commits ago, which is exactly the sort of thing a fail-open default hides. **A filesystem root lost its leading separator when deriving the lock path.** `installLockPath` stripped every trailing separator, so `/` became the relative `.lock` and `C:\` the drive-relative `C:.lock`. Two processes started from different working directories would then take different locks while installing into the same directory, which is the concurrent-npm mutation the lock exists to prevent. Roots are now left intact; trailing separators are still stripped everywhere else so `<dir>/` and `<dir>` agree on one lock. Gates: typecheck 13/13 (forced); marker check ok; lint 5903 warnings and 1 error, identical to `main`'s baseline; `packages/drivers` 258 pass; `test/altimate` 4226 pass. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * test(drivers): add a chdir arm, so a green suite means something about `--dir` Six pilots were spent on a driver-load failure that only appeared under `--dir`, which calls `process.chdir()` (cli/cmd/run.ts) before any driver is loaded. Every local verification — and the rig's own pre-flight probe — ran without a chdir, so a green suite said nothing about the configuration that actually failed. This is the same class as an `--instance-dir` arm that never exercised the code it existed to test. Four tests: resolve before and after a chdir and compare, resolve from a directory unrelated to the install tree, assert a package present only under the working directory is never resolved out of it, and check that a chdir between two resolutions does not change the answer. They pass on the current resolver, which is the point — this is a guard, not a bug report. `resolveOptionalPackage` drives resolution from the explicit `paths` argument rather than from the anchor's base, so the working directory does not currently reach the result. That is a property worth pinning, because it is invisible in review and its absence is expensive to diagnose. Measured, not assumed: on Debian 12, with a binary cross-compiled using the production compile options against a real `npm install -g` tree and a real `npm install duckdb`, run as root from an unrelated directory, adding a `process.chdir()` between process start and driver load changed nothing. Bare import, `import(file://)`, `createRequire(abs)`, the `__dirname` a loaded CJS module sees, and `loadOptionalDriver` were byte-identical with and without it, and all succeeded. So on Bun 1.3.14 a compiled binary does not re-anchor absolute module paths after a chdir, and chdir alone does not explain the field failure. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): handle UNC paths, and give the install lock a per-holder budget Three review-round gaps, two of them Windows shapes that a macOS suite cannot notice on its own. **`installLockPath` broke on a UNC share root.** It special-cased the POSIX and drive-letter roots but not `\\server\share`, which stripped to `\\server\share.lock` — naming a *different network share* rather than anything inside the directory being locked. Two processes installing into a share would then take different locks, which is the concurrent npm mutation the lock exists to prevent. Root detection now covers all three shapes and the lock always lands inside the root. **The harvesting regex accepted no UNC path.** A Windows error quoting `\\server\share\node_modules\duckdb\package.json` yielded no roots at all, so a driver on a share stayed unfindable even though the error named its exact location. The pattern is now a named export, so the claim is testable from any platform instead of resting on inspection. **The lock wait outlasted only one peer.** Every process counts its deadline from its own start, so a single budget covers a single holder; with three or more contenders the last one's deadline expired part-way through somebody else's install and it fell through to an unlocked `performInstall`. The budget is now per holder — seeing the lock change hands is proof the queue is moving rather than wedged — with a bounded number of extensions so a machine that keeps feeding in contenders cannot block a caller indefinitely. Each test fails with its own fix reverted and passes with it: 1 of 10 for each Windows shape, 1 of 14 for the multi-peer wait. The multi-peer test spawns real processes, holds 400ms against a 600ms budget, and asserts none of the three ran unlocked. 12/12 repeat runs green. Also hardens the chdir arm per review. It asked for `duckdb`, which the repo's own `packages/drivers/node_modules` satisfies through the execPath and module-location roots no matter what the resolver does — it would have passed while proving nothing. It now uses specifiers that exist nowhere but the tree each test builds and asserts on an export marker, so a pass establishes which root satisfied the load. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): load a driver from its own directory, not through the process cwd Measured twice independently, from a global install run under `--dir`: the package manifest consulted while loading a resolved driver is looked up at the working directory concatenated onto the driver's absolute path. ENOENT ... open '<cwd>/usr/lib/.../duckdb/package.json' while the file exists at that path without the `<cwd>` prefix. Materialising that concatenated path makes the failing invocation pass, and so does running with cwd `/`, which makes the concatenation a no-op. Both directions agree, so the working directory is an input to a lookup that has no business consulting it. There is no invocation-level fix: `--dir` is itself what sets the working directory (`cli/cmd/run.ts`), so cwd `/` and `--dir <run_dir>` are mutually exclusive. `import(pathToFileURL(abs))` is not enough, because it is the ESM loader's own manifest lookup that goes through cwd. Loading through a CommonJS require anchored at the resolved file makes every nested lookup — the manifest included — relative to the driver's own directory. The drivers loaded this way are CommonJS; an ESM-only package still needs the loader, so that path remains and is taken only for the error that specifically means "this is ESM". `process.chdir()` around the load would also neutralise the concatenation and is deliberately not used. It is global mutable state, these loads happen under concurrency, and it would corrupt resolution for unrelated work non-deterministically — worse than the fault it patches. The regression test loads a fixture that reports its own `__dirname` while the working directory is elsewhere, so the assertion is about which directory the module came from rather than that it merely loaded. With the load site reverted it fails, and it fails with the field's exact shape: "failed to load from the default module resolution: ENOENT ... package.json / A copy at ... was also tried and failed to load". Note what that test does and does not establish. It proves the load no longer routes through the ESM loader, which is the fix's mechanism. It does not simulate the runtime's cwd-joining itself — no environment outside the affected rig has reproduced that, across eight eliminated hypotheses — so the end-to-end confirmation has to come from a build run there. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): keep our command line out of a driver's own resolution `@mapbox/node-pre-gyp`, which packages several native modules including DuckDB, resolves a module's manifest by parsing the **host application's** `process.argv`. `find()` passes `argv: process.argv` into its own `Run`, `nopt` abbreviation-matches whatever it finds against node-pre-gyp's option list, and `node-pre-gyp.js:164` then does: package_json_path = path.join(this.opts.directory, package_json_path) `path.join`, not `path.resolve` — so an absolute manifest path is not discarded. Our `--dir` abbreviates to node-pre-gyp's `--directory`, so `altimate-code run --dir <project>` made the driver look for its manifest at `<project>` concatenated with the manifest's own absolute path, and the load failed with an ENOENT naming a path that had never existed. That the failing path also looked cwd-prefixed was a coincidence: `--dir` is what sets the working directory, so the two values were always equal. Driver loads now run with `process.argv` trimmed to `[argv[0], argv[1]]`. Swapping a global is safe here in a way `process.chdir()` would not be: the load is a synchronous `require`, JavaScript is single-threaded, and there is no `await` between the swap and the restore, so no concurrent work can observe it. The asynchronous ESM fallback is deliberately left alone for that reason. Considered and rejected: renaming `--dir`. The collision is real, but the flag is public, renaming breaks every existing invocation, and it would fix only the one option name that happens to collide today rather than the mechanism. Not specific to us, to DuckDB, or to a compiled binary — any CLI embedding a node-pre-gyp-packaged module and accepting a flag that abbreviates to `--directory` is exposed. A long argv in a compiled binary only made it visible. Of the drivers installed here, only `duckdb` pulls in node-pre-gyp today; the fix is applied to every driver load rather than to DuckDB, because the exposure is a property of the packaging tool, not of the driver. The regression test reproduces the arithmetic of the one line that broke: a fixture that reads `--dir` out of `process.argv` and joins it onto its own absolute manifest path. With the neutralisation reverted it fails, reporting the flag as seen. Two further tests pin that the command line is restored afterwards, including when the load throws. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix(drivers): address PR review threads on the install lock and resolver Resolves 17 of the 28 open review threads on this PR (chatgpt-codex-connector, cubic-dev-ai, kilo-code-bot, coderabbitai), leaves 7 open with a design question or a duplicate pointer, and folds in #1205's empty-searched-roots error-message fix (that PR was closed as redundant with this one). Fixed: - `isWorkspaceRoot()` compared a harvested root against ancestor `node_modules` by exact equality, so a *nested* dependency root under an ancestor (`.../node_modules/host/node_modules/duckdb`) slipped past the exclusion when the CLI started below the project root — a real workspace-boundary bypass (P1). Now uses containment via a new `isWithin` helper. - `isRequireOfEsm()` did not recognize Node's `ERR_REQUIRE_ASYNC_MODULE` or Bun's equivalent message, so an ESM-only driver with top-level `await` was reported as broken instead of falling through to dynamic import. Verified the exact Node 22 error shape empirically. - `LockHolder` now carries an optional `budgetMs`, published from the holder's own effective `hardStaleAfterMs` (itself now derived from the configured install `timeoutMs`, not a fixed constant). `isStaleLock` prefers a live holder's declared budget over a contender's own generic backstop, so a default-configured contender no longer reclaims a lock whose owner explicitly configured a longer install timeout. - `isFilesystemRoot()` now recognizes extended-length UNC (`\\?\UNC\server\share`) and forward-slash UNC (`//server/share`) roots, alongside the plain backslash UNC form already handled. - `DriverNotInstalledError` named no location when the searched-roots list was empty (bare "Searched 0 locations:"); now names the expected managed `node_modules` directory. This is #1205's fix, folded in here since that PR was closed as redundant. - Two detached JSDoc blocks reattached to the functions they document. - Test-quality fixes: `resolve-chdir.test.ts` beforeEach now restores cwd so tests don't depend on execution order; the cwd-prefix-repair test now actually constructs a concatenated path instead of one already resolvable without the repair; `install-lock.test.ts`'s multi-peer test gained a start barrier so it can't pass vacuously; `resolve-argv-isolation.test.ts` tests marked `test.serial` against a future concurrent test run. Left open (design questions, replied in-thread rather than resolved unilaterally): - Stale-lock reclamation and release are built on comparing owner records across a directory rename, which several reviewers (from different angles — ownerless-lock races, inode reuse, TOCTOU on stale-claim restore, non-atomic release) converge on as needing a different primitive: an `O_EXCL` sentinel carrying an acquisition token, not the current rename-and-compare protocol. Four of these were already triaged and left open in-thread before this pass; seven more converging on the same conclusion are now replied to and left open as duplicates or extensions of that same finding. - A contender's wait deadline is still derived only from its own `timeoutMs`, not a holder's declared budget — attempted an unconditional extension and reverted it after it blocked a short-timeout caller for a holder's entire remaining budget (over two hours in testing). Left open with two bounded-design options rather than picking one. - Canonicalizing `ALTIMATE_DRIVER_DIR` before deriving the lock path (so a symlink and its real path share a lock) conflicts with the explicitly handled cold-start case where the directory does not exist yet. Left open with two options. Gates: `bun test packages/drivers` 289/289 green (stable across repeated runs), `bun run typecheck` clean across all 15 typechecked packages, upstream marker check clean (drivers is not upstream-shared). * fix(drivers): replace inode-based lock ownership with an atomic token The Driver E2E CI failure on the previous push (packages/drivers/test/ install-lock.test.ts:381, "does not delete a successor lock that has no owner record yet") was not a flake: it is the exact inode-reuse race that four review threads on this PR converged on (kilo-code-bot, cubic-dev-ai, chatgpt-codex-connector), reproducing on Linux/tmpfs in CI and not on macOS/APFS locally, which is why it was invisible before. Redesign: the lock is now a single file, created with `fs.writeFileSync(..., { flag: "wx" })` — O_CREAT|O_EXCL, portable to POSIX and Windows — whose content, a `crypto.randomUUID()` acquisition token, is written as part of that same atomic call. The earlier scheme created a directory atomically and then wrote the ownership token into a file *inside* it as a second, non-atomic step, leaving a real window where a lock existed with no way to tell who owned it. `releaseInstallLock` no longer falls back to `fs.statSync(...).ino` in that window — it no longer exists, so the fallback is gone entirely. The stale-claim restore step (rename a judged-stale lock aside, then put it back if a peer had re-taken it) now uses `fs.linkSync` instead of `fs.renameSync` for the restore. `rename` onto an existing destination silently replaces it, which is how a third process's freshly-created live lock could get clobbered during a stale-claim race; `link` fails closed with EEXIST there instead. Regression test: `does not delete a successor lock with no readable token, even at the same inode`. It overwrites the lock file's content in place (same inode guaranteed, different owner) rather than relying on the OS's own inode-reuse timing, so it reproduces deterministically on every platform instead of only on Linux/tmpfs. Verified directly that it fails against the old inode-fallback logic (temporarily restored it, confirmed the failure, reverted) before relying on it as a regression guard. Two review comments landed on the previous push's own new code and are fixed here too: - `isWithin()` (this PR's own containment fix for the nested-roots P1) used `rel.startsWith("..")` to detect parent traversal, which also matches a genuine descendant whose name merely starts with those two characters (e.g. `base/..evil/node_modules`) — wrongly admitting a workspace-controlled root across the permission boundary. Fixed to check `rel === ".."` or `rel.startsWith(".." + sep)` specifically. - `isFilesystemRoot()`'s UNC recognition (this PR's own forward-slash/ extended-length UNC fix) did not accept a mixed separator spelling (`\\server/share`), which Node's Windows path handling normalizes to the same root as the all-backslash form. Consolidated into one separator-flexible pattern; the `\\?\` extended-length prefix stays backslash-only since it explicitly disables that normalization. Thread disposition (of the 7 threads escalated as needing this exact redesign, plus 2 of the original 4 pre-session escalations describing the same root cause): resolved — dfzlm (stale-claim restore), df1hp (kilo-code-bot inode reuse), df2j- (ownerless-lock race), df2kA (TOCTOU on reclaim, substantially — the realistic 2-process race is closed, a narrower 3-process syscall-level interleaving remains structurally possible and is noted rather than claimed fully closed), df2kI (inode uniqueness, duplicate), df6Rf (reusable inodes, duplicate), diqPm (atomic release, substantially — same residual as df2kA). Left open, unaffected by this redesign: dfzlo (acquire before readiness), dfzlq (npm process-group tracking), dimYr (contender wait deadline), dimYu (canonicalize before lock) — all orthogonal to ownership representation. Gates: `bun test packages/drivers` 293/293 green, stable across repeated local runs; full monorepo `bun run typecheck --force` clean (13 packages, genuine re-execution not cache replay); upstream marker check clean. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent dd4c3a2 commit 136c695

9 files changed

Lines changed: 2196 additions & 19 deletions

‎packages/drivers/src/resolve.ts‎

Lines changed: 856 additions & 12 deletions
Large diffs are not rendered by default.

‎packages/drivers/test/install-lock.test.ts‎

Lines changed: 425 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
2+
import fs from "node:fs"
3+
import os from "node:os"
4+
import path from "node:path"
5+
6+
import { loadOptionalDriver } from "../src/resolve"
7+
8+
// `@mapbox/node-pre-gyp` resolves a native module's manifest by parsing the
9+
// HOST APPLICATION's `process.argv`. `find()` passes `argv: process.argv` into
10+
// its own `Run`, `nopt` abbreviation-matches our flags against node-pre-gyp's
11+
// option list, and `node-pre-gyp.js:164` then does
12+
//
13+
// package_json_path = path.join(this.opts.directory, package_json_path)
14+
//
15+
// `path.join`, not `path.resolve` — so an absolute manifest path is not
16+
// discarded. Our `--dir` abbreviates to `--directory`, and the driver ended up
17+
// looking for its manifest at `<--dir value>` + the manifest's absolute path.
18+
//
19+
// The fixture below reproduces exactly that arithmetic. It does not stand in
20+
// for node-pre-gyp in general; it stands in for the one line that broke.
21+
22+
const FIXTURE = "altimate-argv-fixture"
23+
const savedArgv = process.argv
24+
25+
let root = ""
26+
let nodeModules = ""
27+
let pkgDir = ""
28+
29+
beforeEach(() => {
30+
root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "resolve-argv-")))
31+
nodeModules = path.join(root, "node_modules")
32+
pkgDir = path.join(nodeModules, FIXTURE)
33+
fs.mkdirSync(pkgDir, { recursive: true })
34+
fs.writeFileSync(
35+
path.join(pkgDir, "package.json"),
36+
JSON.stringify({ name: FIXTURE, version: "1.0.0", main: "index.js" }),
37+
)
38+
fs.writeFileSync(
39+
path.join(pkgDir, "index.js"),
40+
[
41+
"const path = require('path')",
42+
"const i = process.argv.indexOf('--dir')",
43+
"const directory = i !== -1 ? process.argv[i + 1] : undefined",
44+
"const manifest = path.join(__dirname, 'package.json')",
45+
"module.exports = {",
46+
" sawDirFlag: directory !== undefined,",
47+
" manifestPath: directory ? path.join(directory, manifest) : manifest,",
48+
"}",
49+
"",
50+
].join("\n"),
51+
)
52+
})
53+
54+
afterEach(() => {
55+
process.argv = savedArgv
56+
if (root) fs.rmSync(root, { recursive: true, force: true })
57+
})
58+
59+
/** Reach the fixture the way the real failure does: via the harvested root. */
60+
function importerThrowingAt(target: string) {
61+
const ambient = Object.assign(new Error(`ENOENT: no such file or directory, open '${target}'`), { code: "ENOENT" })
62+
return async () => {
63+
throw ambient
64+
}
65+
}
66+
67+
describe("a driver load does not see the host's command line", () => {
68+
// These three tests share module-level state — process.argv, root, pkgDir —
69+
// and each afterEach removes root. Bun runs a file's tests sequentially by
70+
// default, so nothing races today, but that is an ambient property of how
71+
// this suite happens to be invoked (no `--concurrent` flag anywhere in this
72+
// repo), not a guarantee the tests themselves enforce. test.serial pins it,
73+
// so a future `bun test --concurrent` cannot make one test's argv mutation
74+
// or afterEach cleanup clobber another's.
75+
test.serial("the loaded module cannot observe --dir", async () => {
76+
process.argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project", "--print-logs"]
77+
78+
const loaded: any = await loadOptionalDriver(
79+
"duckdb",
80+
FIXTURE,
81+
importerThrowingAt(path.join(pkgDir, "package.json")),
82+
)
83+
const mod = loaded?.default ?? loaded
84+
85+
// Without argv neutralisation the fixture sees the flag and joins, which is
86+
// precisely what sent the driver after a manifest that never existed.
87+
expect(mod.sawDirFlag).toBe(false)
88+
expect(mod.manifestPath).toBe(path.join(pkgDir, "package.json"))
89+
expect(mod.manifestPath.startsWith("/some/project")).toBe(false)
90+
})
91+
92+
test.serial("restores the command line afterwards", async () => {
93+
const argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project"]
94+
process.argv = argv
95+
96+
await loadOptionalDriver("duckdb", FIXTURE, importerThrowingAt(path.join(pkgDir, "package.json")))
97+
98+
expect(process.argv).toEqual(argv)
99+
})
100+
101+
test.serial("restores the command line even when the load throws", async () => {
102+
fs.writeFileSync(path.join(pkgDir, "index.js"), "throw new Error('broken driver')\n")
103+
const argv = [savedArgv[0], savedArgv[1], "run", "--dir", "/some/project"]
104+
process.argv = argv
105+
106+
let failed = false
107+
try {
108+
await loadOptionalDriver("duckdb", FIXTURE, importerThrowingAt(path.join(pkgDir, "package.json")))
109+
} catch {
110+
failed = true
111+
}
112+
113+
expect(failed).toBe(true)
114+
expect(process.argv).toEqual(argv)
115+
})
116+
})
Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
import { afterEach, beforeEach, describe, expect, test } from "bun:test"
2+
import fs from "node:fs"
3+
import os from "node:os"
4+
import path from "node:path"
5+
6+
import { createRequire } from "node:module"
7+
import { pathToFileURL } from "node:url"
8+
9+
import { driverSearchRoots, loadOptionalDriver, resolveOptionalPackage, searchRootsFromError } from "../src/resolve"
10+
11+
// Specifiers that exist nowhere but the tree each test builds. Asking for a
12+
// real driver name would let the repo's own `packages/drivers/node_modules`
13+
// satisfy the lookup through the execPath and module-location roots — which no
14+
// environment isolation can suppress — so the test would pass while proving
15+
// nothing about which root actually won.
16+
const FIXTURE = "altimate-chdir-fixture"
17+
const CWD_ONLY = "altimate-cwd-only-fixture"
18+
const MARKER = "resolved-from-the-fixture-tree"
19+
20+
// Six pilots were spent on a driver-load failure that only appeared under
21+
// `--dir`, which calls `process.chdir()` (cli/cmd/run.ts) before any driver is
22+
// loaded. Every local verification — and the rig's own pre-flight probe — ran
23+
// without a chdir, so a green suite said nothing about the configuration that
24+
// actually failed.
25+
//
26+
// This arm exists so that stops being true. Resolution must not depend on the
27+
// working directory the process happens to hold when a driver is loaded, and a
28+
// regression that reintroduces a cwd anchor has to fail here.
29+
30+
let root = ""
31+
let pkgRoot = ""
32+
let nodeModules = ""
33+
let elsewhere = ""
34+
const originalCwd = process.cwd()
35+
36+
function writePackage(dir: string, name: string, main: string, body: string) {
37+
const pkgDir = path.join(dir, name)
38+
fs.mkdirSync(pkgDir, { recursive: true })
39+
fs.writeFileSync(path.join(pkgDir, "package.json"), JSON.stringify({ name, version: "1.0.0", main }))
40+
fs.writeFileSync(path.join(pkgDir, main), body)
41+
return pkgDir
42+
}
43+
44+
beforeEach(() => {
45+
// Each test's starting cwd must not depend on the previous test's afterEach
46+
// having run — restore it here too, so a test that fails before reaching its
47+
// own afterEach (or a future `.concurrent` run) cannot leave a stale cwd for
48+
// the next test.
49+
process.chdir(originalCwd)
50+
root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "resolve-chdir-")))
51+
pkgRoot = path.join(root, "lib", "node_modules", "altimate-code")
52+
nodeModules = path.join(pkgRoot, "node_modules")
53+
fs.mkdirSync(nodeModules, { recursive: true })
54+
writePackage(nodeModules, FIXTURE, "index.js", `module.exports = { marker: ${JSON.stringify(MARKER)} }\n`)
55+
elsewhere = path.join(root, "unrelated-run-dir")
56+
fs.mkdirSync(elsewhere, { recursive: true })
57+
})
58+
59+
afterEach(() => {
60+
process.chdir(originalCwd)
61+
if (root) fs.rmSync(root, { recursive: true, force: true })
62+
})
63+
64+
describe("resolution does not depend on the working directory", () => {
65+
test("resolves the same package before and after a chdir", () => {
66+
const before = resolveOptionalPackage(FIXTURE, [nodeModules])
67+
expect(before).toBeDefined()
68+
69+
process.chdir(elsewhere)
70+
const after = resolveOptionalPackage(FIXTURE, [nodeModules])
71+
expect(after).toBe(before)
72+
})
73+
74+
test("resolves from a directory that is not an ancestor of the package", () => {
75+
// `elsewhere` shares only the temp root with the package tree, so nothing
76+
// about it can contribute to resolution. This is the rig's shape: the run
77+
// directory and the install tree are unrelated.
78+
process.chdir(elsewhere)
79+
const resolved = resolveOptionalPackage(FIXTURE, [nodeModules])
80+
expect(resolved).toBeDefined()
81+
expect(resolved!.startsWith(nodeModules)).toBe(true)
82+
// Load it and read the marker, so the test reports which root satisfied the
83+
// lookup rather than merely that something was found.
84+
const loaded = createRequire(pathToFileURL(resolved!).href)(resolved!)
85+
expect(loaded.marker).toBe(MARKER)
86+
})
87+
88+
test("does not resolve out of the working directory's own node_modules", () => {
89+
// A package present only under cwd must stay invisible: project trees are
90+
// workspace-controlled executable content and are deliberately not searched.
91+
const cwdModules = path.join(elsewhere, "node_modules")
92+
fs.mkdirSync(cwdModules, { recursive: true })
93+
writePackage(cwdModules, CWD_ONLY, "index.js", `module.exports = { marker: ${JSON.stringify(MARKER)} }\n`)
94+
95+
process.chdir(elsewhere)
96+
// The specifier exists nowhere else on the machine, so this is absence with
97+
// a known cause: anything but undefined means the lookup reached into the
98+
// working directory.
99+
expect(resolveOptionalPackage(CWD_ONLY, driverSearchRoots())).toBeUndefined()
100+
})
101+
102+
test("loads from the package's own directory while cwd is somewhere else", async () => {
103+
// Resolution being cwd-independent is not enough: the *load* consults the
104+
// package manifest too, and in a compiled binary that lookup was observed
105+
// resolving against the process working directory —
106+
// `ENOENT ... open '<cwd>/usr/lib/.../duckdb/package.json'` for a file that
107+
// exists at that path without the prefix. This pins the load itself.
108+
//
109+
// The fixture reports its own __dirname, so the assertion is about which
110+
// directory the module was loaded from rather than merely that it loaded.
111+
const pkgDir = path.join(nodeModules, FIXTURE)
112+
fs.writeFileSync(path.join(pkgDir, "index.js"), "module.exports = { dir: __dirname }\n")
113+
114+
process.chdir(elsewhere)
115+
116+
// The only route to the fixture root is the path named in the ambient
117+
// failure, which is how the real failure surfaces it. Naming the plain
118+
// absolute path here would find the fixture regardless of whether
119+
// `repairCwdPrefixedPath` works — that path already exists on disk, so
120+
// this test would still pass with the repair removed. Instead reproduce
121+
// the observed shape exactly: the working directory concatenated onto the
122+
// already-absolute manifest path, `<cwd>/usr/lib/.../duckdb/package.json`
123+
// — a location that exists only after the `<cwd>` prefix is stripped.
124+
const real = path.join(pkgDir, "package.json")
125+
const concatenated = elsewhere + real
126+
expect(fs.existsSync(concatenated)).toBe(false)
127+
const ambient = Object.assign(new Error(`ENOENT: no such file or directory, open '${concatenated}'`), {
128+
code: "ENOENT",
129+
})
130+
const importer = async () => {
131+
throw ambient
132+
}
133+
134+
const loaded: any = await loadOptionalDriver("duckdb", FIXTURE, importer)
135+
const mod = loaded?.default ?? loaded
136+
expect(mod.dir).toBe(fs.realpathSync(pkgDir))
137+
})
138+
139+
test("treats a descendant named with a leading '..' as workspace-controlled, not parent traversal", () => {
140+
// A directory component that merely STARTS WITH ".." — "..evil" — is a
141+
// real child of cwd, not parent traversal, even though the relative path
142+
// string built from it also starts with the two characters "..". Only
143+
// ".." itself, or ".." followed by a separator, means a path actually
144+
// climbed above cwd; a naive `rel.startsWith("..")` check conflates the
145+
// two and would admit this as an external, non-workspace root.
146+
const evilNodeModules = path.join(root, "..evil", "node_modules", "duckdb")
147+
fs.mkdirSync(evilNodeModules, { recursive: true })
148+
fs.writeFileSync(path.join(evilNodeModules, "package.json"), JSON.stringify({ name: "duckdb", version: "1.0.0" }))
149+
150+
process.chdir(root)
151+
152+
const named = path.join(evilNodeModules, "package.json")
153+
const ambient = Object.assign(new Error(`ENOENT: no such file or directory, open '${named}'`), {
154+
code: "ENOENT",
155+
})
156+
157+
const roots = searchRootsFromError(ambient)
158+
159+
// The workspace-controlled node_modules under "..evil" must not be
160+
// returned — it is a real descendant of cwd, so it stays excluded exactly
161+
// like any other project node_modules.
162+
expect(roots).toEqual([])
163+
})
164+
165+
test("excludes a nested dependency root under an ancestor node_modules when cwd starts below the project root", () => {
166+
// The CLI can start in a subdirectory that has no node_modules of its own
167+
// — cwd = <project>/packages/app — while the hoisted node_modules lives at
168+
// the project root above it. A nested dependency there,
169+
// <project>/node_modules/host/node_modules/duckdb, is workspace-controlled
170+
// through its *outer* node_modules even though it is not itself one of the
171+
// exact directories `nodeModulesUpward(cwd)` walks to.
172+
const project = path.join(root, "project")
173+
const nested = path.join(project, "node_modules", "host", "node_modules", "duckdb")
174+
fs.mkdirSync(nested, { recursive: true })
175+
fs.writeFileSync(path.join(nested, "package.json"), JSON.stringify({ name: "duckdb", version: "1.0.0" }))
176+
177+
const cwd = path.join(project, "packages", "app")
178+
fs.mkdirSync(cwd, { recursive: true })
179+
process.chdir(cwd)
180+
181+
const named = path.join(nested, "package.json")
182+
const ambient = Object.assign(new Error(`ENOENT: no such file or directory, open '${named}'`), {
183+
code: "ENOENT",
184+
})
185+
186+
const roots = searchRootsFromError(ambient)
187+
188+
// Neither the inner nested root nor the outer project node_modules it sits
189+
// under may be returned: both are workspace-controlled, and importing from
190+
// either during a warehouse read/test would cross the permission boundary
191+
// driverSearchRoots() otherwise enforces.
192+
expect(roots).not.toContain(fs.realpathSync(nested))
193+
expect(roots).not.toContain(fs.realpathSync(path.join(project, "node_modules")))
194+
expect(roots).toEqual([])
195+
})
196+
197+
test("a chdir between resolve and re-resolve does not change the answer", () => {
198+
process.chdir(elsewhere)
199+
const first = resolveOptionalPackage(FIXTURE, [nodeModules])
200+
process.chdir(originalCwd)
201+
const second = resolveOptionalPackage(FIXTURE, [nodeModules])
202+
process.chdir(root)
203+
const third = resolveOptionalPackage(FIXTURE, [nodeModules])
204+
expect(second).toBe(first)
205+
expect(third).toBe(first)
206+
})
207+
})

0 commit comments

Comments
 (0)